Hey folks!

We unfortunately had about half an hour of unplanned downtime today. This was caused by an issue with our hosting provider. The issue is solved for now, and I am planning to make some changes to prevent similar issues in the future. Sorry for the inconvenience!

Technical details

Our servers are communicating with our database over Hetzner’s “vSwitch” service. Unfortunately, this service seems to be quite flaky - over the past few months, I have had to deal with the connection just dropping without recovering many times. Mostly this has not resulted in any noticeable downtime, as we have redundant servers, so even if one of them stops working, it won’t affect lemm.ee users. However, in this instance, all of our API servers lost their connection to our database at the same time, which resulted in actual downtime.

I have now decided to migrate our setup away from the vSwitch in the near future to hopefully stop these issues for good. Should be possible to do this migration without any downtime, I just need to set aside some time to actually create an alternative solution for us, most likely over the coming weekend. I will update this post once the migration is complete.

Update: the migration is now complete! You can read more here.

  • notabot@lemm.ee
    0·
    6 months ago

    Thanks for the update. If you have the time and energy after the migration I’d be interested to hear details of the new setup, we have servers using vswitch too, and while I haven’t run into trouble myself I’d like to avoid it if possible. You know what they say about the wise learning from others’ experience!

  • Kroxx@lemm.ee
    0·
    6 months ago

    Thanks for the update, even with that little downage .ee is insanely stable!

  • Evil_Shrubbery@lemm.ee
    0·
    6 months ago

    In the future please schedule your unplanned downtime around my (also unplanned) pooping schedule, thank you. Today’s experience was an almost barely noticeable inconvenience!

    (Ofc I kid, lemm.ee is awesome!)

  • Zatore@lemm.ee
    0·
    6 months ago

    Thanks for telling us what happened! I know there’s nothing we can do to help, but I’m always curious to know what happened. It can also be a great learning experience.