New Server - Round #2
Intro
Last time I mentioned that my ZimaBoard 832 server died on me. Let’s see how someone with a small homelab can manage these kind of situations.
D(isaster)
I was not at home when the server died.
As Rule #0, I think we can safely say that these things always happen when you cannot access the hardware physically.
The first thing to keep in mind is that it’s not a good idea to put all your eggs in the same basket. For example, I was not completely lost, because my access to my home network was not affected, as the WireGuard VPN endpoint runs on my router. Of course, if the router dies, that’s a whole different problem, but still, you probably understand what I’m trying to say here. Which is:
Rule #1: Separate your services, if you are able to do so.
R(ecovery)
So. How can I get back the bare minimum of services? I could not live without my RSS reader, for example. My better half was at home, trying to watch a Hallmark movie, with some popcorn already made. I was not able to access my Vaultwarden vault. Pick one of these as the highest priority.
Rule #2: Have a spare hardware you can use to restore services to.
The whole moving to a new server was prompted by the fact that my Synology NAS could not handle the load anymore. But a bare minimum of services? No sweat.
Docker was still running on it, my docker-compose files are also stored on it (and in other places, like on the dead server, but that’s not gonna help now, does it?), so regarding services, I was fine. But what about the data those servers are, well… serving?
Rule #3: You should always have backup.
I try to implement the classic 3-2-1 backup rule at home, which apparently just saved my ass. I was able to restore everything from last night’s backup of the docker volumes. Girl movie night was saved, I could read my news, and also did not have to remember all my passwords.
P(lan)
So to reiterate:
- Reliable access to your network is paramount if you are not on location — and believe me, you will not be on location when the shit hits the fan.
- Spare hardware. I was lucky this time, it could have been worse.
- Backup. Backup. Backup. And test those backups regularly.
Aftermath
After all this excitement I realized how dependent I am on my little home setup. This realization helped me decide on buying a new server. I chose an Intel NUC this time, more specifically an Intel NUC 11 Essential Kit with a 500 GB SSD and 32 GB of RAM. It is somewhat of an overkill for my use case, especially on the RAM front, but I’m sure I’ll be able to overload this one as well sometime in the near future. I installed Proxmox on it, so a nice Windows 11 VM might just do the job. Not that I needed Windows in my life, but why the hell not.
In the meantime, I got in touch with Icewhale (the maker of ZimaBoard), and after a few attempts of troubleshooting, they just sent me a replacement board. I’d like to praise their absolute helpfulness in this matter. Honestly, I was surprised by the level of commitment they show towards their customers.
The replacement arrived after a few weeks.
It’s running a Proxmox Backup Server, because you know… Backups are important.