LavaTech Infrastructure Review, 2020 Q1 Edition

Hey guys, this is our first ever LavaTech Infrastructure Review. This is a set of posts that we aim to do every quarter, even if there's no change.

However, in this quarter, we did have some significant changes.

This document was mostly written by Ave, but it was reviewed and approved by both LavaTech members.

Cold storage for backups

One of the main changes we made in this quarter was moving important old backups from hot storage to cold storage, and nuking unneeded backups.

In the end we removed around 50GB of files and moved around 800GB of files.

We use Backblaze B2 for hot storage. We chose Scaleway C14 Cold Storage for cold storage (I explained why here).

The move of backups is still not automated, but it'll be done when I find the time. Right now we use rclone to upload, here's our config.

I live tweeted our move, but here's an important note:

For 500GB: – B2 costs $2.5/mo – B2 download costs $5 – C14 Cold Storage costs EUR1/mo – C14 download costs EUR5 (if not to a server on scaleway itself, which let's be honest, very few will do that)

So, while C14 Cold Storage is ~2.5x cheaper to store, it costs the roughly the same to get data off of it.

For us, it'll make more sense to upload to B2 and C14 simultaneously, and then delete files off of B2 after a week.

It cost us roughly a month's worth of B2 fees just in download fees, and C14 does cost ~2.5x less as I stated above, so quick maths states that it'll pay itself off in a couple months.

Netbox

Until around mid 2019, we used to either informally exchange information about servers. In mid 2019, we created a private repo called infra under elixire group on Gitlab.com.

This was where we kept our issues about servers, and where we kept markdown files describing some stuff that may be handy to know about servers.

However, this was not standardized, and didn't cover everything. It was mostly just notes.

I did know that netbox existed as our friends at general programming have one, and I intended to set up one for ourselves, but I only really got time to do so a couple weeks ago.

So, I set up a netbox instance, and after Luna and me working on and off for weeks, we successfully completed adding everything. Everything being anything from cloud servers to home servers, containers to virtual machines (both were added as VMs), IPs to services have been added.

During this process, we identified numerous unused servers, and numerous anomalies (such as 10GB of ram on a container that just runs a webserver), which led to us nuking numerous VPSes and CTs.

Bye bye VPS: avefedi

avefedi was a server that was dedicated to hosting the Pleroma instance on p.a3.pm.

It previously hosted 90dns and they were both on same server, but server crashed randomly and I created a new server for 90dns as I worked with Scaleway to bring this machine up. This server came up after a day or two of downtime, and was dedicated to hosting the p.a3.pm Pleroma instance ever since. It's honestly overkill for just a Pleroma instance, which is why I moved it to a container.

"Instance has been deleted"

Bye bye VPS: demonhouse

demonhouse was a shared server that we gave access to close friends to host small stuff they needed to host. Everyone had root, and it was in use as late as October 2019.

Bye bye VPS: einf

einf was a server that was dedicated to moving files between cloud providers, then ran a honeypot for a couple weeks.

Bye bye CTs: firefly, lasagnatube

firefly was a personal container hosting a Firefly III instance. Firefly III is an open source financial tracking software, one I tried to use to track my spending, but it quickly became a burden on me to keep it up to date, and i stopped using it.

lasagnatube was a container hosting LasagnaTube. While LasagnaTube is owned by Lasagna Ltd, it was hosted on LavaTech servers, which is why it's mentioned here. LasagnaTube files were moved into cold storage.

(Tags to jump to other related posts: #infrareview #report)