Hey guys, this is our first ever LavaTech Infrastructure Review. This is a set of posts that we aim to do every quarter, even if there's no change.

However, in this quarter, we did have some significant changes.

This document was mostly written by Ave, but it was reviewed and approved by both LavaTech members.

Cold storage for backups

One of the main changes we made in this quarter was moving important old backups from hot storage to cold storage, and nuking unneeded backups.

In the end we removed around 50GB of files and moved around 800GB of files.

We use Backblaze B2 for hot storage. We chose Scaleway C14 Cold Storage for cold storage (I explained why here).

The move of backups is still not automated, but it'll be done when I find the time. Right now we use rclone to upload, here's our config.

I live tweeted our move, but here's an important note:

For 500GB: – B2 costs $2.5/mo – B2 download costs $5 – C14 Cold Storage costs EUR1/mo – C14 download costs EUR5 (if not to a server on scaleway itself, which let's be honest, very few will do that)

So, while C14 Cold Storage is ~2.5x cheaper to store, it costs the roughly the same to get data off of it.

For us, it'll make more sense to upload to B2 and C14 simultaneously, and then delete files off of B2 after a week.

It cost us roughly a month's worth of B2 fees just in download fees, and C14 does cost ~2.5x less as I stated above, so quick maths states that it'll pay itself off in a couple months.

Netbox

Until around mid 2019, we used to either informally exchange information about servers. In mid 2019, we created a private repo called infra under elixire group on Gitlab.com.

This was where we kept our issues about servers, and where we kept markdown files describing some stuff that may be handy to know about servers.

However, this was not standardized, and didn't cover everything. It was mostly just notes.

I did know that netbox existed as our friends at general programming have one, and I intended to set up one for ourselves, but I only really got time to do so a couple weeks ago.

So, I set up a netbox instance, and after Luna and me working on and off for weeks, we successfully completed adding everything. Everything being anything from cloud servers to home servers, containers to virtual machines (both were added as VMs), IPs to services have been added.

During this process, we identified numerous unused servers, and numerous anomalies (such as 10GB of ram on a container that just runs a webserver), which led to us nuking numerous VPSes and CTs.

Bye bye VPS: avefedi

avefedi was a server that was dedicated to hosting the Pleroma instance on p.a3.pm.

It previously hosted 90dns and they were both on same server, but server crashed randomly and I created a new server for 90dns as I worked with Scaleway to bring this machine up. This server came up after a day or two of downtime, and was dedicated to hosting the p.a3.pm Pleroma instance ever since. It's honestly overkill for just a Pleroma instance, which is why I moved it to a container.

• Provider: Scaleway (start1-xs)
• Ran between: December 1, 2018 – March 17, 2020 (1 year, 3 months, 16 days)
• Saved: 2EUR/mo

Bye bye VPS: demonhouse

demonhouse was a shared server that we gave access to close friends to host small stuff they needed to host. Everyone had root, and it was in use as late as October 2019.

• Provider: Hetzner (CX11)
• Ran between: October 10, 2018 – March 4, 2020 (1 year, 4 months, 25 days)
• Saved: 2.49EUR/mo

Bye bye VPS: einf

einf was a server that was dedicated to moving files between cloud providers, then ran a honeypot for a couple weeks.

• Provider: Hetzner (CX11)
• Ran between: January 21, 2020 – March 4, 2020 (1 month, 14 days)
• Saved: 2.49EUR/mo

Bye bye CTs: firefly, lasagnatube

firefly was a personal container hosting a Firefly III instance. Firefly III is an open source financial tracking software, one I tried to use to track my spending, but it quickly became a burden on me to keep it up to date, and i stopped using it.

lasagnatube was a container hosting LasagnaTube. While LasagnaTube is owned by Lasagna Ltd, it was hosted on LavaTech servers, which is why it's mentioned here. LasagnaTube files were moved into cold storage.

(Tags to jump to other related posts: #infrareview #report)

What's happening forum?

This is our first financial report to date, due to this, there won't be any data to compare to, but we hope that the data is useful regardless.

We also released a transparency report earlier this week, you can read it here.

This is released on 2020-02-27, with numbers as of this very date.

This document was mostly written by Ave, but it was reviewed and approved by both LavaTech members.

• This document was edited on 2020-03-12 to add KernelCare fees. Similarly, title was changed from 2020 to 2019, because calling it 2020 was a bad idea.

Income: Patreon

Patreon is our main source of donations for LavaTech.

• In 2019, we made $563 (minus taxes and fees) through Patreon.
• We had no refunds.
• $83.00 went to supporting other creators (I sadly cannot find a way to pull up a list of who they were, however as of November 2019, we aren't paying for our Patreon patronage through LavaTech).
• Starting in June, we started using Payoneer for server payments to be able to use a debit card.
• Before June, we were using Luna's Paypal, but we had to switch away due to Brazilian Paypal banning the use of balance internationally.
• Move to Payoneer introduced further fees.

After support for other creators, we were left with $480.

Expenses: Taxes and fees

• $36.76 went to VAT.
• $35.93 went to Patreon processing fees.
• $28.15 went to Patreon platform fees.
• $12.00 went to activating out Payoneer debit card.
• $11.62 went to Payoneer maintenance fees.
• $9.32 went to Payoneer (outbound) transaction fees.
• $6.00 went to Payoneer (inbound) transaction fees.

In total, $139.78 (29.12% of the amount that was left after support for other creators) went to taxes and fees in 2019.

Here's a fancy pie chart, because pie charts are good:

Expense: Backblaze B2

Backblaze B2 is our preferred data storage service of choice for backups. As our services and the data we store continue to grow, so does our backup bill. For 2020, I am considering moving old backups to cold storage to accommodate for this.

• $35.07 went to B2.
• While we paid $0.78 on January, we paid $5.17 on December.
• (Informational) We paid $6.43 on February 2020.

Expense: Cloudflare

Starting in July, we started hosting switchroot files.

I (ave) made an optimistic move and didn't set up proper RAM caching. Less than 10 seconds after we launched the image files, the server's drives were overloaded, and no one managed to download anything.

As a result, we distributed files to multiple servers and set up Cloudflare Load Balancing (which costs $5/mo), which solved the issue.

• $27.26 went to Cloudflare Load Balancing.
• Domain fees from Cloudflare is not included here.

Expense: Servers

Obviously, our services require servers. A lot of them, in fact. We have a lot of them for that very reason.

Hetzner is used for most of our server needs and hosts most of our services. Online.net hosts the mirrors, Scaleway hosts (European) 90dns.

All of our server payments are in Euros.

• €534.94 (~$585.15) went to Hetzner.
• Cheapest months on Hetzner were April to July, with each month costing €40.85.
• Most expensive months on Hetzner was December, costing €49.57.
• €143.88 (~$157.45) went to Online.net. All months were €11.99.
• €55.99 (~$61.26) went to Scaleway.

In total, we paid €734.81 (~$803.80) for servers in 2019.

Expense: Domains

And obviously, what good is an image host without domains?

Do note that these numbers aren't perfect, as lines somewhat blur between LavaTech domain costs (like elixi.re) and non-LavaTech domain costs (like ave.zone). I've however left out registrars that I only have non-LavaTech domains from, such as nictr (now metunic).

• $146.21 went to Porkbun.
• $27.46 went to Dynadot.
• $23.96 went to Namesilo (As of 2020, we are no longer working with Namesilo).
• $22.53 went to Namecheap.
• $8.03 went to Cloudflare.

In total, we paid $228.19 for domains in 2019.

Expense: HIBP

We had a HIBP API subscription for 3 months to be used with our bitwarden server, so we paid $10.5 (+$0.18 payoneer fees) to that.

Expense: KernelCare

We have KernelCare on our main hypervisor, edgebleed. We try to keep downtime to a minimum, and kcare helps with that.

We get a 2 server license, so we pay $2.95/mo per server, but as one of them is used by our friends at General Programming, it's not included in this specific report.

• $35.40 was spent for KernelCare in 2019 ($2.95/mo).

Personal Expenses through LavaTech funds: G Suite, Steam

These were expenses we made for ourselves, and were rare. We are trying to keep these to a minimum. Considering we pay a significant chunk of the LavaTech expenses from pocket, we hope that they'll be excused.

These numbers include payoneer fees.

• $5.97 was spent on Steam
• $20.32 was spent on G Suite

In total, we paid $26.29 for personal expenses through LavaTech funds in 2019.

In conclusion

• We made $563.
• We got to keep $340.22 of it after fees, taxes and support towards other creators.
• We spent $1140.22 (doesn't include personal expenses).
• After fees, taxes and support towards other creators, 29.83% of our expenses were paid by donations. Exactly $800 was paid out of pocket.

Also, another pie chart, this time showing expenses (somewhat outdated, doesn't include kernelcare):

We'd like to thank all of you for supporting us, by using our services, by recommending our services, and by donating domains and funds to our services.

Shameless plug: If you'd like to help make that percentage be higher for 2020, here's our patreon. Anything helps.

(Tags to jump to other related posts: #financialreport #report)

What's happening forum?

This is our first transparency report to date, due to this, there won't be any data to compare to, but we hope that the data is useful regardless.

We are also considering releasing a financial report, including server costs and income from patreon.

We also released a financial report, you can read it here.

This is released on 2020-02-25, with numbers as of this very date.

• This document was edited on 2020-03-12 to correct the phrase “We haven't received any warrant canaries” to “We haven't received any warrants”. Similarly, title was changed from 2020 to 2019, because calling it 2020 was a bad idea.

In general...

We haven't received any warrants or anything like that to date.

No DMCAs in any service in 2019.

No C&Ds in any service in 2019.

elixi.re

elixi.re is continuing to be a thing as similar services go up and down. 2019 saw the death of bisoga, and as we're writing this, ratelimited.me is down due to a drive failure.

We're at 60 domains, 7 of them are currently without any users. We've switched to votes for domains in 2019 to remedy this issue, and it seems to work well so far. (only domains that ended up without users were shitpost domains I bought and added on impulse without voting, oops -ave)

In 2019, we did a server upgrade which resulted in less drive space but a lot better compute capabilities. Right now the storage space isn't a major concern.

v3 is under development. ETA: Soon™.

Right now, there's 606 active and 293 inactive users.

There's 473480 files (equaling 115503.07MiB), with 5333 of them from last week (equaling 1361.02MiB).

There's 1923 shortens, with 27 of them from the last week.

a3.pm XMPP services

User count is 1042.

2 people were banned for violating the LSA, one for commercial activity, one for being a “Problematic Person”.

We have shut down registrations for the foreseeable future to allow for moderation to be possible. Our server hardware is also struggling, especially postgresql during ejabberd's boot, and this also contributed to the registration being closed. if you want to help us diagnose this issue, feel free to contact us!

Over the year we've followed the latest updates, and we're currently running latest upstream. Similarly, we've continued to keep our config files open to encourage people to deploy XMPP servers.

90dns

90dns is going strong. There's ~20k unique IPs per day, 4k of these in the US instance.

Gitdab

Gitdab is growing steadily, as it is open to register and receives updates fairly soon after upstream. And god, the dab meme will never die, will it?

• 323 users
• 61 organizations
• 153 repositories
• 70 teams

Mirrors

We now have an official Manjaro ARM mirror at https://manjaroarm.mirrors.lavatech.top.

To fit with the URL scheme, https://archmirror.lavatech.top was moved to https://arch.mirrors.lavatech.top, and it now has proper, working IPv6.

BSNX

It's down, it's been down, and it doesn't seem like I'll have the energy to fix it soon. The split within the scene of people who are supportive of such services and people who are against it does make this harder, as while this is an easy technical feat, it's definitely not easy from a bureaucracy perspective.

LavaDNS

Reminder: LavaDNS is not 90dns.

LavaDNS is relatively deprecated. The future of it is uncertain, maybe it'll turn out to be unbound, or still experimental handwritten servers.

LavaSearx

LavaSearx was kept maintained for quite a while, but it's rather slow as it uses Searx. (and I'll be honest: I moved back to DDG. -ave)

The service is still up, but it's at a degraded state. It's not synced to upstream, and ddg seems to be broken. Similarly, Google banned our IP, heh.

(Tags to jump to other related posts: #transparencyreport #report)