This is our second transparency report to date. We've provided comparisons in this document, but if you'd like to review the 2019 one yourself regardless, you can access it here.

We also released the 2020 financial report, you can read it here.

This document was prepared on 2021-03-14, with numbers based around this date. It was mostly written by Ave, but it was reviewed and approved by both LavaTech members.

In general...

We haven't received any warrants or anything like that to date.

No DMCAs in any service in 2020.

No C&Ds in any service in 2020.

elixi.re

elixi.re is alive, as it has been for the past few years.

We're at 66 domains (including admin-only ones), compared to 60 of 2019.

We've had a complete hardware change in 2020, moving from a Hetzner dedicated server to a colocated server. Compared to 2019, storage space is a bigger concern than before, and we're exploring options.

v3 backend is feature complete, frontend still needs work. ETA: Soon™.

We have 691 active and 803 inactive users, which is to say that we got 85 new active users and 510 new inactive users since 2019. We're quite picky, I suppose.

There's 747922 files (taking up 182.21GiB), with 3881 of them from last week (taking up 982.94MiB).

In 2019, we had 473480 files (112.79GiB), and at the week of writing the blog, we had 5333 uploads in the last week (1361.02MiB).

While our total file count has increased by around 50%, our weekly counts slowed down a bit, even with Julian uploading a ton of cat pictures.

There's 1927 shortens, with 4 of them from the last week. That's only 4 new shortens in a whole year. It's not the most popular feature, you see.

a3.pm XMPP services

User count is 1072, up from 1042 in 2019.

No one was banned in 2020.

We've kept our server up to date as before, albeit slower, and we're running the latest version right now. We're still releasing our latest config files open to encourage people to deploy their own XMPP servers.

90dns

90dns has almost doubled in daily unique IP counts.

Right now, there's ~39k unique IPs per day (up from ~20k), ~10k of these in the US instance (up from ~4k).

Gitdab

Gitdab had a major user registration spam issue over the year, and we had to constantly wipe unverified users. Sadly, we also had to switch from a built-in captcha to hcaptcha after a while to stop this spam, which ended up working fine.

• 473 users (2019: 323)
• 98 organizations (2019: 61)
• 415 repositories (2019: 153)

Bitwarden

We spun up a bitwarden+bitbetter instance in 2020, and after a couple months of non-stop issues, we moved to bitwarden_rs.

We're quite happy with bitwarden_rs so far.

• 57 users

Mirrors

We didn't add any new mirrors in 2020, but we've maintained our existing mirrors at:

• https://manjaroarm.mirrors.lavatech.top
• https://arch.mirrors.lavatech.top

LavaDNS

LavaDNS was killed... and... LavaDNS v2 was born: https://dns.lavate.ch/

LavaDNS v2 uses industry standard software, and provides DNS, DoH, DoT and dnscrypt. Oh, and it doesn't keep logs, and is available in two locations: US and Finland.

LavaSearx

Heck, I (ave) forgot LavaSearx was even a thing!

The service is still up, but it's at a degraded state. It's not synced to upstream, and is probably broken.

We should probably take it down.

(Tags to jump to other related posts: #transparencyreport #report)

This is our second financial report to date. We've provided comparisons in this document, but if you'd like to review the 2019 one yourself regardless, you can access it here.

We also released the 2020 transparency report, you can read it here.

This document was mostly written by Ave, but it was reviewed and approved by both LavaTech members.

Income: Patreon

Patreon is our main source of donations for LavaTech.

• In 2020, we made $1028 (minus fees) through Patreon (2019: $563).
• We had no refunds (2019: same).

Before fees, we made $1028.

Expenses: Fees

• $53.00 went to Patreon processing fees (2019: $35.93).
• $51.40 went to Patreon platform fees (2019: $28.15).
• $36.00 went to Payoneer maintenance fees (2019: $11.62).
• $21.62 went to Payoneer (outbound) transaction fees (2019: $9.32).
• $12.00 went to Payoneer (inbound) transaction fees (2019: $6.00).

In total, $172.02 (16.73% of the income) went to taxes and fees in 2020 (2019: $139.78, 29.12%).

Here's a lazy pie chart for your enjoyment:

Expense: Backblaze B2

Backblaze B2 is our preferred data storage service of choice for backups.

• We paid $98.25 to B2 in 2020 (2019: $35.07).
• Our bill fluctuated over the months. We started the year with a $5.89 bill, went up to $13.83 on February, went down to $3.55 on April after we moved lots of old data to cold storage (relevant blog post). We finished the year with a $10.77 bill on December.

Expense: Cloudflare

We started using Cloudflare Load Balancing for Switchroot in July 2019 and stopped in July 2020.

• $12 went to Cloudflare Load Balancing.
• Domain fees from Cloudflare are not included here. You can find them in the domain expenses section of this post.

Expense: Servers

2020 was an interesting year as we've moved from a dedicated server in Hetzner to a colocated server in August.

The new server belongs to Ave, the network and rack belongs to Lasagna Ltd (a company owned by Ave, unrelated to LavaTech). As such, server, network and colocation costs are NOT reflected here.

• We paid €470.70 ($532.58) to Hetzner. (2019: €534.94)
• Most expensive month on Hetzner was July with €49.98 and cheapest months were September, October and November with €20.60. The infrastructure review we did in Q1 2020 also helped with reducing expenses.
• We paid €143.88 (~$171.98) to Online.net (Scaleway Dedibox). All months were €11.99. (2019: €143.88)
• We paid €67.90 (~$81.16) to Scaleway Elements (2019: €55.99).

Note: All of Scaleway was paid directly by Ave. All but 3 months (€35.97) of Online.net were paid by Ave.

In total, we paid €682.48 (~$785.72) for servers in 2020 (2019: €734.81).

Expense: Domains

We bought many cute domains this year.

While we do our best to reflect the correct numbers here, they're not perfect. Stuff like elixi.re domains we pay for renewal of (but don't own yet) aren't included, and we may have forgotten to include some registrars.

Do note that 2019 numbers do not exclude personal purchases while 2020 numbers do.

• We paid $86.85 to Porkbun. (2019: $146.21)
• We paid $27.96 to Dynadot. (2019: $27.46)
• We paid $41.96 to Namecheap (2019: $22.53).
• We paid $16.21 to Cloudflare (2019: $8.03).

In total, we paid $172.98 for domains in 2020 (2019: $228.19).

Expense: KernelCare

We have KernelCare on our main hypervisor, laserjet.

We have a 2 server license, one in use for LavaTech and other in use by our friends at General Programming (which is excluded from the number below). This arrangement allows us to get a slightly favorable per server cost.

During certain parts of the year we bought more licenses temporarily to test other servers and move between servers.

• We paid $46.01 for KernelCare in 2020 (2019: $35.40)

Note: Only one month ($6.90) was paid by LavaTech funds. Rest was paid directly by Ave.

Personal Expenses through LavaTech funds

I (Ave) owe a personal apology here for using the wrong card for one payment. Considering I also paid for a large number of expenses with my own funds, I hope this can be excused.

• $53.53 was paid to General Programming, LLC for helping install servers. (without fees: $53)

In total, we paid $53.53 for personal expenses through LavaTech funds in 2020 (2019: $26.29).

In conclusion

• We made $1028. (2019: $563)
• We got to keep $855.98 of it after fees. (2019: $340.22)
• We spent $1114.96 (doesn't include personal expenses) (2019: $1140.22).
• After fees, %76.77 of our expenses were paid by donations (2019: %29.83).
• Exactly $258.98 was paid out of pocket (2019: $800).

We'd like to thank all of you for supporting us, by using our services, by recommending our services, and by donating domains and funds to our services.

We're closer than ever to being self sustaining, and that makes us unbelievably happy. Thank you so much to everyone who used our services, who recommended our services and those who supported us in any way.

Shameless plug: If you'd like to help make that percentage be higher for 2021, here's our patreon. Anything helps.

(Tags to jump to other related posts: #financialreport #report)

Hey guys, this is our first ever LavaTech Infrastructure Review. This is a set of posts that we aim to do every quarter, even if there's no change.

However, in this quarter, we did have some significant changes.

This document was mostly written by Ave, but it was reviewed and approved by both LavaTech members.

Cold storage for backups

One of the main changes we made in this quarter was moving important old backups from hot storage to cold storage, and nuking unneeded backups.

In the end we removed around 50GB of files and moved around 800GB of files.

We use Backblaze B2 for hot storage. We chose Scaleway C14 Cold Storage for cold storage (I explained why here).

The move of backups is still not automated, but it'll be done when I find the time. Right now we use rclone to upload, here's our config.

I live tweeted our move, but here's an important note:

For 500GB: – B2 costs $2.5/mo – B2 download costs $5 – C14 Cold Storage costs EUR1/mo – C14 download costs EUR5 (if not to a server on scaleway itself, which let's be honest, very few will do that)

So, while C14 Cold Storage is ~2.5x cheaper to store, it costs the roughly the same to get data off of it.

For us, it'll make more sense to upload to B2 and C14 simultaneously, and then delete files off of B2 after a week.

It cost us roughly a month's worth of B2 fees just in download fees, and C14 does cost ~2.5x less as I stated above, so quick maths states that it'll pay itself off in a couple months.

Netbox

Until around mid 2019, we used to either informally exchange information about servers. In mid 2019, we created a private repo called infra under elixire group on Gitlab.com.

This was where we kept our issues about servers, and where we kept markdown files describing some stuff that may be handy to know about servers.

However, this was not standardized, and didn't cover everything. It was mostly just notes.

I did know that netbox existed as our friends at general programming have one, and I intended to set up one for ourselves, but I only really got time to do so a couple weeks ago.

So, I set up a netbox instance, and after Luna and me working on and off for weeks, we successfully completed adding everything. Everything being anything from cloud servers to home servers, containers to virtual machines (both were added as VMs), IPs to services have been added.

During this process, we identified numerous unused servers, and numerous anomalies (such as 10GB of ram on a container that just runs a webserver), which led to us nuking numerous VPSes and CTs.

Bye bye VPS: avefedi

avefedi was a server that was dedicated to hosting the Pleroma instance on p.a3.pm.

It previously hosted 90dns and they were both on same server, but server crashed randomly and I created a new server for 90dns as I worked with Scaleway to bring this machine up. This server came up after a day or two of downtime, and was dedicated to hosting the p.a3.pm Pleroma instance ever since. It's honestly overkill for just a Pleroma instance, which is why I moved it to a container.

• Provider: Scaleway (start1-xs)
• Ran between: December 1, 2018 – March 17, 2020 (1 year, 3 months, 16 days)
• Saved: 2EUR/mo

Bye bye VPS: demonhouse

demonhouse was a shared server that we gave access to close friends to host small stuff they needed to host. Everyone had root, and it was in use as late as October 2019.

• Provider: Hetzner (CX11)
• Ran between: October 10, 2018 – March 4, 2020 (1 year, 4 months, 25 days)
• Saved: 2.49EUR/mo

Bye bye VPS: einf

einf was a server that was dedicated to moving files between cloud providers, then ran a honeypot for a couple weeks.

• Provider: Hetzner (CX11)
• Ran between: January 21, 2020 – March 4, 2020 (1 month, 14 days)
• Saved: 2.49EUR/mo

Bye bye CTs: firefly, lasagnatube

firefly was a personal container hosting a Firefly III instance. Firefly III is an open source financial tracking software, one I tried to use to track my spending, but it quickly became a burden on me to keep it up to date, and i stopped using it.

lasagnatube was a container hosting LasagnaTube. While LasagnaTube is owned by Lasagna Ltd, it was hosted on LavaTech servers, which is why it's mentioned here. LasagnaTube files were moved into cold storage.

(Tags to jump to other related posts: #infrareview #report)

What's happening forum?

This is our first financial report to date, due to this, there won't be any data to compare to, but we hope that the data is useful regardless.

We also released a transparency report earlier this week, you can read it here.

This is released on 2020-02-27, with numbers as of this very date.

This document was mostly written by Ave, but it was reviewed and approved by both LavaTech members.

• This document was edited on 2020-03-12 to add KernelCare fees. Similarly, title was changed from 2020 to 2019, because calling it 2020 was a bad idea.

Income: Patreon

Patreon is our main source of donations for LavaTech.

• In 2019, we made $563 (minus taxes and fees) through Patreon.
• We had no refunds.
• $83.00 went to supporting other creators (I sadly cannot find a way to pull up a list of who they were, however as of November 2019, we aren't paying for our Patreon patronage through LavaTech).
• Starting in June, we started using Payoneer for server payments to be able to use a debit card.
• Before June, we were using Luna's Paypal, but we had to switch away due to Brazilian Paypal banning the use of balance internationally.
• Move to Payoneer introduced further fees.

After support for other creators, we were left with $480.

Expenses: Taxes and fees

• $36.76 went to VAT.
• $35.93 went to Patreon processing fees.
• $28.15 went to Patreon platform fees.
• $12.00 went to activating out Payoneer debit card.
• $11.62 went to Payoneer maintenance fees.
• $9.32 went to Payoneer (outbound) transaction fees.
• $6.00 went to Payoneer (inbound) transaction fees.

In total, $139.78 (29.12% of the amount that was left after support for other creators) went to taxes and fees in 2019.

Here's a fancy pie chart, because pie charts are good:

Expense: Backblaze B2

Backblaze B2 is our preferred data storage service of choice for backups. As our services and the data we store continue to grow, so does our backup bill. For 2020, I am considering moving old backups to cold storage to accommodate for this.

• $35.07 went to B2.
• While we paid $0.78 on January, we paid $5.17 on December.
• (Informational) We paid $6.43 on February 2020.

Expense: Cloudflare

Starting in July, we started hosting switchroot files.

I (ave) made an optimistic move and didn't set up proper RAM caching. Less than 10 seconds after we launched the image files, the server's drives were overloaded, and no one managed to download anything.

As a result, we distributed files to multiple servers and set up Cloudflare Load Balancing (which costs $5/mo), which solved the issue.

• $27.26 went to Cloudflare Load Balancing.
• Domain fees from Cloudflare is not included here.

Expense: Servers

Obviously, our services require servers. A lot of them, in fact. We have a lot of them for that very reason.

Hetzner is used for most of our server needs and hosts most of our services. Online.net hosts the mirrors, Scaleway hosts (European) 90dns.

All of our server payments are in Euros.

• €534.94 (~$585.15) went to Hetzner.
• Cheapest months on Hetzner were April to July, with each month costing €40.85.
• Most expensive months on Hetzner was December, costing €49.57.
• €143.88 (~$157.45) went to Online.net. All months were €11.99.
• €55.99 (~$61.26) went to Scaleway.

In total, we paid €734.81 (~$803.80) for servers in 2019.

Expense: Domains

And obviously, what good is an image host without domains?

Do note that these numbers aren't perfect, as lines somewhat blur between LavaTech domain costs (like elixi.re) and non-LavaTech domain costs (like ave.zone). I've however left out registrars that I only have non-LavaTech domains from, such as nictr (now metunic).

• $146.21 went to Porkbun.
• $27.46 went to Dynadot.
• $23.96 went to Namesilo (As of 2020, we are no longer working with Namesilo).
• $22.53 went to Namecheap.
• $8.03 went to Cloudflare.

In total, we paid $228.19 for domains in 2019.

Expense: HIBP

We had a HIBP API subscription for 3 months to be used with our bitwarden server, so we paid $10.5 (+$0.18 payoneer fees) to that.

Expense: KernelCare

We have KernelCare on our main hypervisor, edgebleed. We try to keep downtime to a minimum, and kcare helps with that.

We get a 2 server license, so we pay $2.95/mo per server, but as one of them is used by our friends at General Programming, it's not included in this specific report.

• $35.40 was spent for KernelCare in 2019 ($2.95/mo).

Personal Expenses through LavaTech funds: G Suite, Steam

These were expenses we made for ourselves, and were rare. We are trying to keep these to a minimum. Considering we pay a significant chunk of the LavaTech expenses from pocket, we hope that they'll be excused.

These numbers include payoneer fees.

• $5.97 was spent on Steam
• $20.32 was spent on G Suite

In total, we paid $26.29 for personal expenses through LavaTech funds in 2019.

In conclusion

• We made $563.
• We got to keep $340.22 of it after fees, taxes and support towards other creators.
• We spent $1140.22 (doesn't include personal expenses).
• After fees, taxes and support towards other creators, 29.83% of our expenses were paid by donations. Exactly $800 was paid out of pocket.

Also, another pie chart, this time showing expenses (somewhat outdated, doesn't include kernelcare):

We'd like to thank all of you for supporting us, by using our services, by recommending our services, and by donating domains and funds to our services.

Shameless plug: If you'd like to help make that percentage be higher for 2020, here's our patreon. Anything helps.

(Tags to jump to other related posts: #financialreport #report)

What's happening forum?

This is our first transparency report to date, due to this, there won't be any data to compare to, but we hope that the data is useful regardless.

We also released a financial report, you can read it here.

This is released on 2020-02-25, with numbers as of this very date.

• This document was edited on 2020-03-12 to correct the phrase “We haven't received any warrant canaries” to “We haven't received any warrants”. Similarly, title was changed from 2020 to 2019, because calling it 2020 was a bad idea.

In general...

We haven't received any warrants or anything like that to date.

No DMCAs in any service in 2019.

No C&Ds in any service in 2019.

elixi.re

elixi.re is continuing to be a thing as similar services go up and down. 2019 saw the death of bisoga, and as we're writing this, ratelimited.me is down due to a drive failure.

We're at 60 domains, 7 of them are currently without any users. We've switched to votes for domains in 2019 to remedy this issue, and it seems to work well so far. (only domains that ended up without users were shitpost domains I bought and added on impulse without voting, oops -ave)

In 2019, we did a server upgrade which resulted in less drive space but a lot better compute capabilities. Right now the storage space isn't a major concern.

v3 is under development. ETA: Soon™.

Right now, there's 606 active and 293 inactive users.

There's 473480 files (equaling 115503.07MiB), with 5333 of them from last week (equaling 1361.02MiB).

There's 1923 shortens, with 27 of them from the last week.

a3.pm XMPP services

User count is 1042.

2 people were banned for violating the LSA, one for commercial activity, one for being a “Problematic Person”.

We have shut down registrations for the foreseeable future to allow for moderation to be possible. Our server hardware is also struggling, especially postgresql during ejabberd's boot, and this also contributed to the registration being closed. if you want to help us diagnose this issue, feel free to contact us!

Over the year we've followed the latest updates, and we're currently running latest upstream. Similarly, we've continued to keep our config files open to encourage people to deploy XMPP servers.

90dns

90dns is going strong. There's ~20k unique IPs per day, 4k of these in the US instance.

Gitdab

Gitdab is growing steadily, as it is open to register and receives updates fairly soon after upstream. And god, the dab meme will never die, will it?

• 323 users
• 61 organizations
• 153 repositories
• 70 teams

Mirrors

We now have an official Manjaro ARM mirror at https://manjaroarm.mirrors.lavatech.top.

To fit with the URL scheme, https://archmirror.lavatech.top was moved to https://arch.mirrors.lavatech.top, and it now has proper, working IPv6.

BSNX

It's down, it's been down, and it doesn't seem like I'll have the energy to fix it soon. The split within the scene of people who are supportive of such services and people who are against it does make this harder, as while this is an easy technical feat, it's definitely not easy from a bureaucracy perspective.

LavaDNS

Reminder: LavaDNS is not 90dns.

LavaDNS is relatively deprecated. The future of it is uncertain, maybe it'll turn out to be unbound, or still experimental handwritten servers.

LavaSearx

LavaSearx was kept maintained for quite a while, but it's rather slow as it uses Searx. (and I'll be honest: I moved back to DDG. -ave)

The service is still up, but it's at a degraded state. It's not synced to upstream, and ddg seems to be broken. Similarly, Google banned our IP, heh.

(Tags to jump to other related posts: #transparencyreport #report)