We also released the 2020 financial report, you can read it here.

This document was prepared on 2021-03-14, with numbers based around this date. It was mostly written by Ave, but it was reviewed and approved by both LavaTech members.

In general...

We haven't received any warrants or anything like that to date.

No DMCAs in any service in 2020.

No C&Ds in any service in 2020.

elixi.re

elixi.re is alive, as it has been for the past few years.

We're at 66 domains (including admin-only ones), compared to 60 of 2019.

We've had a complete hardware change in 2020, moving from a Hetzner dedicated server to a colocated server. Compared to 2019, storage space is a bigger concern than before, and we're exploring options.

v3 backend is feature complete, frontend still needs work. ETA: Soon™.

We have 691 active and 803 inactive users, which is to say that we got 85 new active users and 510 new inactive users since 2019. We're quite picky, I suppose.

There's 747922 files (taking up 182.21GiB), with 3881 of them from last week (taking up 982.94MiB).

In 2019, we had 473480 files (112.79GiB), and at the week of writing the blog, we had 5333 uploads in the last week (1361.02MiB).

While our total file count has increased by around 50%, our weekly counts slowed down a bit, even with Julian uploading a ton of cat pictures.

There's 1927 shortens, with 4 of them from the last week. That's only 4 new shortens in a whole year. It's not the most popular feature, you see.

a3.pm XMPP services

User count is 1072, up from 1042 in 2019.

No one was banned in 2020.

We've kept our server up to date as before, albeit slower, and we're running the latest version right now. We're still releasing our latest config files open to encourage people to deploy their own XMPP servers.

90dns

90dns has almost doubled in daily unique IP counts.

Right now, there's ~39k unique IPs per day (up from ~20k), ~10k of these in the US instance (up from ~4k).

Gitdab

Gitdab had a major user registration spam issue over the year, and we had to constantly wipe unverified users. Sadly, we also had to switch from a built-in captcha to hcaptcha after a while to stop this spam, which ended up working fine.

• 473 users (2019: 323)
• 98 organizations (2019: 61)
• 415 repositories (2019: 153)

Bitwarden

We spun up a bitwarden+bitbetter instance in 2020, and after a couple months of non-stop issues, we moved to bitwarden_rs.

We're quite happy with bitwarden_rs so far.

• 57 users

Mirrors

We didn't add any new mirrors in 2020, but we've maintained our existing mirrors at:

• https://manjaroarm.mirrors.lavatech.top
• https://arch.mirrors.lavatech.top

LavaDNS

LavaDNS was killed... and... LavaDNS v2 was born: https://dns.lavate.ch/

LavaDNS v2 uses industry standard software, and provides DNS, DoH, DoT and dnscrypt. Oh, and it doesn't keep logs, and is available in two locations: US and Finland.

LavaSearx

Heck, I (ave) forgot LavaSearx was even a thing!

The service is still up, but it's at a degraded state. It's not synced to upstream, and is probably broken.

We should probably take it down.

(Tags to jump to other related posts: #transparencyreport #report)

We also released the 2020 transparency report, you can read it here.

This document was mostly written by Ave, but it was reviewed and approved by both LavaTech members.

Income: Patreon

Patreon is our main source of donations for LavaTech.

• In 2020, we made $1028 (minus fees) through Patreon (2019: $563).
• We had no refunds (2019: same).

Before fees, we made $1028.

Expenses: Fees

• $53.00 went to Patreon processing fees (2019: $35.93).
• $51.40 went to Patreon platform fees (2019: $28.15).
• $36.00 went to Payoneer maintenance fees (2019: $11.62).
• $21.62 went to Payoneer (outbound) transaction fees (2019: $9.32).
• $12.00 went to Payoneer (inbound) transaction fees (2019: $6.00).

In total, $172.02 (16.73% of the income) went to taxes and fees in 2020 (2019: $139.78, 29.12%).

Here's a lazy pie chart for your enjoyment:

Expense: Backblaze B2

Backblaze B2 is our preferred data storage service of choice for backups.

• We paid $98.25 to B2 in 2020 (2019: $35.07).
• Our bill fluctuated over the months. We started the year with a $5.89 bill, went up to $13.83 on February, went down to $3.55 on April after we moved lots of old data to cold storage (relevant blog post). We finished the year with a $10.77 bill on December.

Expense: Cloudflare

We started using Cloudflare Load Balancing for Switchroot in July 2019 and stopped in July 2020.

• $12 went to Cloudflare Load Balancing.
• Domain fees from Cloudflare are not included here. You can find them in the domain expenses section of this post.

Expense: Servers

2020 was an interesting year as we've moved from a dedicated server in Hetzner to a colocated server in August.

The new server belongs to Ave, the network and rack belongs to Lasagna Ltd (a company owned by Ave, unrelated to LavaTech). As such, server, network and colocation costs are NOT reflected here.

• We paid €470.70 ($532.58) to Hetzner. (2019: €534.94)
• Most expensive month on Hetzner was July with €49.98 and cheapest months were September, October and November with €20.60. The infrastructure review we did in Q1 2020 also helped with reducing expenses.
• We paid €143.88 (~$171.98) to Online.net (Scaleway Dedibox). All months were €11.99. (2019: €143.88)
• We paid €67.90 (~$81.16) to Scaleway Elements (2019: €55.99).

Note: All of Scaleway was paid directly by Ave. All but 3 months (€35.97) of Online.net were paid by Ave.

In total, we paid €682.48 (~$785.72) for servers in 2020 (2019: €734.81).

Expense: Domains

We bought many cute domains this year.

While we do our best to reflect the correct numbers here, they're not perfect. Stuff like elixi.re domains we pay for renewal of (but don't own yet) aren't included, and we may have forgotten to include some registrars.

Do note that 2019 numbers do not exclude personal purchases while 2020 numbers do.

• We paid $86.85 to Porkbun. (2019: $146.21)
• We paid $27.96 to Dynadot. (2019: $27.46)
• We paid $41.96 to Namecheap (2019: $22.53).
• We paid $16.21 to Cloudflare (2019: $8.03).

In total, we paid $172.98 for domains in 2020 (2019: $228.19).

Expense: KernelCare

We have KernelCare on our main hypervisor, laserjet.

We have a 2 server license, one in use for LavaTech and other in use by our friends at General Programming (which is excluded from the number below). This arrangement allows us to get a slightly favorable per server cost.

During certain parts of the year we bought more licenses temporarily to test other servers and move between servers.

• We paid $46.01 for KernelCare in 2020 (2019: $35.40)

Note: Only one month ($6.90) was paid by LavaTech funds. Rest was paid directly by Ave.

Personal Expenses through LavaTech funds

I (Ave) owe a personal apology here for using the wrong card for one payment. Considering I also paid for a large number of expenses with my own funds, I hope this can be excused.

• $53.53 was paid to General Programming, LLC for helping install servers. (without fees: $53)

In total, we paid $53.53 for personal expenses through LavaTech funds in 2020 (2019: $26.29).

In conclusion

• We made $1028. (2019: $563)
• We got to keep $855.98 of it after fees. (2019: $340.22)
• We spent $1114.96 (doesn't include personal expenses) (2019: $1140.22).
• After fees, %76.77 of our expenses were paid by donations (2019: %29.83).
• Exactly $258.98 was paid out of pocket (2019: $800).

We'd like to thank all of you for supporting us, by using our services, by recommending our services, and by donating domains and funds to our services.

We're closer than ever to being self sustaining, and that makes us unbelievably happy. Thank you so much to everyone who used our services, who recommended our services and those who supported us in any way.

Shameless plug: If you'd like to help make that percentage be higher for 2021, here's our patreon. Anything helps.

(Tags to jump to other related posts: #financialreport #report)

0) Open Conversations, go to the main view, hit the 3 dots on top right

1) Go to Manage accounts

2) Pick your account

3) Hit the 3 dots on top right, smash that “Change password” button

4) Change your password by putting your current password to the first box and the new one to the second, and hitting the “Change Password” button

Thank you for reading my guide.

(Tags to jump to other related posts: #guide, #a3pm)

• I've updated the server from ejabberd 20.03 to 20.04, released 4 days ago.
• I've enabled STUN/TURN to enable Video/Audio calls. This is somewhat big. Here's the relevant issue, and a number of links explaining this. You may use this feature with Conversations. I haven't tested it yet, however if you do manage to try it out, please let me know if it works or not on email or XMPP a@a3.pm.
• I synced the configuration on the server to the one on the repository. The actual settings were the same, but locations of them weren't, and they missed notes etc from each other. I've synced these, mostly from the repo to the server. You can find the relevant commit here.
• I've open sourced our MAM wiping systemd service/timer.
• We've updated the SSL certificate. This is a downtime we have to face roughly every 80 days.

Thanks as always for using our services, Ave

(Tags to jump to other related posts: #a3pm)

As title says, we used to be a Manjaro ARM mirror for several months, but exactly 4 weeks ago, it became a part of Manjaro's regular mirrors. This was rather big and honestly amazing news for Manjaro ARM.

It did however mean that our mirror would be deprecated.

We thought of becoming a regular mirror, but couldn't find easy resources on how to become an official one quickly, and I (ave) was quite busy at the time, so it got kind of ignored.

Well, that changes today! We're now an official Manjaro mirror. Well, it was ready yesterday, but it got approved today, and we only switched it to the manjaro.org upstream today (that's the equivalent of a T2 Arch repo going to T1).

You shouldn't need to do much and it should get activated automatically based on ping, but if you want to add it manually, our mirror is at https://manjaro.mirrors.lavatech.top/

Signed, LavaTech Team

(Tags to jump to other related posts: #manjaro #mirror)

I've just enabled experimental IPv6 support on a3.pm XMPP services.

By that, I mean that I've:

• Added AAAA records
• Tested basic XMPP functionality (successful connection, fetching message history, sending messages) with an IPv6-only connection (which succeeded)
• Updated nginx to also serve on IPv6 on port 80
• Updated sslh to also serve on IPv6 on port 443

One thing that I haven't tested that has a chance of failing is HTTP file uploads/fetches with IPv6, though it very likely will work.

Please let us know if anything doesn't work! As always, you can reach us on a@a3.pm or l@a3.pm via XMPP or email, or use our Discord Guild.

As always, thanks for your interest in our services, Ave

(Tags to jump to other related posts: #a3pm #ipv6)

To make LavaTech Bitwarden better for both you and us, we're moving to bitwarden_rs, a third party bitwarden server.

We're sure that you're all well aware of certain issues LavaTech Bitwarden faced, especially regarding downtimes and email issues.

Bitwarden itself had certain issues:

• It was written in .net core, used mssql etc, and was rather heavy for a small scale deployment like ours.
• It required the use of docker-compose and was made out of ~10 containers. This didn't play nicely with our Docker network, leading to occasional downtimes and email issues.
• It didn't allow accessing premium features or creating an organization without purchasing a license directly from Bitwarden, making self-hosting rather useless.
• This was able to be bypassed with use of projects such as BitBetter (https://github.com/jakeswenson/BitBetter). We did use this, but that still required us to manually generate licenses for people.

bitwarden_rs has premium features enabled for everyone, you can create an organization without purchasing a license or asking us for a license file.

It's all self contained in one docker container, and plays rather nicely with our network. We've also fixed the email issues for both instances (that was mostly due to our messy network tho, which is now also sorted out).

And according to our early tests, it performs much better than Bitwarden itself, both in terms of actual performance and stability. If you had issues with it before, we recommend trying it out again.

Existing users

As the database in the backend isn't compatible with bitwarden_rs, we need existing users that want to preserve their data to follow these instructions:

• Login to bw.lavatech.top on a browser.
• Go to Tools –> Export Vault, pick “.json”, enter your password, download your file.
• Go to bitwarden.lavatech.top on a browser, register a new account, verify email, log in.
• Go to Tools –> Import Data, pick “Bitwarden (json)”, select the file you downloaded, hit “Import Data”, and wait for it to finish. This process may take several minutes. There's a potential that it might time out and give an error, however it will still add everything in the background. Please do not try to add it again, it will just add duplicates.
• Log out on any other device, extension etc you use Lavatech Bitwarden on, update domain to bitwarden.lavatech.top and login again.

Our old Bitwarden instance will be shutting down in a month (May 18, 2020), or when we confirm that all users have migrated, whichever is sooner.

Thanks for your interest, LavaTech Team

(Tags to jump to other related posts: #bitwarden)

Update, April 2, 2020: Date of post was bolded.

Elixi.re and Ratelimited.me have announced today the planned merger of their operations to offer a unified file sharing solution to the market. Elixi.re will acquire the Ratelimited.me assets, community and hosting, and the merged business will operate under the ElixiLimited brand.

Elixi.re, founded in 2018, provides intelligent file sharing solutions that boost productivity, supercharge collaboration, and keep the organization connected. Its rich heritage in amusing domains and high uptime with long-standing partnerships with numerous other file sharing services provide a market leading proposition. Elixi.re is used by hundreds of users worldwide.

The expanded business will serve 1200+ users and 100.000+ files per day.

Elixi.re is a main File Hosting Services provider in Istanbul. It has been solely focused on offering file hosting since the year 2018. Since its inception, Elixi.re’s business and service offering have evolved to meet the unique demands of the dynamic and maturing image host market.

To be the leading image hosting provider, Elixi.re has made stability the central element to everything that it does.

More information about the software products it offers can be found at the website of LavaTech: https://lavatech.top

A spokesperson for Elixi.re shared that they expect definitive agreements to be completed by 2 April 2020 and the operations and assets of Ratelimited.me to be transferred to Elixi.re shortly thereafter.

Contact info: Name: Ave Ozkal Organization: LavaTech Address: Chynoweth House Trevissome Park Apt 30074 Truro TR4 8UN United Kingdom Phone: +1-240-621-1337

(Tags to jump to other related posts: #aprilfools)

However, in this quarter, we did have some significant changes.

This document was mostly written by Ave, but it was reviewed and approved by both LavaTech members.

Cold storage for backups

One of the main changes we made in this quarter was moving important old backups from hot storage to cold storage, and nuking unneeded backups.

In the end we removed around 50GB of files and moved around 800GB of files.

We use Backblaze B2 for hot storage. We chose Scaleway C14 Cold Storage for cold storage (I explained why here).

The move of backups is still not automated, but it'll be done when I find the time. Right now we use rclone to upload, here's our config.

I live tweeted our move, but here's an important note:

For 500GB: – B2 costs $2.5/mo – B2 download costs $5 – C14 Cold Storage costs EUR1/mo – C14 download costs EUR5 (if not to a server on scaleway itself, which let's be honest, very few will do that)

So, while C14 Cold Storage is ~2.5x cheaper to store, it costs the roughly the same to get data off of it.

For us, it'll make more sense to upload to B2 and C14 simultaneously, and then delete files off of B2 after a week.

It cost us roughly a month's worth of B2 fees just in download fees, and C14 does cost ~2.5x less as I stated above, so quick maths states that it'll pay itself off in a couple months.

Netbox

Until around mid 2019, we used to either informally exchange information about servers. In mid 2019, we created a private repo called infra under elixire group on Gitlab.com.

This was where we kept our issues about servers, and where we kept markdown files describing some stuff that may be handy to know about servers.

However, this was not standardized, and didn't cover everything. It was mostly just notes.

I did know that netbox existed as our friends at general programming have one, and I intended to set up one for ourselves, but I only really got time to do so a couple weeks ago.

So, I set up a netbox instance, and after Luna and me working on and off for weeks, we successfully completed adding everything. Everything being anything from cloud servers to home servers, containers to virtual machines (both were added as VMs), IPs to services have been added.

During this process, we identified numerous unused servers, and numerous anomalies (such as 10GB of ram on a container that just runs a webserver), which led to us nuking numerous VPSes and CTs.

Bye bye VPS: avefedi

avefedi was a server that was dedicated to hosting the Pleroma instance on p.a3.pm.

It previously hosted 90dns and they were both on same server, but server crashed randomly and I created a new server for 90dns as I worked with Scaleway to bring this machine up. This server came up after a day or two of downtime, and was dedicated to hosting the p.a3.pm Pleroma instance ever since. It's honestly overkill for just a Pleroma instance, which is why I moved it to a container.

• Provider: Scaleway (start1-xs)
• Ran between: December 1, 2018 – March 17, 2020 (1 year, 3 months, 16 days)
• Saved: 2EUR/mo

Bye bye VPS: demonhouse

demonhouse was a shared server that we gave access to close friends to host small stuff they needed to host. Everyone had root, and it was in use as late as October 2019.

• Provider: Hetzner (CX11)
• Ran between: October 10, 2018 – March 4, 2020 (1 year, 4 months, 25 days)
• Saved: 2.49EUR/mo

Bye bye VPS: einf

einf was a server that was dedicated to moving files between cloud providers, then ran a honeypot for a couple weeks.

• Provider: Hetzner (CX11)
• Ran between: January 21, 2020 – March 4, 2020 (1 month, 14 days)
• Saved: 2.49EUR/mo

Bye bye CTs: firefly, lasagnatube

firefly was a personal container hosting a Firefly III instance. Firefly III is an open source financial tracking software, one I tried to use to track my spending, but it quickly became a burden on me to keep it up to date, and i stopped using it.

lasagnatube was a container hosting LasagnaTube. While LasagnaTube is owned by Lasagna Ltd, it was hosted on LavaTech servers, which is why it's mentioned here. LasagnaTube files were moved into cold storage.

(Tags to jump to other related posts: #infrareview #report)

This is our first financial report to date, due to this, there won't be any data to compare to, but we hope that the data is useful regardless.

We also released a transparency report earlier this week, you can read it here.

This is released on 2020-02-27, with numbers as of this very date.

This document was mostly written by Ave, but it was reviewed and approved by both LavaTech members.

• This document was edited on 2020-03-12 to add KernelCare fees. Similarly, title was changed from 2020 to 2019, because calling it 2020 was a bad idea.

Income: Patreon

Patreon is our main source of donations for LavaTech.

• In 2019, we made $563 (minus taxes and fees) through Patreon.
• We had no refunds.
• $83.00 went to supporting other creators (I sadly cannot find a way to pull up a list of who they were, however as of November 2019, we aren't paying for our Patreon patronage through LavaTech).
• Starting in June, we started using Payoneer for server payments to be able to use a debit card.
• Before June, we were using Luna's Paypal, but we had to switch away due to Brazilian Paypal banning the use of balance internationally.
• Move to Payoneer introduced further fees.

After support for other creators, we were left with $480.

Expenses: Taxes and fees

• $36.76 went to VAT.
• $35.93 went to Patreon processing fees.
• $28.15 went to Patreon platform fees.
• $12.00 went to activating out Payoneer debit card.
• $11.62 went to Payoneer maintenance fees.
• $9.32 went to Payoneer (outbound) transaction fees.
• $6.00 went to Payoneer (inbound) transaction fees.

In total, $139.78 (29.12% of the amount that was left after support for other creators) went to taxes and fees in 2019.

Here's a fancy pie chart, because pie charts are good:

Expense: Backblaze B2

Backblaze B2 is our preferred data storage service of choice for backups. As our services and the data we store continue to grow, so does our backup bill. For 2020, I am considering moving old backups to cold storage to accommodate for this.

• $35.07 went to B2.
• While we paid $0.78 on January, we paid $5.17 on December.
• (Informational) We paid $6.43 on February 2020.

Expense: Cloudflare

Starting in July, we started hosting switchroot files.

I (ave) made an optimistic move and didn't set up proper RAM caching. Less than 10 seconds after we launched the image files, the server's drives were overloaded, and no one managed to download anything.

As a result, we distributed files to multiple servers and set up Cloudflare Load Balancing (which costs $5/mo), which solved the issue.

• $27.26 went to Cloudflare Load Balancing.
• Domain fees from Cloudflare is not included here.

Expense: Servers

Obviously, our services require servers. A lot of them, in fact. We have a lot of them for that very reason.

Hetzner is used for most of our server needs and hosts most of our services. Online.net hosts the mirrors, Scaleway hosts (European) 90dns.

All of our server payments are in Euros.

• €534.94 (~$585.15) went to Hetzner.
• Cheapest months on Hetzner were April to July, with each month costing €40.85.
• Most expensive months on Hetzner was December, costing €49.57.
• €143.88 (~$157.45) went to Online.net. All months were €11.99.
• €55.99 (~$61.26) went to Scaleway.

In total, we paid €734.81 (~$803.80) for servers in 2019.

Expense: Domains

And obviously, what good is an image host without domains?

Do note that these numbers aren't perfect, as lines somewhat blur between LavaTech domain costs (like elixi.re) and non-LavaTech domain costs (like ave.zone). I've however left out registrars that I only have non-LavaTech domains from, such as nictr (now metunic).

• $146.21 went to Porkbun.
• $27.46 went to Dynadot.
• $23.96 went to Namesilo (As of 2020, we are no longer working with Namesilo).
• $22.53 went to Namecheap.
• $8.03 went to Cloudflare.

In total, we paid $228.19 for domains in 2019.

Expense: HIBP

We had a HIBP API subscription for 3 months to be used with our bitwarden server, so we paid $10.5 (+$0.18 payoneer fees) to that.

Expense: KernelCare

We have KernelCare on our main hypervisor, edgebleed. We try to keep downtime to a minimum, and kcare helps with that.

We get a 2 server license, so we pay $2.95/mo per server, but as one of them is used by our friends at General Programming, it's not included in this specific report.

• $35.40 was spent for KernelCare in 2019 ($2.95/mo).

Personal Expenses through LavaTech funds: G Suite, Steam

These were expenses we made for ourselves, and were rare. We are trying to keep these to a minimum. Considering we pay a significant chunk of the LavaTech expenses from pocket, we hope that they'll be excused.

These numbers include payoneer fees.

• $5.97 was spent on Steam
• $20.32 was spent on G Suite

In total, we paid $26.29 for personal expenses through LavaTech funds in 2019.

In conclusion

• We made $563.
• We got to keep $340.22 of it after fees, taxes and support towards other creators.
• We spent $1140.22 (doesn't include personal expenses).
• After fees, taxes and support towards other creators, 29.83% of our expenses were paid by donations. Exactly $800 was paid out of pocket.

Also, another pie chart, this time showing expenses (somewhat outdated, doesn't include kernelcare):

We'd like to thank all of you for supporting us, by using our services, by recommending our services, and by donating domains and funds to our services.

Shameless plug: If you'd like to help make that percentage be higher for 2020, here's our patreon. Anything helps.

(Tags to jump to other related posts: #financialreport #report)