LavaTech Blog

Official blog of LavaTech

Hello everyone welcome to my quick guide on how to change password in Conversations:

0) Open Conversations, go to the main view, hit the 3 dots on top right

1) Go to Manage accounts

2) Pick your account

3) Hit the 3 dots on top right, smash that “Change password” button

4) Change your password by putting your current password to the first box and the new one to the second, and hitting the “Change Password” button

Thank you for reading my guide.

(Tags to jump to other related posts: #guide, #a3pm)

Hello everyone, a3.pm XMPP Services had several (unannounced) downtimes today as I did some changes to a3.pm.

  • I've updated the server from ejabberd 20.03 to 20.04, released 4 days ago.
  • I've enabled STUN/TURN to enable Video/Audio calls. This is somewhat big. Here's the relevant issue, and a number of links explaining this. You may use this feature with Conversations. I haven't tested it yet, however if you do manage to try it out, please let me know if it works or not on email or XMPP a[email protected]
  • I synced the configuration on the server to the one on the repository. The actual settings were the same, but locations of them weren't, and they missed notes etc from each other. I've synced these, mostly from the repo to the server. You can find the relevant commit here.
  • I've open sourced our MAM wiping systemd service/timer.
  • We've updated the SSL certificate. This is a downtime we have to face roughly every 80 days.

Thanks as always for using our services, Ave

(Tags to jump to other related posts: #a3pm)

Hey all,

As title says, we used to be a Manjaro ARM mirror for several months, but exactly 4 weeks ago, it became a part of Manjaro's regular mirrors. This was rather big and honestly amazing news for Manjaro ARM.

It did however mean that our mirror would be deprecated.

We thought of becoming a regular mirror, but couldn't find easy resources on how to become an official one quickly, and I (ave) was quite busy at the time, so it got kind of ignored.

Well, that changes today! We're now an official Manjaro mirror. Well, it was ready yesterday, but it got approved today, and we only switched it to the manjaro.org upstream today (that's the equivalent of a T2 Arch repo going to T1).

You shouldn't need to do much and it should get activated automatically based on ping, but if you want to add it manually, our mirror is at https://manjaro.mirrors.lavatech.top/

Signed, LavaTech Team

(Tags to jump to other related posts: #manjaro #mirror)

Hello everyone,

I've just enabled experimental IPv6 support on a3.pm XMPP services.

By that, I mean that I've:

  • Added AAAA records
  • Tested basic XMPP functionality (successful connection, fetching message history, sending messages) with an IPv6-only connection (which succeeded)
  • Updated nginx to also serve on IPv6 on port 80
  • Updated sslh to also serve on IPv6 on port 443

One thing that I haven't tested that has a chance of failing is HTTP file uploads/fetches with IPv6, though it very likely will work.

Please let us know if anything doesn't work! As always, you can reach us on [email protected] or [email protected] via XMPP or email, or use our Discord Guild.

As always, thanks for your interest in our services, Ave

(Tags to jump to other related posts: #a3pm #ipv6)

Hello everyone,

To make LavaTech Bitwarden better for both you and us, we're moving to bitwarden_rs, a third party bitwarden server.

We're sure that you're all well aware of certain issues LavaTech Bitwarden faced, especially regarding downtimes and email issues.

Bitwarden itself had certain issues:

  • It was written in .net core, used mssql etc, and was rather heavy for a small scale deployment like ours.
  • It required the use of docker-compose and was made out of ~10 containers. This didn't play nicely with our Docker network, leading to occasional downtimes and email issues.
  • It didn't allow accessing premium features or creating an organization without purchasing a license directly from Bitwarden, making self-hosting rather useless.
  • This was able to be bypassed with use of projects such as BitBetter (https://github.com/jakeswenson/BitBetter). We did use this, but that still required us to manually generate licenses for people.

bitwarden_rs has premium features enabled for everyone, you can create an organization without purchasing a license or asking us for a license file.

It's all self contained in one docker container, and plays rather nicely with our network. We've also fixed the email issues for both instances (that was mostly due to our messy network tho, which is now also sorted out).

And according to our early tests, it performs much better than Bitwarden itself, both in terms of actual performance and stability. If you had issues with it before, we recommend trying it out again.

Existing users

As the database in the backend isn't compatible with bitwarden_rs, we need existing users that want to preserve their data to follow these instructions:

  • Login to bw.lavatech.top on a browser.
  • Go to Tools –> Export Vault, pick “.json”, enter your password, download your file.
  • Go to bitwarden.lavatech.top on a browser, register a new account, verify email, log in.
  • Go to Tools –> Import Data, pick “Bitwarden (json)”, select the file you downloaded, hit “Import Data”, and wait for it to finish. This process may take several minutes. There's a potential that it might time out and give an error, however it will still add everything in the background. Please do not try to add it again, it will just add duplicates.
  • Log out on any other device, extension etc you use Lavatech Bitwarden on, update domain to bitwarden.lavatech.top and login again.

Our old Bitwarden instance will be shutting down in a month (May 18, 2020), or when we confirm that all users have migrated, whichever is sooner.

Thanks for your interest, LavaTech Team

(Tags to jump to other related posts: #bitwarden)

Istanbul, TR, Release: April 1, 2020. For Immediate Release.

Update, April 2, 2020: Date of post was bolded.

Elixi.re and Ratelimited.me have announced today the planned merger of their operations to offer a unified file sharing solution to the market. Elixi.re will acquire the Ratelimited.me assets, community and hosting, and the merged business will operate under the ElixiLimited brand.

ElixiLimited Logo

Elixi.re, founded in 2018, provides intelligent file sharing solutions that boost productivity, supercharge collaboration, and keep the organization connected. Its rich heritage in amusing domains and high uptime with long-standing partnerships with numerous other file sharing services provide a market leading proposition. Elixi.re is used by hundreds of users worldwide.

The expanded business will serve 1200+ users and 100.000+ files per day.

Elixi.re is a main File Hosting Services provider in Istanbul. It has been solely focused on offering file hosting since the year 2018. Since its inception, Elixi.re’s business and service offering have evolved to meet the unique demands of the dynamic and maturing image host market.

To be the leading image hosting provider, Elixi.re has made stability the central element to everything that it does.

More information about the software products it offers can be found at the website of LavaTech: https://lavatech.top

A spokesperson for Elixi.re shared that they expect definitive agreements to be completed by 2 April 2020 and the operations and assets of Ratelimited.me to be transferred to Elixi.re shortly thereafter.

Contact info: Name: Ave Ozkal Organization: LavaTech Address: Chynoweth House Trevissome Park Apt 30074 Truro TR4 8UN United Kingdom Phone: +1-240-621-1337

(Tags to jump to other related posts: #aprilfools)

Hey guys, this is our first ever LavaTech Infrastructure Review. This is a set of posts that we aim to do every quarter, even if there's no change.

However, in this quarter, we did have some significant changes.

This document was mostly written by Ave, but it was reviewed and approved by both LavaTech members.

Cold storage for backups

One of the main changes we made in this quarter was moving important old backups from hot storage to cold storage, and nuking unneeded backups.

In the end we removed around 50GB of files and moved around 800GB of files.

We use Backblaze B2 for hot storage. We chose Scaleway C14 Cold Storage for cold storage (I explained why here).

The move of backups is still not automated, but it'll be done when I find the time. Right now we use rclone to upload, here's our config.

I live tweeted our move, but here's an important note:

For 500GB: – B2 costs $2.5/mo – B2 download costs $5 – C14 Cold Storage costs EUR1/mo – C14 download costs EUR5 (if not to a server on scaleway itself, which let's be honest, very few will do that)

So, while C14 Cold Storage is ~2.5x cheaper to store, it costs the roughly the same to get data off of it.

For us, it'll make more sense to upload to B2 and C14 simultaneously, and then delete files off of B2 after a week.

It cost us roughly a month's worth of B2 fees just in download fees, and C14 does cost ~2.5x less as I stated above, so quick maths states that it'll pay itself off in a couple months.

Netbox

Until around mid 2019, we used to either informally exchange information about servers. In mid 2019, we created a private repo called infra under elixire group on Gitlab.com.

This was where we kept our issues about servers, and where we kept markdown files describing some stuff that may be handy to know about servers.

However, this was not standardized, and didn't cover everything. It was mostly just notes.

I did know that netbox existed as our friends at general programming have one, and I intended to set up one for ourselves, but I only really got time to do so a couple weeks ago.

So, I set up a netbox instance, and after Luna and me working on and off for weeks, we successfully completed adding everything. Everything being anything from cloud servers to home servers, containers to virtual machines (both were added as VMs), IPs to services have been added.

During this process, we identified numerous unused servers, and numerous anomalies (such as 10GB of ram on a container that just runs a webserver), which led to us nuking numerous VPSes and CTs.

Bye bye VPS: avefedi

avefedi was a server that was dedicated to hosting the Pleroma instance on p.a3.pm.

It previously hosted 90dns and they were both on same server, but server crashed randomly and I created a new server for 90dns as I worked with Scaleway to bring this machine up. This server came up after a day or two of downtime, and was dedicated to hosting the p.a3.pm Pleroma instance ever since. It's honestly overkill for just a Pleroma instance, which is why I moved it to a container.

  • Provider: Scaleway (start1-xs)
  • Ran between: December 1, 2018 – March 17, 2020 (1 year, 3 months, 16 days)
  • Saved: 2EUR/mo

Bye bye VPS: demonhouse

demonhouse was a shared server that we gave access to close friends to host small stuff they needed to host. Everyone had root, and it was in use as late as October 2019.

  • Provider: Hetzner (CX11)
  • Ran between: October 10, 2018 – March 4, 2020 (1 year, 4 months, 25 days)
  • Saved: 2.49EUR/mo

Bye bye VPS: einf

einf was a server that was dedicated to moving files between cloud providers, then ran a honeypot for a couple weeks.

  • Provider: Hetzner (CX11)
  • Ran between: January 21, 2020 – March 4, 2020 (1 month, 14 days)
  • Saved: 2.49EUR/mo

Bye bye CTs: firefly, lasagnatube

firefly was a personal container hosting a Firefly III instance. Firefly III is an open source financial tracking software, one I tried to use to track my spending, but it quickly became a burden on me to keep it up to date, and i stopped using it.

lasagnatube was a container hosting LasagnaTube. While LasagnaTube is owned by Lasagna Ltd, it was hosted on LavaTech servers, which is why it's mentioned here. LasagnaTube files were moved into cold storage.

(Tags to jump to other related posts: #infrareview #report)

What's happening forum?

This is our first financial report to date, due to this, there won't be any data to compare to, but we hope that the data is useful regardless.

We also released a transparency report earlier this week, you can read it here.

This is released on 2020-02-27, with numbers as of this very date.

This document was mostly written by Ave, but it was reviewed and approved by both LavaTech members.

  • This document was edited on 2020-03-12 to add KernelCare fees. Similarly, title was changed from 2020 to 2019, because calling it 2020 was a bad idea.

Income: Patreon

Patreon is our main source of donations for LavaTech.

Graph of Patreon income

  • In 2019, we made $563 (minus taxes and fees) through Patreon.
  • We had no refunds.
  • $83.00 went to supporting other creators (I sadly cannot find a way to pull up a list of who they were, however as of November 2019, we aren't paying for our Patreon patronage through LavaTech).
  • Starting in June, we started using Payoneer for server payments to be able to use a debit card.
  • Before June, we were using Luna's Paypal, but we had to switch away due to Brazilian Paypal banning the use of balance internationally.
  • Move to Payoneer introduced further fees.

After support for other creators, we were left with $480.

Expenses: Taxes and fees

  • $36.76 went to VAT.
  • $35.93 went to Patreon processing fees.
  • $28.15 went to Patreon platform fees.
  • $12.00 went to activating out Payoneer debit card.
  • $11.62 went to Payoneer maintenance fees.
  • $9.32 went to Payoneer (outbound) transaction fees.
  • $6.00 went to Payoneer (inbound) transaction fees.

In total, $139.78 (29.12% of the amount that was left after support for other creators) went to taxes and fees in 2019.

Here's a fancy pie chart, because pie charts are good:

Pie chart showing distribution of taxes and feses

Expense: Backblaze B2

Backblaze B2 is our preferred data storage service of choice for backups. As our services and the data we store continue to grow, so does our backup bill. For 2020, I am considering moving old backups to cold storage to accommodate for this.

  • $35.07 went to B2.
  • While we paid $0.78 on January, we paid $5.17 on December.
  • (Informational) We paid $6.43 on February 2020.

Expense: Cloudflare

Starting in July, we started hosting switchroot files.

I (ave) made an optimistic move and didn't set up proper RAM caching. Less than 10 seconds after we launched the image files, the server's drives were overloaded, and no one managed to download anything.

As a result, we distributed files to multiple servers and set up Cloudflare Load Balancing (which costs $5/mo), which solved the issue.

  • $27.26 went to Cloudflare Load Balancing.
  • Domain fees from Cloudflare is not included here.

Expense: Servers

Obviously, our services require servers. A lot of them, in fact. We have a lot of them for that very reason.

Hetzner is used for most of our server needs and hosts most of our services. Online.net hosts the mirrors, Scaleway hosts (European) 90dns.

All of our server payments are in Euros.

  • €534.94 (~$585.15) went to Hetzner.
  • Cheapest months on Hetzner were April to July, with each month costing €40.85.
  • Most expensive months on Hetzner was December, costing €49.57.
  • €143.88 (~$157.45) went to Online.net. All months were €11.99.
  • €55.99 (~$61.26) went to Scaleway.

In total, we paid €734.81 (~$803.80) for servers in 2019.

Expense: Domains

And obviously, what good is an image host without domains?

Do note that these numbers aren't perfect, as lines somewhat blur between LavaTech domain costs (like elixi.re) and non-LavaTech domain costs (like ave.zone). I've however left out registrars that I only have non-LavaTech domains from, such as nictr (now metunic).

In total, we paid $228.19 for domains in 2019.

Expense: HIBP

We had a HIBP API subscription for 3 months to be used with our bitwarden server, so we paid $10.5 (+$0.18 payoneer fees) to that.

Expense: KernelCare

We have KernelCare on our main hypervisor, edgebleed. We try to keep downtime to a minimum, and kcare helps with that.

We get a 2 server license, so we pay $2.95/mo per server, but as one of them is used by our friends at General Programming, it's not included in this specific report.

  • $35.40 was spent for KernelCare in 2019 ($2.95/mo).

Personal Expenses through LavaTech funds: G Suite, Steam

These were expenses we made for ourselves, and were rare. We are trying to keep these to a minimum. Considering we pay a significant chunk of the LavaTech expenses from pocket, we hope that they'll be excused.

These numbers include payoneer fees.

  • $5.97 was spent on Steam
  • $20.32 was spent on G Suite

In total, we paid $26.29 for personal expenses through LavaTech funds in 2019.

In conclusion

  • We made $563.
  • We got to keep $340.22 of it after fees, taxes and support towards other creators.
  • We spent $1140.22 (doesn't include personal expenses).
  • After fees, taxes and support towards other creators, 29.83% of our expenses were paid by donations. Exactly $800 was paid out of pocket.

Also, another pie chart, this time showing expenses (somewhat outdated, doesn't include kernelcare):

Pie chart showing distribution of expenses

We'd like to thank all of you for supporting us, by using our services, by recommending our services, and by donating domains and funds to our services.

Shameless plug: If you'd like to help make that percentage be higher for 2020, here's our patreon. Anything helps.

(Tags to jump to other related posts: #financialreport #report)

What's happening forum?

This is our first transparency report to date, due to this, there won't be any data to compare to, but we hope that the data is useful regardless.

We are also considering releasing a financial report, including server costs and income from patreon.

We also released a financial report, you can read it here.

This is released on 2020-02-25, with numbers as of this very date.

  • This document was edited on 2020-03-12 to correct the phrase “We haven't received any warrant canaries” to “We haven't received any warrants”. Similarly, title was changed from 2020 to 2019, because calling it 2020 was a bad idea.

In general...

We haven't received any warrants or anything like that to date.

No DMCAs in any service in 2019.

No C&Ds in any service in 2019.

elixi.re

elixi.re is continuing to be a thing as similar services go up and down. 2019 saw the death of bisoga, and as we're writing this, ratelimited.me is down due to a drive failure.

We're at 60 domains, 7 of them are currently without any users. We've switched to votes for domains in 2019 to remedy this issue, and it seems to work well so far. (only domains that ended up without users were shitpost domains I bought and added on impulse without voting, oops -ave)

In 2019, we did a server upgrade which resulted in less drive space but a lot better compute capabilities. Right now the storage space isn't a major concern.

v3 is under development. ETA: Soon™.

Right now, there's 606 active and 293 inactive users.

There's 473480 files (equaling 115503.07MiB), with 5333 of them from last week (equaling 1361.02MiB).

There's 1923 shortens, with 27 of them from the last week.

a3.pm XMPP services

User count is 1042.

2 people were banned for violating the LSA, one for commercial activity, one for being a “Problematic Person”.

We have shut down registrations for the foreseeable future to allow for moderation to be possible. Our server hardware is also struggling, especially postgresql during ejabberd's boot, and this also contributed to the registration being closed. if you want to help us diagnose this issue, feel free to contact us!

Over the year we've followed the latest updates, and we're currently running latest upstream. Similarly, we've continued to keep our config files open to encourage people to deploy XMPP servers.

90dns

90dns is going strong. There's ~20k unique IPs per day, 4k of these in the US instance.

Gitdab

Gitdab is growing steadily, as it is open to register and receives updates fairly soon after upstream. And god, the dab meme will never die, will it?

  • 323 users
  • 61 organizations
  • 153 repositories
  • 70 teams

Mirrors

We now have an official Manjaro ARM mirror at https://manjaroarm.mirrors.lavatech.top.

To fit with the URL scheme, https://archmirror.lavatech.top was moved to https://arch.mirrors.lavatech.top, and it now has proper, working IPv6.

BSNX

It's down, it's been down, and it doesn't seem like I'll have the energy to fix it soon. The split within the scene of people who are supportive of such services and people who are against it does make this harder, as while this is an easy technical feat, it's definitely not easy from a bureaucracy perspective.

LavaDNS

Reminder: LavaDNS is not 90dns.

LavaDNS is relatively deprecated. The future of it is uncertain, maybe it'll turn out to be unbound, or still experimental handwritten servers.

LavaSearx

LavaSearx was kept maintained for quite a while, but it's rather slow as it uses Searx. (and I'll be honest: I moved back to DDG. -ave)

The service is still up, but it's at a degraded state. It's not synced to upstream, and ddg seems to be broken. Similarly, Google banned our IP, heh.

(Tags to jump to other related posts: #transparencyreport #report)