Installing Mastodon 4 on Rocky Linux 92022-11-22 | devops
The from-source instructions assume Debian, but I'm a Redhat kid.
In the wake of the Twitter acquisition, a not-insubustantial number of tech folks are looking at Mastodon as an alternative. I want to try Mastodon and could totally go with a public server or pre-built server image, but the whole point here is that I can do my own federation and fully understand what the server is doing.
First, scoping. I don't upload a bunch of long-form media-heavy stuff, and I'm expecting somewhere between 2 and 5 users total, almost entirely personas that I'll control. I don't want to deal with storing media in an s3 bucket or data in a separate database, I'm just going to keep all state local for simplicity. That wouldn't scale up much beyond my 2-5 user intention.
Usually I'd drop a docker container on my house server but there aren't a lot of great examples that I could find. The official docs assume VM-as-a-container, which is entirely within my wheelhouse if a little old-school.
I'd love to follow the from-source installation directions but those are Ubuntu 20.04 specific. Instead, I'm going to use a Redhat derivative and update the instructions here instead.
Parts I'm not going to cover:
- Buying and setting up a domain. I'm a fan of gandi.net or godaddy.com for domain registration, in this case for lieurance.social
- Making a virtual machine. I put one in Linode, gave it 2 gig of ram, 50 gig of storage, 1 cpu. It's fine for me based on my scope above.
- Setting your domain to point an A and/or AAAA record to the IP of the VM. I'll leave that to your DNS hosting doc
- Installing the OS. Linode made that pretty easy. I picked Rocky Linux 9 as my respin of choice this time. I used CentOS for years, the Stream thing confuses me. You could totally spin up real Redhat or Oracle Linux if you want to. They're lovely also.
- SMTP setup, you'll want a way to send email with a submission host, username, and password.
These instructions assume you've got a pretty basic Rocky Linux 9 install that you've got root access to and a publicly routable IP with DNS set up on a domain you like. My examples below will be using lieurance.social. Modify the instructions for your domain accordingly.
Covering basics, this is a little bit of hardening that I like to do. You should absolutely do more also.
# Install EPEL dnf install -y epel-release /usr/bin/crb enable # Install fail2ban # See [https://www.redhat.com/sysadmin/protect-systems-fail2ban](https://www.redhat.com/sysadmin/protect-systems-fail2ban) for how to configure it a little better dnf install -y fail2ban systemctl enable fail2ban systemctl start fail2ban # set the hostname hostnamectl set-hostname lieurance.social # Open up the inbound firewall to our expected web ports, and drop that silly management tool. firewall-cmd --remove-service=cockpit --permanent firewall-cmd --add-service=http --permanent firewall-cmd --add-service=https --permanent firewall-cmd --reload
Now let's get the mastodon install going.
# Make a low-privilege Mastodon user useradd mastodon echo "DenyUsers mastodon" >> /etc/ssh/sshd_config.d/20-no-mastodon.conf usermod -s /sbin/nologin mastodon systemctl reload sshd
Get to installing packages
# Needs Node16 and basic C stuff for compilation of native gems. Fine. dnf install -y nodejs yarnpkg gcc g++ libicu-devel zlib-devel openssl-devel libidn-devel git # It actually runs ruby. I'm a fan of "system ruby" for single-purpose VMs, much like in docker containers. # If you prefer to go the rbenv or rvm route, do that instead of this. dnf install -y ruby ruby-devel # And some postgres dnf install -y postgresql postgresql-server libpq-devel # And some redis for the cache dnf install -y redis # Image and video things dnf install -y ImageMagick ffmpeg-free # Network things dnf install -y certbot nginx python3-certbot-nginx
Let's turn on the database-y stuff we need. Because we're doing an all-in-one server, we can use local auth for postgres.
# Set up postgres /usr/bin/postgresql-setup --initdb systemctl enable postgresql systemctl start postgresql sudo -u postgres psql CREATE USER mastodon CREATEDB; exit # Turn on redis systemctl enable redis systemctl start redis
Now let's install the app. The instructions run the thing literally out of a git checkout which is...fine.
During setup I came across this issue that notes a dependency on a gem that doesn't play well with OpenSSL 3. Until that gets sorted out, I had to tell nodejs to work around it.
This is also where you'll need things like the SMTP auth credentials, other cloud credentials if you need them, and knowledge of what user you want to be the default admin.
This took forever to compile the assets on my tiny box, but it eventually succeeded.
If you get something wrong and want to start over, in order to get it to drop the unused db, you'll want to do
export DISABLE_DATABASE_ENVIRONMENT_CHECK=1 before the rake script again.
At this point, you've hopefully got the app up and configured. Now to turn it on and connect it to the public Internet.
As root again.
# Copy up the nginx configs to get it attached to the network cp /home/mastodon/live/dist/nginx.conf /etc/nginx/conf.d/mastodon.conf sed -i 's/example.com/lieurance.social/' /etc/nginx/conf.d/mastodon.conf
Then, comment out the 443 server and get your SSL cert created. Certbot is basically magic, but this is the only part of the directions where I hand-edited some files. Then uncomment the 443 server, replace the SSL instructions in there with the block that certbot added, and restart
# LetsEncrypt SSL vi /etc/nginx/conf.d/mastodon.conf systemctl restart nginx certbot --nginx -d lieurance.social vi /etc/nginx/conf.d/mastodon.conf systemctl enable nginx systemctl restart nginx
Then, get the process management and startup-after-reboot stuff running. Different from the managed instructions, we didn't compile our own ruby with jemalloc so there's no need to do a library linking dance.
cp /home/mastodon/live/dist/mastodon-*.service /etc/systemd/system/ sed -i '/LD_PRELOAD.*jemalloc/d' /etc/systemd/system/mastodon-*.service sed -i 's|=.*bundle|=/usr/bin/bundle|' /etc/systemd/system/mastodon-*.service systemctl daemon-reload systemctl enable --now mastodon-web mastodon-sidekiq mastodon-streaming
If you're following along with the instructions, it should work!
In normal ops fashion it totally didn't. Trying to open the page gave a
502 Bad Gateway out of nginx. To the logs!
/var/log/nginx/error.log I see some lines like
2022/11/23 07:10:58 [crit] 32643#32643: *87 stat() "/home/mastodon/live/public/favicon.ico" failed (13: Permission denied), client: 2603:300a::170f, server: lieurance.social, request: "GET /favicon.ico HTTP/2.0", host: "lieurance.social", referrer: "https://lieurance.social/" 2022/11/23 07:10:58 [crit] 32643#32643: *87 connect() to 127.0.0.1:3000 failed (13: Permission denied) while connecting to upstream, client: 2603:300a::170f, server: lieurance.social, request: "GET /favicon.ico HTTP/2.0", upstream: "http://127.0.0.1:3000/favicon.ico", host: "lieurance.social", referrer: "https://lieurance.social/"
(13: Permission denied) is the key telling us what error it is (13 == EACCES). The
stat() call is happening trying to look at a file in a home directory. That doesn't work in modern linux without some fiddling.
connect() call is a little more subtle and stranger to think about. It's using nginx as a reverse proxy which is reasonably modern but not the default for a locked-down OS like our Redhat derivatives. Again picking EACCES as the error message is weird since there's no normal file actually being accessed, but that's also the error code that selinux gives when it blockes this kernel function.
So we need to tell the system to let this work happen. First, basic user and group stuff. We've got to let the user that
# Give the nginx user access to the mastodon group, to get through file system ACLs usermod -a -G mastodon nginx # Give the mastodon group access to the mastodon home directory chmod g+rx ~mastodon
Then, because we live in the present, selinux will stop the web server by default from reading from home directories. That's intentional and it super surpises me that Ubuntu lets you do that by default. Wild. I found that by looking in the selinux log at
/var/log/audit/audit.log and letting
audit2why tell me what was going on with the entries. More documentation is available at nginx's official selinux page.
# Tell selinux that nginx is a reverse proxy and that's ok sesetbool -P httpd_can_network_connect 1 # Tell selinux that nginx should be allowed to read from home directories sesetbool -P httpd_read_user_content 1 sesetbool -P httpd_enable_homedirs 1
That was it! Now, https://lieurance.social is up, I have a login with a randomly generated password, and can start managing my Mastodon instance.
Extra credit for making this collection of shell snippits into a more manageable block of Ansible or Puppet or Chef or Salt or...
Updated 2022-12-31 with suggestions from @firstname.lastname@example.org on Mastodon. Thanks!