Installing Mastodon 4 on Rocky Linux 9

The from-source instructions assume Debian, but I'm a Redhat kid.

In the wake of the Twitter acquisition, a not-insubustantial number of tech folks are looking at Mastodon as an alternative. I want to try Mastodon and could totally go with a public server or pre-built server image, but the whole point here is that I can do my own federation and fully understand what the server is doing.

First, scoping. I don't upload a bunch of long-form media-heavy stuff, and I'm expecting somewhere between 2 and 5 users total, almost entirely personas that I'll control. I don't want to deal with storing media in an s3 bucket or data in a separate database, I'm just going to keep all state local for simplicity. That wouldn't scale up much beyond my 2-5 user intention.

Usually I'd drop a docker container on my house server but there aren't a lot of great examples that I could find. The official docs assume VM-as-a-container, which is entirely within my wheelhouse if a little old-school.

I'd love to follow the from-source installation directions but those are Ubuntu 20.04 specific. Instead, I'm going to use a Redhat derivative and update the instructions here instead.

Parts I'm not going to cover:

• Buying and setting up a domain. I'm a fan of gandi.net or godaddy.com for domain registration, in this case for lieurance.social
• Making a virtual machine. I put one in Linode, gave it 2 gig of ram, 50 gig of storage, 1 cpu. It's fine for me based on my scope above.
• Setting your domain to point an A and/or AAAA record to the IP of the VM. I'll leave that to your DNS hosting doc
• Installing the OS. Linode made that pretty easy. I picked Rocky Linux 9 as my respin of choice this time. I used CentOS for years, the Stream thing confuses me. You could totally spin up real Redhat or Oracle Linux if you want to. They're lovely also.
• SMTP setup, you'll want a way to send email with a submission host, username, and password.

These instructions assume you've got a pretty basic Rocky Linux 9 install that you've got root access to and a publicly routable IP with DNS set up on a domain you like. My examples below will be using lieurance.social. Modify the instructions for your domain accordingly.

Covering basics, this is a little bit of hardening that I like to do. You should absolutely do more also.

# Install EPEL
  dnf install -y epel-release
  /usr/bin/crb enable

# Install fail2ban  
# See [https://www.redhat.com/sysadmin/protect-systems-fail2ban](https://www.redhat.com/sysadmin/protect-systems-fail2ban) for how to configure it a little better
  dnf install -y fail2ban
  systemctl enable fail2ban
  systemctl start fail2ban

# set the hostname
  hostnamectl set-hostname lieurance.social

# Open up the inbound firewall to our expected web ports, and drop that silly management tool.
  firewall-cmd --remove-service=cockpit --permanent
  firewall-cmd --add-service=http --permanent
  firewall-cmd --add-service=https --permanent
  firewall-cmd --reload

Now let's get the mastodon install going.

# Make a low-privilege Mastodon user
  useradd mastodon
  echo "DenyUsers mastodon" >> /etc/ssh/sshd_config.d/20-no-mastodon.conf
  usermod -s /sbin/nologin mastodon
  systemctl reload sshd

Get to installing packages

# Needs Node16 and basic C stuff for compilation of native gems.  Fine.
  dnf install -y nodejs yarnpkg gcc g++ libicu-devel zlib-devel openssl-devel libidn-devel git

# It actually runs ruby.  I'm a fan of "system ruby" for single-purpose VMs, much like in docker containers.
# If you prefer to go the rbenv or rvm route, do that instead of this.
  dnf install -y ruby ruby-devel

# And some postgres
  dnf install -y postgresql postgresql-server libpq-devel
  
# And some redis for the cache
  dnf install -y redis

# Image and video things
  dnf install -y ImageMagick ffmpeg-free

# Network things
  dnf install -y certbot nginx python3-certbot-nginx

Let's turn on the database-y stuff we need. Because we're doing an all-in-one server, we can use local auth for postgres.

# Set up postgres
  /usr/bin/postgresql-setup --initdb
  systemctl enable postgresql
  systemctl start postgresql
  sudo -u postgres psql
  CREATE USER mastodon CREATEDB;
  exit

# Turn on redis
  systemctl enable redis
  systemctl start redis

Now let's install the app. The instructions run the thing literally out of a git checkout which is...fine.

During setup I came across this issue that notes a dependency on a gem that doesn't play well with OpenSSL 3. Until that gets sorted out, I had to tell nodejs to work around it.

This is also where you'll need things like the SMTP auth credentials, other cloud credentials if you need them, and knowledge of what user you want to be the default admin.

# Become the low-privilege user
  su - mastodon -s /bin/bash

# Get the repo downloaded and set up, grabbing the last tagged release version.
  git clone https://github.com/mastodon/mastodon.git live && cd live
  git checkout $(git tag -l | grep -v 'rc[0-9]*$' | sort -V | tail -n 1)
  bundle config deployment 'true'
  bundle config without 'development test'
  
# Actually do the gem compilation and install tasks
  bundle install -j$(getconf _NPROCESSORS_ONLN)
  yarn install --pure-lockfile

# Javascript asset compilation
  export NODE_OPTIONS=--openssl-legacy-provider
  RAILS_ENV=production bundle exec rake mastodon:setup

# Get back to root
  exit

This took forever to compile the assets on my tiny box, but it eventually succeeded. If you get something wrong and want to start over, in order to get it to drop the unused db, you'll want to do export DISABLE_DATABASE_ENVIRONMENT_CHECK=1 before the rake script again.

At this point, you've hopefully got the app up and configured. Now to turn it on and connect it to the public Internet.

As root again.

# Copy up the nginx configs to get it attached to the network
  cp /home/mastodon/live/dist/nginx.conf /etc/nginx/conf.d/mastodon.conf
  sed -i 's/example.com/lieurance.social/' /etc/nginx/conf.d/mastodon.conf

Then, comment out the 443 server and get your SSL cert created. Certbot is basically magic, but this is the only part of the directions where I hand-edited some files. Then uncomment the 443 server, replace the SSL instructions in there with the block that certbot added, and restart

# LetsEncrypt SSL
  vi /etc/nginx/conf.d/mastodon.conf
  systemctl restart nginx
  certbot --nginx -d lieurance.social
  vi /etc/nginx/conf.d/mastodon.conf
  systemctl enable nginx
  systemctl restart nginx

Then, get the process management and startup-after-reboot stuff running. Different from the managed instructions, we didn't compile our own ruby with jemalloc so there's no need to do a library linking dance.

cp /home/mastodon/live/dist/mastodon-*.service /etc/systemd/system/
sed -i '/LD_PRELOAD.*jemalloc/d' /etc/systemd/system/mastodon-*.service
sed -i 's|=.*bundle|=/usr/bin/bundle|' /etc/systemd/system/mastodon-*.service
systemctl daemon-reload
systemctl enable --now mastodon-web mastodon-sidekiq mastodon-streaming

If you're following along with the instructions, it should work!

In normal ops fashion it totally didn't. Trying to open the page gave a 502 Bad Gateway out of nginx. To the logs!

Looking at /var/log/nginx/error.log I see some lines like

2022/11/23 07:10:58 [crit] 32643#32643: *87 stat() "/home/mastodon/live/public/favicon.ico" failed (13: Permission denied), client: 2603:300a::170f, server: lieurance.social, request: "GET /favicon.ico HTTP/2.0", host: "lieurance.social", referrer: "https://lieurance.social/"
2022/11/23 07:10:58 [crit] 32643#32643: *87 connect() to 127.0.0.1:3000 failed (13: Permission denied) while connecting to upstream, client: 2603:300a::170f, server: lieurance.social, request: "GET /favicon.ico HTTP/2.0", upstream: "http://127.0.0.1:3000/favicon.ico", host: "lieurance.social", referrer: "https://lieurance.social/"

That (13: Permission denied) is the key telling us what error it is (13 == EACCES). The stat() call is happening trying to look at a file in a home directory. That doesn't work in modern linux without some fiddling.

The connect() call is a little more subtle and stranger to think about. It's using nginx as a reverse proxy which is reasonably modern but not the default for a locked-down OS like our Redhat derivatives. Again picking EACCES as the error message is weird since there's no normal file actually being accessed, but that's also the error code that selinux gives when it blockes this kernel function.

So we need to tell the system to let this work happen. First, basic user and group stuff. We've got to let the user that nginx runs as have filesystem access to the parts of the mastodon home directory where the javascript assets live. I'm giving full access to that home directory which isn't ideal but will survive upgrades way easier.

# Give the nginx user access to the mastodon group, to get through file system ACLs
  usermod -a -G mastodon nginx
# Give the mastodon group access to the mastodon home directory
  chmod g+rx ~mastodon

Then, because we live in the present, selinux will stop the web server by default from reading from home directories. That's intentional and it super surpises me that Ubuntu lets you do that by default. Wild. I found that by looking in the selinux log at /var/log/audit/audit.log and letting audit2why tell me what was going on with the entries. More documentation is available at nginx's official selinux page.

# Tell selinux that nginx is a reverse proxy and that's ok
  sesetbool -P httpd_can_network_connect 1

# Tell selinux that nginx should be allowed to read from home directories
  sesetbool -P httpd_read_user_content 1
  sesetbool -P httpd_enable_homedirs 1

That was it! Now, https://lieurance.social is up, I have a login with a randomly generated password, and can start managing my Mastodon instance.

Extra credit for making this collection of shell snippits into a more manageable block of Ansible or Puppet or Chef or Salt or...

Updated 2022-12-31 with suggestions from @walter@selent.ca on Mastodon. Thanks!