Installing the Mastodon Software
December 15, 2022
Back to Self-Hosted Mastodon Project Description
Mastodon Installation
I mainly just planned on following the published instructions from https://docs.joinmastodon.org/admin/install/
I'm taking these notes as I run through the process to try and document what I did and any adjustments that had to be made.
Preparation
https://docs.joinmastodon.org/admin/prerequisites/
I did a bad thing and kept password-based SSH login. Will switch to public keys eventually.
Installed fail2ban and setup the local jail
Installed iptables-persistent and followed the suggested setup instructions from https://docs.joinmastodon.org/admin/prerequisites/#install-a-firewall-and-only-allow-ssh-http-and-https-ports
It allows some basic things:
- loopback traffic
- outbound traffic
- HTTP and HPTTPS connections
- SSH connections
- ping
and then rejects just about everything else.
Installing Pre-Requisites
Basic system packages
sudo apt-get install -y curl wget gnupg apt-transport-https lsb-release ca-certificatesNode.js
I suspect that I could probably use Node 18, but I'm going to stick with the published instructions which (as of this writing) use Node 16
curl -sL https://deb.nodesource.com/setup_16.x | bash -Don't forget to make sure that you're root when doing that!
I had to do a sudo apt-get install -y nodejs after this
PostgreSQL
wget -O /usr/share/keyrings/postgresql.asc https://www.postgresql.org/media/keys/ACCC4CF8.asc
echo "deb [signed-by=/usr/share/keyrings/postgresql.asc] http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/postgresql.listMore required system packages
apt install -y \
imagemagick ffmpeg libpq-dev libxml2-dev libxslt1-dev file git-core \
g++ libprotobuf-dev protobuf-compiler pkg-config nodejs gcc autoconf \
bison build-essential libssl-dev libyaml-dev libreadline6-dev \
zlib1g-dev libncurses5-dev libffi-dev libgdbm-dev \
nginx redis-server redis-tools postgresql postgresql-contrib \
certbot python3-certbot-nginx libidn11-dev libicu-dev libjemalloc-devYarn
This next step (as written online) made it seem like Yarn should have already been installed along with Node. This did not happen, but I realized it was because I wasn't root (again)
Gotta su - root for this. Got it.
Installing Ruby
I created a mastodon user, per the instructions
adduser --disabled-login mastodonand switched to that user
su - mastodonRuby version managers
The posted instructions call for using rbenv and rbenv-build
I have rvm on my local system, so decided to use that instead for some more consistency. I'm hoping that it won't make a difference for anything later on down the road.
Instructions: https://rvm.io/rvm/install
First tried to install the GPG keys:
mastodon@huxley:~$ gpg --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
gpg: directory '/home/mastodon/.gnupg' created
gpg: keybox '/home/mastodon/.gnupg/pubring.kbx' created
gpg: keyserver receive failed: Server indicated a failureI was able to successfully receive the keys by trying a different keyserver (as suggested here: https://rvm.io/rvm/security)
gpg --keyserver hkp://pgp.mit.edu --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDBI could then install using the RVM installation script:
curl -sSL https://get.rvm.io | bash -s stable --ruby
This failed, because I was running the script as mastodon and it wanted to be able to sudo - and of course this account isn't set up to sudo.
So I dropped back into my own user account, and attempted grabbing the keys and installing. I think that this should work - so long as RVM correctly adds ruby to the PATH, which I'm hoping it will.
bpettis@huxley:~$ gpg --keyserver hkp://pgp.mit.edu --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
gpg: directory '/home/bpettis/.gnupg' created
gpg: keybox '/home/bpettis/.gnupg/pubring.kbx' created
gpg: key 105BD0E739499BDB: 2 duplicate signatures removed
gpg: /home/bpettis/.gnupg/trustdb.gpg: trustdb created
gpg: key 105BD0E739499BDB: public key "Piotr Kuczynski <piotr.kuczynski@gmail.com>" imported
gpg: key 3804BB82D39DC0E3: public key "Michal Papis (RVM signing) <mpapis@gmail.com>" imported
gpg: Total number processed: 2
gpg: imported: 2
bpettis@huxley:~$ curl -sSL https://get.rvm.io | bash -s stable --ruby
Downloading https://github.com/rvm/rvm/archive/1.29.12.tar.gz
Downloading https://github.com/rvm/rvm/releases/download/1.29.12/1.29.12.tar.gz.asc
gpg: Signature made Fri 15 Jan 2021 12:46:22 PM CST
gpg: using RSA key 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
gpg: Good signature from "Piotr Kuczynski <piotr.kuczynski@gmail.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 7D2B AF1C F37B 13E2 069D 6956 105B D0E7 3949 9BDB
GPG verified '/home/bpettis/.rvm/archives/rvm-1.29.12.tgz'
Installing RVM to /home/bpettis/.rvm/
Adding rvm PATH line to /home/bpettis/.profile /home/bpettis/.mkshrc /home/bpettis/.bashrc /home/bpettis/.zshrc.
Adding rvm loading line to /home/bpettis/.profile /home/bpettis/.bash_profile /home/bpettis/.zlogin.
Installation of RVM in /home/bpettis/.rvm/ is almost complete:
* To start using RVM you need to run `source /home/bpettis/.rvm/scripts/rvm`
in all your open shell windows, in rare cases you need to reopen all shell windows.
Thanks for installing RVM 🙏
Please consider donating to our open collective to help us maintain RVM.
👉 Donate: https://opencollective.com/rvm/donate
Ruby enVironment Manager 1.29.12 (latest) (c) 2009-2020 Michal Papis, Piotr Kuczynski, Wayne E. Seguin
Searching for binary rubies, this might take some time.
No binary rubies available for: debian/11/x86_64/ruby-3.0.0.
Continuing with compilation. Please read 'rvm help mount' to get more information on binary rubies.
Checking requirements for debian.
Installing requirements for debian.
Updating system....
Installing required packages: gawk, libsqlite3-dev, libtool, sqlite3, libgmp-dev........
Requirements installation successful.
Installing Ruby from source to: /home/bpettis/.rvm/rubies/ruby-3.0.0, this may take a while depending on your cpu(s)...
ruby-3.0.0 - #downloading ruby-3.0.0, this may take a while depending on your connection...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 18.6M 100 18.6M 0 0 9381k 0 0:00:02 0:00:02 --:--:-- 9381k
ruby-3.0.0 - #extracting ruby-3.0.0 to /home/bpettis/.rvm/src/ruby-3.0.0.....
ruby-3.0.0 - #configuring.........................................................................
ruby-3.0.0 - #post-configuration..
ruby-3.0.0 - #compiling...................................................................................-
ruby-3.0.0 - #installing.......................
ruby-3.0.0 - #making binaries executable...
Installed rubygems 3.2.3 is newer than 3.0.9 provided with installed ruby, skipping installation, use --force to force installation.
ruby-3.0.0 - #gemset created /home/bpettis/.rvm/gems/ruby-3.0.0@global
ruby-3.0.0 - #importing gemset /home/bpettis/.rvm/gemsets/global.gems.....................................-
ruby-3.0.0 - #generating global wrappers........
ruby-3.0.0 - #gemset created /home/bpettis/.rvm/gems/ruby-3.0.0
ruby-3.0.0 - #importing gemsetfile /home/bpettis/.rvm/gemsets/default.gems evaluated to empty gem list
ruby-3.0.0 - #generating default wrappers........
ruby-3.0.0 - #adjusting #shebangs for (gem irb erb ri rdoc testrb rake).
Install of ruby-3.0.0 - #complete
Ruby was built without documentation, to build it run: rvm docs generate-ri
Creating alias default for ruby-3.0.0...
* To start using RVM you need to run `source /home/bpettis/.rvm/scripts/rvm`
in all your open shell windows, in rare cases you need to reopen all shell windows.
bpettis@huxley:~$ Ah shoot. So this won't work - we can see that it installed rvm to /home/bpettis/.rvm/scripts/rvm
Let's try again... This time making sure to prepend sudo to appropriate commands so we end up doing an installation that is available to all users.
bpettis@huxley:~$ sudo gpg --keyserver hkp://pgp.mit.edu --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
gpg: key 105BD0E739499BDB: 2 duplicate signatures removed
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key 105BD0E739499BDB: public key "Piotr Kuczynski <piotr.kuczynski@gmail.com>" imported
gpg: key 3804BB82D39DC0E3: public key "Michal Papis (RVM signing) <mpapis@gmail.com>" imported
gpg: Total number processed: 2
gpg: imported: 2
bpettis@huxley:~$ curl -L https://get.rvm.io | sudo bash -s stable
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 194 100 194 0 0 713 0 --:--:-- --:--:-- --:--:-- 713
100 24535 100 24535 0 0 62910 0 --:--:-- --:--:-- --:--:-- 62910
Downloading https://github.com/rvm/rvm/archive/1.29.12.tar.gz
Downloading https://github.com/rvm/rvm/releases/download/1.29.12/1.29.12.tar.gz.asc
gpg: Signature made Fri 15 Jan 2021 12:46:22 PM CST
gpg: using RSA key 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
gpg: Good signature from "Piotr Kuczynski <piotr.kuczynski@gmail.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 7D2B AF1C F37B 13E2 069D 6956 105B D0E7 3949 9BDB
GPG verified '/usr/local/rvm/archives/rvm-1.29.12.tgz'
Creating group 'rvm'
Installing RVM to /usr/local/rvm/
Installation of RVM in /usr/local/rvm/ is almost complete:
* First you need to add all users that will be using rvm to 'rvm' group,
and logout - login again, anyone using rvm will be operating with `umask u=rwx,g=rwx,o=rx`.
* To start using RVM you need to run `source /etc/profile.d/rvm.sh`
in all your open shell windows, in rare cases you need to reopen all shell windows.
* Please do NOT forget to add your users to the rvm group.
The installer no longer auto-adds root or users to the rvm group. Admins must do this.
Also, please note that group memberships are ONLY evaluated at login time.
This means that users must log out then back in before group membership takes effect!
Thanks for installing RVM 🙏
Please consider donating to our open collective to help us maintain RVM.
👉 Donate: https://opencollective.com/rvm/donate
bpettis@huxley:~$ This installs rvm into /usr/local/rvm - much better. Now I just have to add users to the rvm group to let them access it. First, I add myself:
sudo usermod -aG rvm bpettis
And then I add the mastodon user:
sudo usermod -aG rvm mastodon
Don't forget that you'll need to log out and log back in for user group changes to apply. After logging back in, we can check that everything works as expected:
mastodon@huxley:~$ rvm -v
rvm 1.29.12 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]bpettis@huxley:~$ rvm -v
rvm 1.29.12 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]Next, as the mastodon user, I install ruby 3.0.4, making sure to install with jemalloc
mastodon@huxley:~$ rvm install 3.0.4 -C --with-jemalloc
Checking requirements for debian.
Requirements installation successful.
Installing Ruby from source to: /home/mastodon/.rvm/rubies/ruby-3.0.4, this may take a while depending on your cpu(s)...
ruby-3.0.4 - #downloading ruby-3.0.4, this may take a while depending on your connection...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20.1M 100 20.1M 0 0 9779k 0 0:00:02 0:00:02 --:--:-- 9779k
No checksum for downloaded archive, recording checksum in user configuration.
ruby-3.0.4 - #extracting ruby-3.0.4 to /home/mastodon/.rvm/src/ruby-3.0.4.....
ruby-3.0.4 - #configuring.........................................................................
ruby-3.0.4 - #post-configuration..
ruby-3.0.4 - #compiling...................................................................................-
ruby-3.0.4 - #installing.......................
ruby-3.0.4 - #making binaries executable...
Installed rubygems 3.2.33 is newer than 3.0.9 provided with installed ruby, skipping installation, use --force to force installation.
ruby-3.0.4 - #gemset created /home/mastodon/.rvm/gems/ruby-3.0.4@global
ruby-3.0.4 - #importing gemset /home/mastodon/.rvm/gemsets/global.gems....................................-
ruby-3.0.4 - #generating global wrappers........
ruby-3.0.4 - #gemset created /home/mastodon/.rvm/gems/ruby-3.0.4
ruby-3.0.4 - #importing gemsetfile /home/mastodon/.rvm/gemsets/default.gems evaluated to empty gem list
ruby-3.0.4 - #generating default wrappers........
ruby-3.0.4 - #adjusting #shebangs for (gem irb erb ri rdoc testrb rake).
Install of ruby-3.0.4 - #complete
Ruby was built without documentation, to build it run: rvm docs generate-ri
mastodon@huxley:~$
Sidenote - it's so cool to see the CPU status LEDs working correctly while the system was compiling Ruby. Very glad that I got these working!
Checking that Ruby installed:
mastodon@huxley:~$ rvm list
=* ruby-3.0.4 [ x86_64 ]
# => - current
# =* - current && default
# * - defaultAnd then installing bundler:
mastodon@huxley:~$ gem install bundler --no-document
Fetching bundler-2.3.26.gem
Successfully installed bundler-2.3.26
1 gem installedAlright, now (I think) I can swap back to the published mastodon instructions - jumping back in at the "PostgreSQL steps"
PostgreSQL
I chose to skip over the suggestion of using pgTune to optimize the performance, since I'm lazy and figure since I'll be the only one using the instance I can just deal with it later on if it becomes an issue.
Creating a user
Again, trying to follow the posted instructions as closely as possible, I created a mastodon user in postgres
root@huxley:~# sudo -u postgres psql
could not change directory to "/root": Permission denied
psql (15.1 (Debian 15.1-1.pgdg110+1))
Type "help" for help.
postgres=# CREATE USER mastodon CREATEDB;
CREATE ROLE
postgres=# Mastodon
Don't forget to swap back over to the mastodon user: su - mastodon
Clone the repository and pull the latest release
mastodon@huxley:~$ git clone https://github.com/mastodon/mastodon.git live && cd live
Cloning into 'live'...
remote: Enumerating objects: 146562, done.
remote: Total 146562 (delta 0), reused 0 (delta 0), pack-reused 146562
Receiving objects: 100% (146562/146562), 178.83 MiB | 10.86 MiB/s, done.
Resolving deltas: 100% (105280/105280), done.
ruby-3.0.4 - #gemset created /home/mastodon/.rvm/gems/ruby-3.0.4@mastodon
ruby-3.0.4 - #generating mastodon wrappers..............
mastodon@huxley:~/live$ git checkout $(git tag -l | grep -v 'rc[0-9]*$' | sort -V | tail -n 1)
Note: switching to 'v4.0.2'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at 03b0f3ac8 Bump version to 4.0.2 (#20725)
mastodon@huxley:~/live$ Installing more dependencies
bundle config deployment 'true'
bundle config without 'development test'
bundle install -j$(getconf _NPROCESSORS_ONLN)
yarn install --pure-lockfileMore CPU LEDs woohoo! Relaxen und watchen das blinkenlights
Configuration
Sticking with the theme of 'follow the instructions to the letter' (except for the times I dont), I went ahead with the interactive setup wizard:
RAILS_ENV=production bundle exec rake mastodon:setup- I used
benpettis.ninjaas the domain name, but figure I will need to check theLOCAL_DOMAINandWEB_DOMAINsettings before launching fully. - When I reached the email setup stage, I realized that I hadn't actually set up my email yet. Shit.
SMTP server
I will be using SendGrid for emails, since it's free for low volumes (https://sendgrid.com/pricing/)
- Sign up for the free tier, and create an account and provide info. I just listed my company as "Pettis Homelab" and put my personal website as the URL.
- Confirmed my personal email address for my SendGrid account
- Set up 2FA because we love security!
- Set up
benpettis.ninjaas my email domain, and used the default advanced settings (Yes automated security, but no custom return path and no custom DKIM selector). SendGrid gave me a bunch of CNAME records to add to my DNS through my registrar - Take the dog on a walk and wait a while for DNS records to propagate
- Realize that I miscopied the CNAME records and try again.
- Get a SendGrid API key
Finally can go back to the mastodon setup wizard
More Configuration
SMTP server: smtp.sendgrid.net
Port: 587
Username: apikey
Password: <API KEY>
Authentication: plain
OpenSSL verify mode: none
STARTTLS: auto
sent-from email address: mastodon-notifications@benpettis.ninja
And holy shit the test email worked! I'm pleasantly surprised that it worked as expected.
I finished the configuration, let it run the database migrations, and compile the assets.
Domain Name Configurations
I entered benpettis.ninja as the domain name during the configuration. This is because I want my username to appear as @bpettis@benpettis.ninja
BUT. I want to access mastodon at mastodon.benpettis.ninja and keep benpettis.ninja redirecting to benpettis.com
I opened up .env.production to edit the config
It had correctly created LOCAL_DOMAIN=benpettis.ninja, so I just had to add WEB_DOMAIN=mastodon.benpettis.ninja
nginx
I started out by copying the base configuration, but with the knowledge that I'll need to make some adjustments with the strangeness of my environment.
cp /home/mastodon/live/dist/nginx.conf /etc/nginx/sites-available/mastodon
ln -s /etc/nginx/sites-available/mastodon /etc/nginx/sites-enabled/mastodonYou'll need to be root to do these.
Okay, I think I need to set this up so that nginx is only listening for HTTP requests - since it's only going to be directly receiving traffic internally. It's my other Apache server that will be handling HTTPS.
I essentially merged the two server sections from the default config and removed the SSL bits. Here's my nginx config:
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
upstream backend {
server 127.0.0.1:3000 fail_timeout=0;
}
upstream streaming {
server 127.0.0.1:4000 fail_timeout=0;
}
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=CACHE:10m inactive=7d max_size=1g;
server {
listen 80;
listen [::]:80;
server_name mastodon.benpettis.ninja;
root /home/mastodon/live/public;
location /.well-known/acme-challenge/ { allow all; }
keepalive_timeout 70;
sendfile on;
client_max_body_size 80m;
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript image/svg+xml image/x-icon;
location / {
try_files $uri @proxy;
}
# If Docker is used for deployment and Rails serves static files,
# then needed must replace line `try_files $uri =404;` with `try_files $uri @proxy;`.
location = /sw.js {
add_header Cache-Control "public, max-age=604800, must-revalidate";
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
try_files $uri =404;
}
location ~ ^/assets/ {
add_header Cache-Control "public, max-age=2419200, must-revalidate";
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
try_files $uri =404;
}
location ~ ^/avatars/ {
add_header Cache-Control "public, max-age=2419200, must-revalidate";
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
try_files $uri =404;
}
location ~ ^/emoji/ {
add_header Cache-Control "public, max-age=2419200, must-revalidate";
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
try_files $uri =404;
}
location ~ ^/headers/ {
add_header Cache-Control "public, max-age=2419200, must-revalidate";
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
try_files $uri =404;
}
location ~ ^/packs/ {
add_header Cache-Control "public, max-age=2419200, must-revalidate";
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
try_files $uri =404;
}
location ~ ^/shortcuts/ {
add_header Cache-Control "public, max-age=2419200, must-revalidate";
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
try_files $uri =404;
}
location ~ ^/sounds/ {
add_header Cache-Control "public, max-age=2419200, must-revalidate";
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
try_files $uri =404;
}
location ~ ^/system/ {
add_header Cache-Control "public, max-age=2419200, immutable";
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
try_files $uri =404;
}
location ^~ /api/v1/streaming {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Proxy "";
proxy_pass http://streaming;
proxy_buffering off;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
tcp_nodelay on;
}
location @proxy {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Proxy "";
proxy_pass_header Server;
proxy_pass http://backend;
proxy_buffering on;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_cache CACHE;
proxy_cache_valid 200 7d;
proxy_cache_valid 410 24h;
proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
add_header X-Cached $upstream_cache_status;
tcp_nodelay on;
}
error_page 404 500 501 502 503 504 /500.html;
}DNS
I hadn't done this before, so I added a CNAME record for the host mastodon.benpettis.ninja and set the answer to bpettis.hopto.org, my DynamicDNS URL
Apache
Next, I hopped back over to my other box which is running my network's primary Web server. I will be using Apache and Virtual Host routing to send requests over to the other box.
First, we create the file /etc/apache2/sites-available/mastodon.benpettis.ninja.conf and set up the proxy:
<VirtualHost *:80>
ServerName mastodon.benpettis.ninja
ProxyPreserveHost On
ProxyPass / http://192.168.0.21/
ProxyPassReverse / http://192.168.0.21/
</VirtualHost>Next we enable this site:
sudo a2ensite mastodon.benpettis.ninja
Before we move forward, we need to make sure that we send requests to https://benpettis.ninja/.well-known/webfinger and redirect those over to the other box.
I think this should actually be simple enough by adding the line RedirectMatch 301 ^/.well-known/webfinger(.*) https://mastodon.benpettis.ninja/$1 to my config file for benpettis.ninja
Now we can reload apache:
sudo systemctl reload apache2
I think
that things should be mostly set up - except for one thing! We only
have HTTP sites configured. We need to make sure we get certificates and
set up HTTPS for benpettis.ninja and mastodon.benpettis.ninja, which I do just by running sudo certbot --apache
and then after all that, I restart apache just for good measure sudo systemctl restart apache2
Fixing the Web setup
And just don't a quick test, it would appear that I broke something. Currently nothing from my Apache Web server are accessible. Requests are just timing out....
Or maybe I was just impatient. After trying again a few minutes later, the old sites were accessible again. And https://benpettis.ninja is redirecting to https://benpettis.com
But now the moment of truth... what happens when we go to https://mastodon.benpettis.ninja? It fails. We get a "Too Many Redirects" error... hrm.
The issue was that I messed up setting up the Apache proxy. My mastodon server is going to run on the box at 192.168.0.20 and Apache runs on 192.168.0.21. So in my above config I was proxying requests from apache... back to apache. Whoops. I fixed the IP address and tried again.
And now it's taking me to the default Apache page? That's odd. But I think actually a sign that things are working - since I think this is actually the Apache page on the other box! The problem is that nginx just isn't running, and apache is still up.
We'll need to stop and then disable Apache:
sudo systemctl stop apache2
sudo systemctl disable apache2And then we just enable and start Nginx:
sudo systemctl enable nginx
sudo systemctl restart nginxAnd then we can try once again to head to https://mastodon.benpettis.ninja
And it works! Well, not really. But this is what we expect to see at this point. We can now (again) head back over to the main Mastodon instructions
Starting the mastodon service
We've now (somehow) made it to the end of the instructions (https://docs.joinmastodon.org/admin/install/#setting-up-systemd-services)
We'll make sure that we're root su - root
Next we copy the templates: cp /home/mastodon/live/dist/mastodon-*.service /etc/systemd/system/
We'll quickly check through the files in /etc/systemd/system/mastodon-*.service and double check that there aren't any weird lingering example.coms anywhere
And now we can start and enable the services:
systemctl daemon-reload
systemctl enable --now mastodon-web mastodon-sidekiq mastodon-streamingMoment of truth, does it work!? No!
Looks like some services failed to start:
bpettis@huxley:~$ sudo systemctl status mastodon-web
● mastodon-web.service - mastodon-web
Loaded: loaded (/etc/systemd/system/mastodon-web.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2022-12-13 22:23:52 CST; 652ms ago
Process: 49565 ExecStart=/home/mastodon/.rbenv/shims/bundle exec puma -C config/puma.rb (code=exited, >
Main PID: 49565 (code=exited, status=203/EXEC)
CPU: 25ms
Dec 13 22:23:52 huxley systemd[1]: mastodon-web.service: Scheduled restart job, restart counter is at 5.
Dec 13 22:23:52 huxley systemd[1]: Stopped mastodon-web.
Dec 13 22:23:52 huxley systemd[1]: mastodon-web.service: Start request repeated too quickly.
Dec 13 22:23:52 huxley systemd[1]: mastodon-web.service: Failed with result 'exit-code'.
Dec 13 22:23:52 huxley systemd[1]: Failed to start mastodon-web.
bpettis@huxley:~$ sudo systemctl status mastodon-streaming
● mastodon-streaming.service - mastodon-streaming
Loaded: loaded (/etc/systemd/system/mastodon-streaming.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-12-13 22:23:09 CST; 1min 12s ago
Main PID: 49481 (node)
Tasks: 18 (limit: 38488)
Memory: 83.1M
CPU: 3.088s
CGroup: /system.slice/mastodon-streaming.service
├─49481 /usr/bin/node ./streaming
└─49527 /usr/bin/node /home/mastodon/live/streaming
Dec 13 22:23:09 huxley systemd[1]: Started mastodon-streaming.
Dec 13 22:23:10 huxley node[49481]: WARN Starting streaming API server master with 1 workers
Dec 13 22:23:11 huxley node[49527]: WARN Starting worker 1
Dec 13 22:23:11 huxley node[49527]: WARN Worker 1 now listening on 127.0.0.1:4000
bpettis@huxley:~$ sudo systemctl status mastodon-sidekiq
● mastodon-sidekiq.service - mastodon-sidekiq
Loaded: loaded (/etc/systemd/system/mastodon-sidekiq.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2022-12-13 22:23:10 CST; 1min 49s ago
Process: 49519 ExecStart=/home/mastodon/.rbenv/shims/bundle exec sidekiq -c 25 (code=exited, status=20>
Main PID: 49519 (code=exited, status=203/EXEC)
CPU: 26ms
Dec 13 22:23:10 huxley systemd[1]: mastodon-sidekiq.service: Scheduled restart job, restart counter is at >
Dec 13 22:23:10 huxley systemd[1]: Stopped mastodon-sidekiq.
Dec 13 22:23:10 huxley systemd[1]: mastodon-sidekiq.service: Start request repeated too quickly.
Dec 13 22:23:10 huxley systemd[1]: mastodon-sidekiq.service: Failed with result 'exit-code'.
Dec 13 22:23:10 huxley systemd[1]: Failed to start mastodon-sidekiq.
bpettis@huxley:~$ Troubleshooting
Let's dig into the error logs and see if we can figure out what's going on. We'll start with mastodon-web
bpettis@huxley:~$ sudo journalctl -x -u mastodon-web
-- Journal begins at Sun 2022-08-07 08:25:11 CDT, ends at Tue 2022-12-13 22:27:12 CST. --
Dec 13 22:22:45 huxley systemd[1]: Started mastodon-web.
░░ Subject: A start job for unit mastodon-web.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit mastodon-web.service has finished successfully.
░░
░░ The job identifier is 2251.
Dec 13 22:22:45 huxley systemd[49407]: mastodon-web.service: Failed to locate executable /home/mastodon/.rbenv/shims/bundle: No such file or directory
░░ Subject: Process /home/mastodon/.rbenv/shims/bundle could not be executed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The process /home/mastodon/.rbenv/shims/bundle could not be executed and failed.
░░
░░ The error number returned by this process is ERRNO.
Dec 13 22:22:45 huxley systemd[49407]: mastodon-web.service: Failed at step EXEC spawning /home/mastodon/.rbenv/shims/bundle: No such file or directory
░░ Subject: Process /home/mastodon/.rbenv/shims/bundle could not be executed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The process /home/mastodon/.rbenv/shims/bundle could not be executed and failed.
░░
░░ The error number returned by this process is ERRNO.
Dec 13 22:22:45 huxley systemd[1]: mastodon-web.service: Main process exited, code=exited, status=203/EXEC
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit mastodon-web.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 203.
Dec 13 22:22:45 huxley systemd[1]: mastodon-web.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit mastodon-web.service has entered the 'failed' state with result 'exit-code'.
Dec 13 22:22:45 huxley systemd[1]: mastodon-web.service: Scheduled restart job, restart counter is at 1.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ Automatic restarting of the unit mastodon-web.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Dec 13 22:22:45 huxley systemd[1]: Stopped mastodon-web.
░░ Subject: A stop job for unit mastodon-web.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A stop job for unit mastodon-web.service has finished.
░░
░░ The job identifier is 2479 and the job result is done.
Dec 13 22:22:45 huxley systemd[1]: Started mastodon-web.and what do you know - it looks like my earlier decision to use rvm instead of rbenv is the issue.
I did quickly check the service files in /etc/systemd/system/mastodon-*.service but completely missed that it's trying to start in an rbenv directory:
ExecStart=/home/mastodon/.rbenv/shims/bundle exec puma -C config/puma.rbHrm... looks like I messed up my Ruby installation - and it's not actually accessible for the mastodon user:
mastodon@huxley:~/live$ rvm list
Required ruby-3.0.4 is not installed.
To install do: 'rvm install "ruby-3.0.4"'
# No rvm rubies installed yet. Try 'rvm help install'.While su'd as the mastodon user I ran rvm install ruby-3.0.4 -C --with-jemalloc
I then ran rvm alias create default ruby-3.0.4 to set up the mastodon user to know about Ruby 3.0.4 and maybe set the PATH so it knows to use this version?
mastodon@huxley:/root$ rvm list
=* ruby-3.0.4 [ x86_64 ]
# => - current
# =* - current && default
# * - defaultNow
that everything is (hopefully) installed, I was able to find the
equivalent directory and figure out how to swap out the command in mastodon-web.service
ExecStart = /home/mastodon/.rvm/wrappers/ruby-3.0.4@mastodon/bundle exec puma -C config/puma.rb
I then made a similar change in mastodon-sidekiq.service:
ExecStart = /home/mastodon/.rvm/wrappers/ruby-3.0.4@mastodon/bundle exec sidekiq -c 25After making these changes I ran sudo systemctl daemon-reload and then restarted everything sudo systemctl restart mastodon-web mastodon-sidekiq mastodon-streaming and by checking the status it already looks promising:
bpettis@huxley:~$ sudo systemctl restart mastodon-web mastodon-sidekiq mastodon-streaming
bpettis@huxley:~$ sudo systemctl status mastodon-web
● mastodon-web.service - mastodon-web
Loaded: loaded (/etc/systemd/system/mastodon-web.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-12-13 22:51:29 CST; 7s ago
Main PID: 80220 (ruby)
Tasks: 2 (limit: 38488)
Memory: 151.6M
CPU: 7.386s
CGroup: /system.slice/mastodon-web.service
└─80220 puma 5.6.5 (tcp://127.0.0.1:3000) [live]
Dec 13 22:51:29 huxley systemd[1]: Started mastodon-web.
Dec 13 22:51:31 huxley bundle[80220]: [80220] Puma starting in cluster mode...
Dec 13 22:51:31 huxley bundle[80220]: [80220] * Puma version: 5.6.5 (ruby 3.0.4-p208) ("Birdie's Version")
Dec 13 22:51:31 huxley bundle[80220]: [80220] * Min threads: 5
Dec 13 22:51:31 huxley bundle[80220]: [80220] * Max threads: 5
Dec 13 22:51:31 huxley bundle[80220]: [80220] * Environment: production
Dec 13 22:51:31 huxley bundle[80220]: [80220] * Master PID: 80220
Dec 13 22:51:31 huxley bundle[80220]: [80220] * Workers: 2
Dec 13 22:51:31 huxley bundle[80220]: [80220] * Restarts: (✔) hot (✖) phased
Dec 13 22:51:31 huxley bundle[80220]: [80220] * Preloading application
bpettis@huxley:~$ And... we're now back to the "Too Many Redirects" error??!?!?
I suspect that this is happening in nginx and has to do with my customized config file.
From the other page I was looking at (https://discourse.joinmastodon.org/t/nginx-reverse-proxy-on-another-server/485/7), I found an example nginx config where they were setting the server_name to the IP address instead of the URL. I made a similar change and restarted nginx, and now https://mastodon.benpettis.ninja is directing me to /var/www/html/index.html on the mastodon box. So the request is getting proxied, but the mastodon process is not responding. Hmm.
Another post on that same page notes that they needed to add some additional config to their proxy:
server {
...
location / {
...
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}They were using nginx on both of their boxes, but I suspect I can do something similar in Apache??
Or maybe not. It looks like the issue is that the mastodon nginx config really wants all traffic to come in as HTTPS. And we're proxying things over as HTTP. So the server tries to upgrade the connection, which creates a redirection loop. (See https://www.reddit.com/r/Mastodon/comments/qfg5d4/question_example_nginx_config_for_reverse_proxy/hi86qx7/)
And... it was about this time that the power went out in my apartment and the servers went down. It was late so I decided to call it a night and try looking at this again later.
The only issue is that I'm not sure I can actually make a request to https://192.168.0.20 since there's no good way to obtain a certificate for an internal address.
But, if I can just send everything over to the other server I think I should be okay.
The Next Day
My thinking is that I can try using a self-signed certificate on Huxley so that Kafka can proxy traffic to https://192.168.0.20 - which would then let Mastodon run fully over HTTPS which is what it expects
I'm concerned that the power issue may be a result of me just overloading my apartment's circuits. I have a PHEV and when it's charging it's drawing a lot of power. That, combined with all the computers I have running, may just be too much power draw at once. So we'll have to keep that in mind as a limitation....
Creating a Self-Signed SSL Certificate
I am following the instructions from digital ocean: https://www.digitalocean.com/community/tutorials/how-to-create-a-self-signed-ssl-certificate-for-nginx-on-debian-10
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/mastodon-selfsigned.key -out /etc/ssl/certs/mastodon-selfsigned.crtI also created a Diffie-Hellman group, which I'll be perfectly honest I don't really understand what this is for.
sudo openssl dhparam -out /etc/nginx/dhparam.pem 4096
Next, I re-copied the default mastodon config for nginx - in case my earlier modifications were especially fucking things up:
sudo cp /home/mastodon/live/dist/nginx.conf /etc/nginx/sites-available/mastodon
And then I had to make some more adjustments to use the self-signed keys that I created:
ssl_certificate /etc/ssl/certs/mastodon-selfsigned.crt;
ssl_certificate_key /etc/ssl/private/mastodon-selfsigned.key;And of course a few more adjustments to the server name - since this nginx server will be receiving requests from the proxy - aka to its IP address and not the public FQDN
server {
listen 80;
listen [::]:80;
server_name 192.168.0.20;
root /home/mastodon/live/public;
location /.well-known/acme-challenge/ { allow all; }
location / { return 301 https://$host$request_uri; }
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name 192.168.0.20;
[...]Next we test the config by running sudo nginx -t:
bpettis@huxley:~$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
bpettis@huxley:~$ Okay well I'm certainly hopeful
then that this is going to work. The last thing to do is to hop back
over to the other server and make sure the reverse proxy is sending
traffic to https://192.168.0.20 and not http://192.168.0.20
Don't forget to restart apache (sudo systemctl restart apache2) here and then restart nginx (sudo systemctl restart nginx) on the other box
And this still failed. Gives me a 403 forbidden error.
As a quick test, I set the server_name value back to mastodon.benpettis.ninja but now was getting an 'Internal Server Error' from Apache. So I went back to having server_name set to 192.168.0.20
When I attempt to access https://mastodon.benpettis.ninja I hit that same generic Apache 'Internal Server Error'
When I attempt to access https://192.168.0.20 I get an HTTP 403 error (Forbidden)
I was curious if the mastodon-web service was even running, so I did a quick check:
bpettis@huxley:~$ sudo systemctl status mastodon-web
● mastodon-web.service - mastodon-web
Loaded: loaded (/etc/systemd/system/mastodon-web.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-12-13 22:43:59 CST; 20h ago
Main PID: 567 (ruby)
Tasks: 29 (limit: 38488)
Memory: 285.1M
CPU: 16.124s
CGroup: /system.slice/mastodon-web.service
├─567 puma 5.6.5 (tcp://127.0.0.1:3000) [live]
├─800 puma: cluster worker 0: 567 [live]
└─809 puma: cluster worker 1: 567 [live]
Dec 14 19:17:01 huxley bundle[800]: [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked host: 192.168.0.20
Dec 14 19:19:47 huxley bundle[809]: [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked host: 192.168.0.20
Dec 14 19:19:52 huxley bundle[800]: [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked host: 192.168.0.20
Dec 14 19:19:54 huxley bundle[800]: [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked host: 192.168.0.20
Dec 14 19:20:59 huxley bundle[809]: [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked host: 192.168.0.20
Dec 14 19:21:00 huxley bundle[809]: [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked host: 192.168.0.20
Dec 14 19:21:00 huxley bundle[800]: [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked host: 192.168.0.20
Dec 14 19:21:00 huxley bundle[809]: [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked host: 192.168.0.20
Dec 14 19:21:01 huxley bundle[809]: [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked host: 192.168.0.20
Dec 14 19:21:01 huxley bundle[809]: [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked host: 192.168.0.20These errors about a blocked host are interesting to me - they match the time that I was testing the site.
With a bit of Googling, I find that this is likely expected behavior for Mastodon (https://github.com/mastodon/mastodon/discussions/18944#discussioncomment-3355779) - as it configures Ruby to block requests that are coming with an unexpected host value
Based on that comment, I think I can possibly just set 192.168.0.20 as an allowed domain name with ALTERNATE_DOMAINS=192.168.0.20
after this I restart the web service with systemctl restart mastodon-web and....
Holy shit it works! Kinda. I can access the page at https://192.168.0.20. Looks like a lot of resources are refusing to load because it's violating security policies (regarding request origin)
But https://mastodon.benpettis.ninja still gives me a HTTP 500 error. And we can see from the error message that it's Apache throwing the error.
So now where things are breaking down is in the Apache proxy. I check the log /var/log/apache2/error.log and see that we're having problems with SSL:
[Wed Dec 14 19:30:23.290307 2022] [core:error] [pid 47811] [remote 192.168.0.20:443] AH01961: failed to enable ssl support [Hint: if using mod_ssl, see SSLProxyEngine]
[Wed Dec 14 19:30:23.290362 2022] [proxy:error] [pid 47811] AH00961: https: failed to enable ssl support for 192.168.0.20:443 (192.168.0.20)
[Wed Dec 14 19:31:49.849903 2022] [core:error] [pid 47811] [remote 192.168.0.20:443] AH01961: failed to enable ssl support [Hint: if using mod_ssl, see SSLProxyEngine]
[Wed Dec 14 19:31:49.849944 2022] [proxy:error] [pid 47811] AH00961: https: failed to enable ssl support for 192.168.0.20:443 (192.168.0.20)
[Wed Dec 14 19:31:49.849951 2022] [core:error] [pid 47809] [remote 192.168.0.20:443] AH01961: failed to enable ssl support [Hint: if using mod_ssl, see SSLProxyEngine]
[Wed Dec 14 19:31:49.849968 2022] [proxy:error] [pid 47809] AH00961: https: failed to enable ssl support for 192.168.0.20:443 (192.168.0.20)
[Wed Dec 14 19:31:49.855145 2022] [core:error] [pid 47813] [remote 192.168.0.20:443] AH01961: failed to enable ssl support [Hint: if using mod_ssl, see SSLProxyEngine]
[Wed Dec 14 19:31:49.855192 2022] [proxy:error] [pid 47813] AH00961: https: failed to enable ssl support for 192.168.0.20:443 (192.168.0.20)
[Wed Dec 14 19:31:54.419510 2022] [core:error] [pid 48032] [remote 192.168.0.20:443] AH01961: failed to enable ssl support [Hint: if using mod_ssl, see SSLProxyEngine]
[Wed Dec 14 19:31:54.419545 2022] [proxy:error] [pid 48032] AH00961: https: failed to enable ssl support for 192.168.0.20:443 (192.168.0.20)Thank goodness for clear error messages! I go back into my site conf files and add SSLProxyEngine on and then restart apache sudo systemctl restart apache2
Alright, now we're getting somewhere! We're still getting an HTTP 500 error, but we can see that it's at least trying to use SSL now. And this problem is to be expected - we're using a self-signed certificate so of course Apache does not trust it.
Per (https://www.helphybris.com/2018/02/apache-web-server-for-hybris.html#proxypassSSL) one option could be to just disable SSL verification.
Or, I can use SSLCertificateKeyFile and SSLCertificateFile and specify where the certificate files are (after copying them)
I copy them over to the Apache box (using SCP for the cert file, and then I just copy/pasted the key into a new file. But I wonder if that will mess up the permissions.)
And we then add these into the site config files...?
<VirtualHost *:80>
ServerName mastodon.benpettis.ninja
ProxyPreserveHost On
SSLProxyEngine on
SSLCertificateKeyFile /etc/ssl/private/mastodon-selfsigned.key
SSLCertificateFile /etc/ssl/certs/mastodon-selfsigned.crt
ProxyPass / https://192.168.0.20/
ProxyPassReverse / https://192.168.0.20/
RewriteEngine on
RewriteCond %{SERVER_NAME} =mastodon.benpettis.ninja
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>
And this also failed. Okay fine I will try just disabling SSL verification. This is NOT great security practice, and to do so is to essentially assume that all internal network traffic can be trusted. Which may or may not ever actually be the case. But let's see if it works... We'll add the following into the Apache config:
SSLProxyEngine on
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire offAnd yes! It is now working! This is incredible! I can go to https://mastodon.benpettis.ninja and reach the web interface!
In the next section, I'll outline some of the specific setup and configuration that I did to customize my server and test its connection to other parts of the fediverse!