Performance: Difference between revisions

← Older edit

Performance (view source)

Revision as of 00:11, 9 January 2024

4,161 bytes added , 9 January

no edit summary

Jonny

Bureaucrats, Interface administrators, Administrators

1,035

edits

@@ Line 6: / Line 6: @@
 The changes we have actually made from the default configuration, each is either described below or on a separate page:
+* Split out sidekiq queues into separate service files
+* Optimized postgres using pgtune
+=== Archive ===
+Olde changes that aren't true anymore
 * Increase [[Sidekiq]] <code>DB_POOL</code> and <code>-c</code> values from from 25 to 75
+** 23-11-25: Replaced with separate sidekiq service files
 == [[Sidekiq]] ==
@@ Line 19: / Line 27: @@
 * Make multiple processes for a queue (after making a separate service)
-=== Configuration ===
+=== Default Configuration ===
 By default, the <code>mastodon-sidekiq</code> service is configured with 25 threads, the full service file is as follows:
-<pre>
+<syntaxhighlight lang="ini">
 [Unit]
 Description=mastodon-sidekiq
@@ Line 77: / Line 85: @@
 [Install]
 WantedBy=multi-user.target
-</pre>
+</syntaxhighlight>
+=== Separate Services ===
+Even after increasing the number of worker threads to 75, we were still getting huge backlogs on our queues, particularly <code>pull</code> which was loading up with link crawl workers, presumably the slower jobs were getting in the way of faster jobs and they were piling up.
+We want to split up sidekiq into multiple processes using separate systemd service files. We want to a) make the site responsive by processing high-priority queues quickly but also b) use all our available resources by not having processes sit idle. So we give each of the main queues one service file that has that queue as the top prioriry, and mix the other queues in as secondary priorities - sidekiq will try and process items from the first queue first, second queue second, and so on.
+So we allocate 25 threads (and 25 db connections) each to four service files with the following priority orders, and two additional service files that give 5 threads to the lower-priority queues. Note that we '''only do this after increasing the maximum postgres connections to 200,''' see https://hazelweakly.me/blog/scaling-mastodon/#db_pool-notes-from-nora's-blog
+{{:Sidekiq#Services}}
+Each service file is identical except for this part. (We didn't use the <code>@.service</code> systemd templates because we couldn't find a nice way of doing a list of parameters that could handle multiple queues and variable thread numbers in different services):
+<syntaxhighlight lang="ini">
+Environment="DB_POOL=25"
+ExecStart=/home/mastodon/.rbenv/shims/bundle exec sidekiq -q push -q pull -q default -q ingress -c 25
+</syntaxhighlight>
+and is located in <code>/etc/systemd/system</code> with the name of its primary queue (eg. <code>/etc/systemd/system/mastodon-sidekiq-default.service</code>)
+Then we make one meta-service file <code>mastodon-sidekiq.service</code> that can control the others:
+<syntaxhighlight lang="ini">
+[Unit]
+Description=mastodon-sidekiq
+After=network.target
+Wants=mastodon-sidekiq-default.service
+Wants=mastodon-sidekiq-ingress.service
+Wants=mastodon-sidekiq-mailers.service
+Wants=mastodon-sidekiq-pull.service
+Wants=mastodon-sidekiq-push.service
+Wants=mastodon-sidekiq-scheduler.service
+[Service]
+Type=oneshot
+ExecStart=/bin/echo "mastodon-sidekiq exists only to collectively start and stop mastodon-sidekiq-* instances"
+RemainAfterExit=yes
+[Install]
+WantedBy=multi-user.target
+</syntaxhighlight>
+and make the subsidiary service dependent on the main service
+<syntaxhighlight lang="ini">
+[Install]
+WantedBy=multi-user.target mastodon-sidekiq.service
+</syntaxhighlight>
+This lets sidekiq use all the available CPU (rather than having the queues pile up while the CPU is hovering around 50% usage), which may be good or bad, but it did drain the queues from ~20k to 0 in a matter of minutes.
+== [[Postgresql]] ==
+=== PGTune ===
+Following the advice of PGTune ( https://pgtune.leopard.in.ua/ ), postgres is configured like:
+<code>/etc/postgresql/15/main/postgresql.conf</code>
+<syntaxhighlight>
+# DB Version: 15
+# OS Type: linux
+# DB Type: web
+# Total Memory (RAM): 3 GB
+# CPUs num: 4
+# Connections num: 200
+# Data Storage: ssd
+max_connections = 200
+shared_buffers = 768MB
+effective_cache_size = 2304MB
+maintenance_work_mem = 192MB
+checkpoint_completion_target = 0.9
+wal_buffers = 16MB
+default_statistics_target = 100
+random_page_cost = 1.1
+effective_io_concurrency = 200
+work_mem = 1966kB
+huge_pages = off
+min_wal_size = 1GB
+max_wal_size = 4GB
+max_worker_processes = 4
+max_parallel_workers_per_gather = 2
+max_parallel_workers = 4
+max_parallel_maintenance_workers = 2
+</syntaxhighlight>
 == References ==
@@ Line 85: / Line 179: @@
 * https://hazelweakly.me/blog/scaling-mastodon/
 * https://www.digitalocean.com/community/tutorials/how-to-scale-your-mastodon-server
+* https://hub.sunny.garden/2023/07/08/sidekiq-tuning-small-mastodon-servers/
+** https://sunny.garden/@brook/111475392515987172 - "you can probably reduce your total thread count / db connections considerably if you'd like"
+== See Also ==
+* [[ElasticSearch]] - which needs plenty of perf tuning
 [[Category:Mastodon]]
 [[Category:Admin]]
 [[Category:Tech WG]]