1,035
edits
No edit summary |
No edit summary |
||
(6 intermediate revisions by the same user not shown) | |||
Line 88: | Line 88: | ||
=== Separate Services === | === Separate Services === | ||
Even after increasing the number of worker threads to 75, we were still getting huge backlogs on our queues, particularly <code>pull</code> which was loading up with link crawl workers, presumably the slower jobs were getting in the way of faster jobs and they were piling up. | |||
We want to split up sidekiq into multiple processes using separate systemd service files. We want to a) make the site responsive by processing high-priority queues quickly but also b) use all our available resources by not having processes sit idle. So we give each of the main queues one service file that has that queue as the top prioriry, and mix the other queues in as secondary priorities - sidekiq will try and process items from the first queue first, second queue second, and so on. | We want to split up sidekiq into multiple processes using separate systemd service files. We want to a) make the site responsive by processing high-priority queues quickly but also b) use all our available resources by not having processes sit idle. So we give each of the main queues one service file that has that queue as the top prioriry, and mix the other queues in as secondary priorities - sidekiq will try and process items from the first queue first, second queue second, and so on. | ||
So we allocate 25 threads (and 25 db connections) each to four service files with the following priority orders. Note that we '''only do this after increasing the maximum postgres connections to 200,''' see https://hazelweakly.me/blog/scaling-mastodon/#db_pool-notes-from-nora's-blog | So we allocate 25 threads (and 25 db connections) each to four service files with the following priority orders, and two additional service files that give 5 threads to the lower-priority queues. Note that we '''only do this after increasing the maximum postgres connections to 200,''' see https://hazelweakly.me/blog/scaling-mastodon/#db_pool-notes-from-nora's-blog | ||
{{:Sidekiq#Services}} | |||
Each service file is identical except for this part. (We didn't use the <code>@.service</code> systemd templates because we couldn't find a nice way of doing a list of parameters that could handle multiple queues and variable thread numbers in different services): | |||
<syntaxhighlight lang="ini"> | <syntaxhighlight lang="ini"> | ||
Line 127: | Line 121: | ||
[Service] | [Service] | ||
Type=oneshot | Type=oneshot | ||
ExecStart=/bin/echo "mastodon-sidekiq exists only to collectively start and stop mastodon-sidekiq-* instances | ExecStart=/bin/echo "mastodon-sidekiq exists only to collectively start and stop mastodon-sidekiq-* instances" | ||
RemainAfterExit=yes | RemainAfterExit=yes | ||
Line 141: | Line 135: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
This lets sidekiq use all the available CPU (rather than having the queues pile up while the CPU is hovering around 50% usage), which may be good or bad, but it did drain the queues from ~20k to 0 in a matter of minutes. | |||
== [[Postgresql]] == | == [[Postgresql]] == | ||
Line 157: | Line 149: | ||
# OS Type: linux | # OS Type: linux | ||
# DB Type: web | # DB Type: web | ||
# Total Memory (RAM): | # Total Memory (RAM): 3 GB | ||
# CPUs num: 4 | # CPUs num: 4 | ||
# Connections num: 200 | # Connections num: 200 | ||
Line 163: | Line 155: | ||
max_connections = 200 | max_connections = 200 | ||
shared_buffers = | shared_buffers = 768MB | ||
effective_cache_size = | effective_cache_size = 2304MB | ||
maintenance_work_mem = | maintenance_work_mem = 192MB | ||
checkpoint_completion_target = 0.9 | checkpoint_completion_target = 0.9 | ||
wal_buffers = 16MB | wal_buffers = 16MB | ||
Line 171: | Line 163: | ||
random_page_cost = 1.1 | random_page_cost = 1.1 | ||
effective_io_concurrency = 200 | effective_io_concurrency = 200 | ||
work_mem = | work_mem = 1966kB | ||
huge_pages = off | huge_pages = off | ||
min_wal_size = 1GB | min_wal_size = 1GB | ||
Line 187: | Line 179: | ||
* https://hazelweakly.me/blog/scaling-mastodon/ | * https://hazelweakly.me/blog/scaling-mastodon/ | ||
* https://www.digitalocean.com/community/tutorials/how-to-scale-your-mastodon-server | * https://www.digitalocean.com/community/tutorials/how-to-scale-your-mastodon-server | ||
* https://hub.sunny.garden/2023/07/08/sidekiq-tuning-small-mastodon-servers/ | |||
** https://sunny.garden/@brook/111475392515987172 - "you can probably reduce your total thread count / db connections considerably if you'd like" | |||
== See Also == | |||
* [[ElasticSearch]] - which needs plenty of perf tuning | |||
[[Category:Mastodon]] | [[Category:Mastodon]] | ||
[[Category:Admin]] | [[Category:Admin]] | ||
[[Category:Tech WG]] | [[Category:Tech WG]] |