1,035
edits
Performance: Difference between revisions
no edit summary
No edit summary |
No edit summary |
||
| (5 intermediate revisions by the same user not shown) | |||
| Line 93: | Line 93: | ||
We want to split up sidekiq into multiple processes using separate systemd service files. We want to a) make the site responsive by processing high-priority queues quickly but also b) use all our available resources by not having processes sit idle. So we give each of the main queues one service file that has that queue as the top prioriry, and mix the other queues in as secondary priorities - sidekiq will try and process items from the first queue first, second queue second, and so on. | We want to split up sidekiq into multiple processes using separate systemd service files. We want to a) make the site responsive by processing high-priority queues quickly but also b) use all our available resources by not having processes sit idle. So we give each of the main queues one service file that has that queue as the top prioriry, and mix the other queues in as secondary priorities - sidekiq will try and process items from the first queue first, second queue second, and so on. | ||
So we allocate 25 threads (and 25 db connections) each to four service files with the following priority orders. Note that we '''only do this after increasing the maximum postgres connections to 200,''' see https://hazelweakly.me/blog/scaling-mastodon/#db_pool-notes-from-nora's-blog | So we allocate 25 threads (and 25 db connections) each to four service files with the following priority orders, and two additional service files that give 5 threads to the lower-priority queues. Note that we '''only do this after increasing the maximum postgres connections to 200,''' see https://hazelweakly.me/blog/scaling-mastodon/#db_pool-notes-from-nora's-blog | ||
{{:Sidekiq#Services}} | |||
Each service file is identical except for this part. (We didn't use the <code>@.service</code> systemd templates because we couldn't find a nice way of doing a list of parameters that could handle multiple queues and variable thread numbers in different services): | |||
<syntaxhighlight lang="ini"> | <syntaxhighlight lang="ini"> | ||
| Line 129: | Line 121: | ||
[Service] | [Service] | ||
Type=oneshot | Type=oneshot | ||
ExecStart=/bin/echo "mastodon-sidekiq exists only to collectively start and stop mastodon-sidekiq-* instances | ExecStart=/bin/echo "mastodon-sidekiq exists only to collectively start and stop mastodon-sidekiq-* instances" | ||
RemainAfterExit=yes | RemainAfterExit=yes | ||
| Line 144: | Line 136: | ||
This lets sidekiq use all the available CPU (rather than having the queues pile up while the CPU is hovering around 50% usage), which may be good or bad, but it did drain the queues from ~20k to 0 in a matter of minutes. | This lets sidekiq use all the available CPU (rather than having the queues pile up while the CPU is hovering around 50% usage), which may be good or bad, but it did drain the queues from ~20k to 0 in a matter of minutes. | ||
== [[Postgresql]] == | == [[Postgresql]] == | ||
| Line 159: | Line 149: | ||
# OS Type: linux | # OS Type: linux | ||
# DB Type: web | # DB Type: web | ||
# Total Memory (RAM): | # Total Memory (RAM): 3 GB | ||
# CPUs num: 4 | # CPUs num: 4 | ||
# Connections num: 200 | # Connections num: 200 | ||
| Line 165: | Line 155: | ||
max_connections = 200 | max_connections = 200 | ||
shared_buffers = | shared_buffers = 768MB | ||
effective_cache_size = | effective_cache_size = 2304MB | ||
maintenance_work_mem = | maintenance_work_mem = 192MB | ||
checkpoint_completion_target = 0.9 | checkpoint_completion_target = 0.9 | ||
wal_buffers = 16MB | wal_buffers = 16MB | ||
| Line 173: | Line 163: | ||
random_page_cost = 1.1 | random_page_cost = 1.1 | ||
effective_io_concurrency = 200 | effective_io_concurrency = 200 | ||
work_mem = | work_mem = 1966kB | ||
huge_pages = off | huge_pages = off | ||
min_wal_size = 1GB | min_wal_size = 1GB | ||
| Line 189: | Line 179: | ||
* https://hazelweakly.me/blog/scaling-mastodon/ | * https://hazelweakly.me/blog/scaling-mastodon/ | ||
* https://www.digitalocean.com/community/tutorials/how-to-scale-your-mastodon-server | * https://www.digitalocean.com/community/tutorials/how-to-scale-your-mastodon-server | ||
* https://hub.sunny.garden/2023/07/08/sidekiq-tuning-small-mastodon-servers/ | |||
** https://sunny.garden/@brook/111475392515987172 - "you can probably reduce your total thread count / db connections considerably if you'd like" | |||
== See Also == | |||
* [[ElasticSearch]] - which needs plenty of perf tuning | |||
[[Category:Mastodon]] | [[Category:Mastodon]] | ||
[[Category:Admin]] | [[Category:Admin]] | ||
[[Category:Tech WG]] | [[Category:Tech WG]] | ||