1,035
edits
Performance: Difference between revisions
→Separate Services
| Line 93: | Line 93: | ||
We want to split up sidekiq into multiple processes using separate systemd service files. We want to a) make the site responsive by processing high-priority queues quickly but also b) use all our available resources by not having processes sit idle. So we give each of the main queues one service file that has that queue as the top prioriry, and mix the other queues in as secondary priorities - sidekiq will try and process items from the first queue first, second queue second, and so on. | We want to split up sidekiq into multiple processes using separate systemd service files. We want to a) make the site responsive by processing high-priority queues quickly but also b) use all our available resources by not having processes sit idle. So we give each of the main queues one service file that has that queue as the top prioriry, and mix the other queues in as secondary priorities - sidekiq will try and process items from the first queue first, second queue second, and so on. | ||
So we allocate 25 threads (and 25 db connections) each to four service files with the following priority orders. Note that we '''only do this after increasing the maximum postgres connections to 200,''' see https://hazelweakly.me/blog/scaling-mastodon/#db_pool-notes-from-nora's-blog | So we allocate 25 threads (and 25 db connections) each to four service files with the following priority orders, and two additional service files that give 5 threads to the lower-priority queues. Note that we '''only do this after increasing the maximum postgres connections to 200,''' see https://hazelweakly.me/blog/scaling-mastodon/#db_pool-notes-from-nora's-blog | ||
{|class="wikitable" | |||
! Service Name !! Queues !! Threads | |||
|- | |||
| <code>mastodon-sidekiq-default</code> || default, ingress, pull, push || 25 | |||
|- | |||
| <code>mastodon-sidekiq-ingress</code> || ingress, default, push, pull || 25 | |||
|- | |||
| <code>mastodon-sidekiq-push</code> || push, pull, default, ingress || 25 | |||
|- | |||
| <code>mastodon-sidekiq-pull</code> || pull, push, default, ingress || 25 | |||
|- | |||
| <code>mastodon-sidekiq-mailers</code> || mailers || 5 | |||
|- | |||
| <code>mastodon-sidekiq-scheduler</code> || scheduler || 5 | |||
|} | |||
Each service file is identical except for this part. (We didn't use the <code>@.service</code> systemd templates because we couldn't find a nice way of doing a list of parameters that could handle multiple queues and variable thread numbers in different services): | |||
<syntaxhighlight lang="ini"> | <syntaxhighlight lang="ini"> | ||
| Line 144: | Line 150: | ||
This lets sidekiq use all the available CPU (rather than having the queues pile up while the CPU is hovering around 50% usage), which may be good or bad, but it did drain the queues from ~20k to 0 in a matter of minutes. | This lets sidekiq use all the available CPU (rather than having the queues pile up while the CPU is hovering around 50% usage), which may be good or bad, but it did drain the queues from ~20k to 0 in a matter of minutes. | ||
== [[Postgresql]] == | == [[Postgresql]] == | ||