1,035
edits
Performance: Difference between revisions
no edit summary
No edit summary |
No edit summary |
||
| Line 88: | Line 88: | ||
=== Separate Services === | === Separate Services === | ||
Even after increasing the number of worker threads to 75, we were still getting huge backlogs on our queues, particularly <code>pull</code> which was loading up with link crawl workers, presumably the slower jobs were getting in the way of faster jobs and they were piling up. | |||
We want to split up sidekiq into multiple processes using separate systemd service files. We want to a) make the site responsive by processing high-priority queues quickly but also b) use all our available resources by not having processes sit idle. So we give each of the main queues one service file that has that queue as the top prioriry, and mix the other queues in as secondary priorities - sidekiq will try and process items from the first queue first, second queue second, and so on. | We want to split up sidekiq into multiple processes using separate systemd service files. We want to a) make the site responsive by processing high-priority queues quickly but also b) use all our available resources by not having processes sit idle. So we give each of the main queues one service file that has that queue as the top prioriry, and mix the other queues in as secondary priorities - sidekiq will try and process items from the first queue first, second queue second, and so on. | ||
| Line 141: | Line 143: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
This lets sidekiq use all the available CPU (rather than having the queues pile up while the CPU is hovering around 50% usage), which may be good or bad, but it did drain the queues from ~20k to 0 in a matter of minutes. | |||