146
edits
(sidekiq scheduler queue issues) |
(add link to Tech WG main page) |
||
(2 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
Up to [[Part Of::Tech_WG]] | |||
== 22-12-16 Upgrading Linode Ram == | == 22-12-16 Upgrading Linode Ram == | ||
Line 259: | Line 261: | ||
== 24-04-30 Sidekiq scheduler queue issues == | == 24-04-30 Sidekiq scheduler queue issues == | ||
Noticed this warning about sidekiq [[File:Sidekiq issues.png|frameless]] | Manisha: Noticed this warning about sidekiq [[File:Sidekiq issues.png|frameless]] | ||
And saw that we were hitting our swap limits. | And saw that we were hitting our swap limits. | ||
Line 266: | Line 268: | ||
[https://github.com/mastodon/mastodon/issues/20710#issuecomment-1329680431 Maybe this github discussion would help?] but on checking the sidekiq dashboard, while the no. of processed requests did plummet today, I do see sidekiq running. The warning about the scheduler queue(s) is puzzling as there are scheduled jobs in the sidekiq container! hmmm | [https://github.com/mastodon/mastodon/issues/20710#issuecomment-1329680431 Maybe this github discussion would help?] but on checking the sidekiq dashboard, while the no. of processed requests did plummet today, I do see sidekiq running. The warning about the scheduler queue(s) is puzzling as there are scheduled jobs in the sidekiq container! hmmm | ||
Something is definitely up. sidekiq's status is <code> active (exited)</code>. I restarted ES and that helped free up swap but restarting sideqik isn't helping and the warning isn't going away :( | |||
Jonny: we are sorta always at the limit of RAM and swap, cost of the ES service. disk is full! cron job likely not working | |||
Manisha: ran this <code> RAILS_ENV=production /home/mastodon/live/bin/tootctl media remove --remove-headers</code> as mastodon user. It freed up 10.5 GB and the sidekiq warning went away! :) |