146
edits
Tech WG/Ops Diary: Difference between revisions
→24-04-30 Sidekiq scheduler queue issues
(ES restart) |
|||
| (One intermediate revision by the same user not shown) | |||
| Line 237: | Line 237: | ||
}} | }} | ||
== 24-04-17 == | == 24-04-17 Elastic Search issues == | ||
Manisha: Saw a warning in the admin panel: | Manisha: Saw a warning in the admin panel: | ||
| Line 256: | Line 256: | ||
Had to wait a tiny bit before [https://discord.com/channels/1049136631065628772/1049184335514828860/1230220033481773087 the warning in the admin panel] disappeared. | Had to wait a tiny bit before [https://discord.com/channels/1049136631065628772/1049184335514828860/1230220033481773087 the warning in the admin panel] disappeared. | ||
== 24-04-30 Sidekiq scheduler queue issues == | |||
Noticed this warning about sidekiq [[File:Sidekiq issues.png|frameless]] | |||
And saw that we were hitting our swap limits. | |||
Restarted services. Sidekiq status is active but we are still near our swap limits and the sidqkiq warning in the admin panel hasn't gone away. [https://discord.com/channels/1049136631065628772/1049184335514828860/1235046062171885568 Link to discussion on the technical-wg channel]. | |||
[https://github.com/mastodon/mastodon/issues/20710#issuecomment-1329680431 Maybe this github discussion would help?] but on checking the sidekiq dashboard, while the no. of processed requests did plummet today, I do see sidekiq running. The warning about the scheduler queue(s) is puzzling as there are scheduled jobs in the sidekiq container! hmmm | |||
Looking closely at the service status, something is definitely up. sidekiq and streaming's status is <code> active (exited)</code>. I restarted ES and that helped free up swap but restarting sideqik and streaming isn't helping :( | |||