146
edits
No edit summary |
(add link to Tech WG main page) |
||
(15 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
Up to [[Part Of::Tech_WG]] | |||
== 22-12-16 Upgrading Linode Ram == | == 22-12-16 Upgrading Linode Ram == | ||
Line 117: | Line 119: | ||
|Link=https://discord.com/channels/1049136631065628772/1049184335514828860/1109397579508240404 | |Link=https://discord.com/channels/1049136631065628772/1049184335514828860/1109397579508240404 | ||
}} | }} | ||
== 23-05-22 == | |||
{{Message | |||
|Author=sneakers-the-rat | |||
|Avatar=https://cdn.discordapp.com/avatars/305044217393053697/2970b22bd769d0cd0ee1de79be500e85.png?size=1024 | |||
|Date Sent=23-05-23 01:35:39 | |||
|Channel=techwg-ops | |||
|Text=[[Tech WG/Ops Diary#23-05-22]]: What up it's me and we're about to deploy [[Exclusive Lists]] and update to 4.1.2. starting by making a linode backup and then following instructions on https://github.com/mastodon/mastodon/releases/tag/v4.1.2 | |||
|Link=https://discord.com/channels/1049136631065628772/1052452072806686751/1110380495335870505 | |||
}} | |||
== 23-06-27 == | |||
{{Message | |||
|Author=mannaz | |||
|Avatar=https://cdn.discordapp.com/avatars/540104559079456770/12ce492294d6a3cd6f1ea00cbe8edb04.png?size=1024 | |||
|Date Sent=23-06-27 12:20:04 | |||
|Channel=techwg-ops | |||
|Text=[[Tech WG/Ops Diary#23-06-27]]: restarted systems services as we were hitting swap limits https://neuromatch.social/@manisha/110616054514875151 This time I remembered to take a screenshot before and after | |||
|Link=https://discord.com/channels/1049136631065628772/1052452072806686751/1123226240069947422 | |||
}} | |||
== 23-07-07 == | |||
{{Message | |||
|Author=mannaz | |||
|Avatar=https://cdn.discordapp.com/avatars/540104559079456770/12ce492294d6a3cd6f1ea00cbe8edb04.png?size=1024 | |||
|Date Sent=23-07-07 18:47:50 | |||
|Channel=techwg-ops | |||
|Text=[[Tech WG/Ops Diary#23-07-07]]: masto 4.1.3 update -- security patch. Jonny's on the road so Lina and Manisha are going to give this a try. If we muck something up, please be kind 🥺 starting by making a linode backup and then following instructions on ... | |||
oh wait! dang they already released 4.1.4 just an hour ago | |||
https://github.com/mastodon/mastodon/releases/tag/v4.1.4 | |||
4.1.3 from yesterday: https://github.com/mastodon/mastodon/releases/tag/v4.1.3 | |||
|Link=https://discord.com/channels/1049136631065628772/1052452072806686751/1126947706162651136 | |||
}} | |||
== 23-07-09 == | |||
{{Message | |||
|Author=mannaz | |||
|Avatar=https://cdn.discordapp.com/avatars/540104559079456770/12ce492294d6a3cd6f1ea00cbe8edb04.png?size=1024 | |||
|Date Sent=23-07-10 00:35:37 | |||
|Channel=techwg-ops | |||
|Text=[[Tech WG/Ops Diary#23-07-09]]: Updating reverse proxy config to the recommended one | |||
as root | |||
cd /etc/nginx | |||
cd sites-enabled | |||
vim mastodon | |||
search for "system" | |||
added the following two headers in that block as per the release notes: | |||
`add_header X-Content-Type-Options nosniff; | |||
add_header Content-Security-Policy "default-src 'none'; form-action 'none'";` | |||
save | |||
test nginx configs are valid or not (w/o running) | |||
`nginx -t` | |||
all looked good | |||
`nginx: the configuration file /etc/nginx/nginx.conf syntax is ok | |||
nginx: configuration file /etc/nginx/nginx.conf test is successful` | |||
Reload nginx | |||
`nginx reload` | |||
|Link=https://discord.com/channels/1049136631065628772/1052452072806686751/1127760004980691054 | |||
}} | |||
== 23-09-19 == | |||
{{Message | |||
|Author=sneakers.the.rat | |||
|Avatar=https://cdn.discordapp.com/avatars/305044217393053697/2970b22bd769d0cd0ee1de79be500e85.png?size=1024 | |||
|Date Sent=23-09-20 02:55:44 | |||
|Channel=techwg-ops | |||
|Text=[[Tech WG/Ops Diary#23-09-19]] - Upgrading to 4.2-RC2 while trying to write the [[Upgrade Mastodon]] docs. currently doing the merge from upstream and noticing db migrations for exclusive lists failing, but not fixing now because that would put the fixes in the `glitch-soc-main` branch which is a local mirror of upstream which gets force pushed. Seems like that should happen in the `merge-upstream` branch after upstream is merged | |||
|Link=https://discord.com/channels/1049136631065628772/1052452072806686751/1153887188946268211 | |||
}} | |||
== 23-10-01 == | |||
{{Message | |||
|Author=mannazsci | |||
|Avatar=https://cdn.discordapp.com/avatars/540104559079456770/12ce492294d6a3cd6f1ea00cbe8edb04.png?size=1024 | |||
|Date Sent=23-10-01 07:44:40 | |||
|Channel=techwg-ops | |||
|Text=[[Tech WG/Ops Diary#23-10-01]] I am ready to deploy v4.2 but I encountered a few issues while following the process mentioned in [[Upgrade_Mastodon]]. I'll update the wiki with the steps I took. | |||
|Link=https://discord.com/channels/1049136631065628772/1052452072806686751/1157946171432632383 | |||
}} | |||
== 24-02-01 == | |||
{{Message | |||
|Author=sneakers.the.rat | |||
|Avatar=https://cdn.discordapp.com/avatars/305044217393053697/2970b22bd769d0cd0ee1de79be500e85.png?size=1024 | |||
|Date Sent=24-02-02 01:16:50 | |||
|Channel=techwg-ops | |||
|Text=[[Tech WG/Ops Diary#24-02-01]] - Patching some mysterious critical security vuln thread | |||
|Link=https://discord.com/channels/1049136631065628772/1052452072806686751/1202784662397911150 | |||
}} | |||
(The vulnerability in question) https://github.com/mastodon/mastodon/security/advisories/GHSA-3fjr-858r-92rw | |||
== 24-02-14 == | |||
{{Message | |||
|Author=sneakers.the.rat | |||
|Avatar=https://cdn.discordapp.com/avatars/305044217393053697/2970b22bd769d0cd0ee1de79be500e85.png?size=1024 | |||
|Date Sent=24-02-17 22:16:31 | |||
|Channel=techwg-ops | |||
|Text=[[Tech WG/Ops Diary#24-02-14]] - Patching two more security bugs thread! | |||
|Link=https://discord.com/channels/1049136631065628772/1052452072806686751/1208537492802904064 | |||
}} | |||
== 24-04-17 Elastic Search issues == | |||
Manisha: Saw a warning in the admin panel: | |||
<code> Could not connect to Elasticsearch. Please check that it is running, or disable full-text search </code> | |||
Checked ES status via root | |||
<code> service elasticsearch status</code> | |||
It showed [https://discord.com/channels/1049136631065628772/1049184335514828860/1230219622293438514 status Failed]. | |||
Restarted ES | |||
<code> service elasticsearch restart</code> | |||
Status returned to [https://discord.com/channels/1049136631065628772/1049184335514828860/1230219711543902328 active]. | |||
Had to wait a tiny bit before [https://discord.com/channels/1049136631065628772/1049184335514828860/1230220033481773087 the warning in the admin panel] disappeared. | |||
== 24-04-30 Sidekiq scheduler queue issues == | |||
Manisha: Noticed this warning about sidekiq [[File:Sidekiq issues.png|frameless]] | |||
And saw that we were hitting our swap limits. | |||
Restarted services. Sidekiq status is active but we are still near our swap limits and the sidqkiq warning in the admin panel hasn't gone away. [https://discord.com/channels/1049136631065628772/1049184335514828860/1235046062171885568 Link to discussion on the technical-wg channel]. | |||
[https://github.com/mastodon/mastodon/issues/20710#issuecomment-1329680431 Maybe this github discussion would help?] but on checking the sidekiq dashboard, while the no. of processed requests did plummet today, I do see sidekiq running. The warning about the scheduler queue(s) is puzzling as there are scheduled jobs in the sidekiq container! hmmm | |||
Something is definitely up. sidekiq's status is <code> active (exited)</code>. I restarted ES and that helped free up swap but restarting sideqik isn't helping and the warning isn't going away :( | |||
Jonny: we are sorta always at the limit of RAM and swap, cost of the ES service. disk is full! cron job likely not working | |||
Manisha: ran this <code> RAILS_ENV=production /home/mastodon/live/bin/tootctl media remove --remove-headers</code> as mastodon user. It freed up 10.5 GB and the sidekiq warning went away! :) |