Jump to content

Storage: Difference between revisions

Storage (view source)

Revision as of 02:26, 29 August 2024

3,230 bytes added ,  29 August
no edit summary
No edit summary
Line 45: Line 45:
First we set up the reverse proxy as described in the linode and masto docs - we modified the nginx configuration slightly to use a different cache than the regular masto cache, and because you can't declare a <code>proxy_cache_path</code> within the server block as the linode docs have.
First we set up the reverse proxy as described in the linode and masto docs - we modified the nginx configuration slightly to use a different cache than the regular masto cache, and because you can't declare a <code>proxy_cache_path</code> within the server block as the linode docs have.


<pre>
The linode docs are also incorrect in their spec of the <code>proxy_set_header Host</code> directive. So our full config is below:
proxy_cache_path /var/cache/nginx-object-storage keys_zone=CACHEOBJECT:10m inactive=7d max_size=10g;


<syntaxhighlight lang="nginx">
server {
server {
   # ...
   listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name media.neuromatch.social;
  root /var/www/html;
    ssl_certificate /etc/letsencrypt/live/media.neuromatch.social/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/media.neuromatch.social/privkey.pem; # managed by Certbot
 
  keepalive_timeout 30;
 
  location = / {
    index index.html;
  }
 
  location / {
    try_files $uri @s3;
  }
 
  set $s3_backend 'https://neuromatchstodon.us-east-1.linodeobjects.com';
 
   location @s3 {
   location @s3 {
     # ...
     limit_except GET {
      deny all;
    }
 
    resolver 8.8.8.8;
    proxy_set_header Host neuromatchstodon.us-east-1.linodeobjects.com;
    proxy_set_header Connection '';
    proxy_set_header Authorization '';
    proxy_hide_header Set-Cookie;
    proxy_hide_header 'Access-Control-Allow-Origin';
    proxy_hide_header 'Access-Control-Allow-Methods';
    proxy_hide_header 'Access-Control-Allow-Headers';
    proxy_hide_header x-amz-id-2;
    proxy_hide_header x-amz-request-id;
    proxy_hide_header x-amz-meta-server-side-encryption;
    proxy_hide_header x-amz-server-side-encryption;
    proxy_hide_header x-amz-bucket-region;
    proxy_hide_header x-amzn-requestid;
    proxy_ignore_headers Set-Cookie;
    proxy_pass $s3_backend$uri;
    proxy_intercept_errors off;
 
     proxy_cache CACHEOBJECT;
     proxy_cache CACHEOBJECT;
     # ...
     proxy_cache_valid 200 48h;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    proxy_cache_lock on;
 
    expires 1y;
    add_header Cache-Control public;
    add_header 'Access-Control-Allow-Origin' '*';
    add_header X-Cache-Status $upstream_cache_status;
    add_header X-Content-Type-Options nosniff;
    add_header Content-Security-Policy "default-src 'none'; form-action 'none'";
   }
   }
}
}
</pre>
</syntaxhighlight>
 
=== Configuration ===
 
Like with nginx, linode's docs are slightly wrong, our config looks like:
 
<syntaxhighlight lang="env">
S3_ENABLED=true
S3_PROTOCOL=https
S3_BUCKET=neuromatchstodon
S3_REGION=us-east-1
S3_HOSTNAME=neuromatchstodon.us-east-1.linodeobjects.com
S3_ENDPOINT=https://us-east-1.linodeobjects.com/
AWS_ACCESS_KEY_ID={{ OUR KEY ID }}
AWS_SECRET_ACCESS_KEY={{ OUR SECRET KEY }}
S3_ALIAS_HOST=media.neuromatch.social
</syntaxhighlight>
 
Until you get your existing files transferred to the bucket, keep <code>S3_ENABLED=false</code>.


=== Transition ===
=== Transition ===
Line 92: Line 159:
* ''keys happen''
* ''keys happen''
* Select s3, configure for current bucket (see linode dashboard)
* Select s3, configure for current bucket (see linode dashboard)
* Select "private" ACL
 
 
<div style="background-color:aliceblue;border:5px solid CornflowerBlue;padding:1em;border-radius:10px;box-shadow: 3px 3px 0px blue; margin: 1em;">
<strong>HUGE WASTE OF TIME WARNING:</strong> the default ACL for rclone is <code>private</code>, but mastodon requires objects to be public. You have to set a [https://stackoverflow.com/a/52852181/13113166 global config] in <code>rclone.config</code> like
 
<pre>
[spaces]
acl = public-read
</pre>
 
before you copy/sync. If you do manage to copy all your files in private mode, you can [https://techdocs.akamai.com/cloud-computing/docs/using-s3cmd-with-object-storage use s3cmd afterwards] to [https://www.linode.com/community/questions/20759/how-can-i-change-all-existing-objects-in-my-bucket-to-public-or-private-using-s3 change all their ACLs like ]
 
<syntaxhighlight lang="shell">
s3cmd setacl s3://bucketname --acl-public --recursive
</syntaxhighlight>
</div>


Now copy the data to the bucket!
Now copy the data to the bucket!
Line 161: Line 243:
</pre>
</pre>


==== Nginx ====
==== [[Related To::Nginx]] ====


See [[Wiki/Nginx#Log_Rotation]]
See [[Wiki/Nginx#Log_Rotation]]
Line 168: Line 250:


* Helpful info on managing disk usage with [[tootctl]] - https://ricard.dev/improving-mastodons-disk-usage/
* Helpful info on managing disk usage with [[tootctl]] - https://ricard.dev/improving-mastodons-disk-usage/
[[Category:Nginx]]
Retrieved from "https://wiki.neuromatch.social/Special:MobileDiff/1276"