Offload Media + Files to S3 Steps

More from this category

GITHUB – WORDPRESS AWS TEMPLATES w/permissions/security/all settings ready to deploy

https://github.com/aws-samples/aws-refarch-wordpress

Cloudfront – Extended Configuration Info – AWS DOC

Level 1 Article

https://aws.amazon.com/cloudfront/getting-started/

Level 2 Article (applicable info)

Cloudfront conifg explanation – full article on aws (click here) | Adding security to aws (click here)

https://aws.amazon.com/blogs/startups/how-to-accelerate-your-wordpress-site-with-amazon-cloudfront/

Extra – Bitnami configuration cdn / is there an ubuntu?

Amazon S3 Quick Start Guide

1. Set up AIM USER
2. Set up bucket with permissions
3. Https and CName
4. Cron Job
5. Cloudfront setup for Media Items
6. Bucket Permissions
7. Upload Media
8. File CDN distribution settings – ASSETS PULL ADD-ON – Info on lauch of asset pull
   • Creating a custom origin to serve the assets
   • Edit the default cache behaviour of the distribution
   • Create a cache behaviour for the S3 origin
9. Assets pull testing / fix errors
10. Compatability additions

Introduction

This guide aims to help you start offloading your WordPress Media Library to an Amazon S3 bucket with WP Offload Media as quickly as possible.

We also have Quick Start Guides for other storage providers.

This article covers the following steps:

1. Login to the AWS Console
2. Create an Amazon Web Services (AWS) User
3. Saving the AWS User’s access details for WP Offload Media to use
4. Activate Your WP Offload Media License
5. Configure WP Offload Media to offload newly uploaded media to an Amazon S3 bucket
6. Offload Your Existing Media Library
7. Next steps for setting up WP Offload Media to use a CDN

Login to the AWS Console

Already have an Amazon Web Services (AWS) account? Sign in here.

If you don’t have an AWS account yet, you will need to sign up here.

Create an IAM User

Once you have logged into the console, you will need to create a new IAM user:

Navigate to the IAM Users page in the AWS Console, click the Add user button.

Enter a name for the user in the User name field.

Names are case insensitive and must be unique within your AWS account. User names can be a combination of up to 64 letters, digits, and these characters: plus (+), equal (=), comma (,), period (.), at sign (@), and hyphen (-).

Under Access type select the checkbox for Programmatic access, then click the Next: Permissions button.

To allow the new user to manage buckets and objects in the S3 service, you need to grant it specific permissions.

Click the “Attach existing policies directly” button, and then enter “s3” in the filter policies input box.

Select the “AmazonS3FullAccess” policy, then click the Next: Review button at the bottom of the page.

If everything looks good, click the Create user button.

You will be shown the security credentials for the user, which consists of an Access Key ID and a Secret Access Key. Amazon will not show these again so please download them as a .csv and also copy them somewhere safe. If you lose them, you can always create a new set of keys from the console but you cannot retrieve the secret key again later.

Define Your Access Keys

Now that you have your AWS Access Keys, you need to add them to your site so that WP Offload Media can use them to work with the Amazon S3 service.

For better security, we recommend defining your access keys in your wp-config.php:

define( 'AS3CF_SETTINGS', serialize( array(
    'provider' => 'aws',
    'access-key-id' => '********************',
    'secret-access-key' => '**************************************',
) ) );

These should be placed before the following block of code in your wp-config.php:

/* That's all, stop editing! Happy blogging. */

/** Absolute path to the WordPress directory. */
if ( !defined('ABSPATH') )
    define('ABSPATH', dirname(__FILE__) . '/');

/** Sets up WordPress vars and included files. */
require_once(ABSPATH . 'wp-settings.php');

Warning: If you define your access keys after this block of code, WP Offload Media will not be able to read them.

IAM Roles

If you’re running your site on an Amazon EC2 instance, you might like to use an IAM Role instead. This is even more secure than defining your access keys in wp-config.php.

Questions?

Each of the above quick start guides should help you with getting started with WP Offload Media but if you run into any issues, you can search these docs via the search bar above.

Bucket set up

Configure WP Offload Media

To start offloading newly uploaded media to Amazon S3 you need to first tell WP Offload Media which bucket it should use. If you have not already created a bucket to use with WP Offload Media, it can be created for you from WP Offload Media’s Settings page.

Go to the “Media Library” tab of WP Offload Media’s Settings page in the WordPress admin dashboard. If you have followed the steps above then you should see something like the following.

If you have already created the bucket you want to offload your media to, then you can enter the bucket name in the above input and click the “Save Bucket” button.

If you haven’t created the bucket yet, then you can click the “Create new bucket” link to get a form where you can specify the new bucket’s name and region.

As long as you haven’t restricted your IAM User’s access to list buckets then you can also “Browse existing buckets.”

Regardless of how you specify the bucket, once saved, WP Offload Media will be set up to offload newly-uploaded media to the bucket with some recommended default settings.

this sounds strange to me? but apparently we already did it

Bucket Permissions

Access Denied to Bucket

If you see an “Access Denied to Bucket” error notice after selecting a bucket created in the AWS Console, the chances are you have “Public Access Settings” enabled for the bucket. You can tell by selecting the bucket in the AWS Console’s list of buckets and looking at the “Public access settings” option within the “Permissions” section of the pop-over panel.

WP Offload Media needs to be able to make objects in the bucket public readable (not editable) so that your site visitors can see them, this is safe to do as it is on a per object basis and is different from making the bucket public. Even if you’re only going to use WP Offload Media to offload “private” objects for WooCommerce, Easy Digital Downloads or similar, it still needs to be able to change the permissions on individual objects in the bucket.

If “Public access settings” does not say “Disabled“, click the “Permissions” section link in the pop-over panel and click “Edit”.

Uncheck all of the settings and save the changes, you’ll need to confirm the change if asked.

Refresh WP Offload Media’s settings page and the “Access Denied to Bucket” error should disappear.

IMPORTANT: There is also a Block public access (account settings) page that can be accessed from the Amazon S3 sidebar that can also be used to disable public access to objects in all existing and new buckets. If the AWS account is relatively new, or someone has visited this settings page and turned on the blocking of public access, you may need to visit the page, click “Edit”, clear the “Block all public access” checkbox, and then save the changes.

HTTPS

Configure a Custom Domain for CloudFront with HTTPS

This article assumes that you’ve already configured CloudFront, as detailed in our CloudFront Setup guide.

Generate a Certificate

Log into the AWS console and visit the AWS Certificate Manager.

IMPORTANT: Make sure you switch to the “US East (N. Virginia)” region using the region selector at the top right, CloudFront only works with certificates that are created in that “global” region.

Click ‘Get Started’ and you will be prompted to enter your first domain.

Enter your domain and any subdomains you want the certificate to cover, or enter a wildcard as I have. Click ‘Review and request’ followed by ‘Confirm and request’. Shortly after you should receive an authorization email.

Once you approve the request your certificate is ready to use.

Adding the Certificate to CloudFront

Open the CloudFront console page and select your distribution.

Click ‘Edit’ and add your custom domain to the ‘Alternate Domain Names’ option. Change the ‘SSL Certificate’ to ‘Custom SSL Certificate’ and select your previously created certificate from the drop-down list.

Make a note of your unique CloudFront URL and log into your DNS control panel. Add a new CNAME entry that points to your CloudFront domain. This entry should match that entered in the ‘Alternate Domain Names’ from within the CloudFront console.

Add Your Custom Domain to WP Offload Media

Log into your WordPress dashboard and open the WP Offload Media settings screen. Change the ‘Domain’ option to ‘CloudFront or Custom Domain’ and enter your new custom domain. If you’re using the Assets addon remember to also update the ‘CloudFront or Custom Domain’ option on the Assets tab.

Now if you load the front-end of your site and open the developer tools you should see that the URLs point to your custom domain.

SET UP CRON JOB

Make Sure Cron Is Set Up

We highly recommend that you configure a proper cron job on your server as WP Offload Media relies heavily on background processing. See our Cron Setup doc for details on how to accomplish this.

Cron Setup

WP Offload Media relies heavily on cron functioning correctly for background processing. Out of the box WordPress checks for cron jobs on every single page load by firing a request off to wp-cron.php, which is very inaccurate. If a site doesn’t receive any visits for a set amount of time, cron will not run during this period. This is also true of sites that implement full page caching, as WordPress is generally bypassed. Basic auth and some security plugins can also restrict cron if not correctly configured.

To ensure that cron runs more often, you need another method of hitting wp-cron.php. If your site is running on Linux and you have access to SSH you can you use the built in crontab. If not, a cron service can be used.

Regardless of which method you choose, a cron schedule of 5 minutes or less is recommended.

Cron Services

There are various cron services available and most offer a free plan, however, the free plans usually limit the frequency in which cron jobs can be scheduled.

www.easycron.com
www.setcronjob.com

Signup for an account at your chosen provider and proceed to create your first cron job.

Enter the URL to your site’s cron file. The free plan at www.easycron.com limits cron jobs to 10 minute intervals.

Cron will now fire every 10 minutes, regardless of whether or not any visits are received.

Crontab

SSH to your server.

ssh username@server.com

Open the crontab. If this is the first time, you may be asked to select an editor. Option 2 (nano) is usually the easiest.

crontab -e

Add a new line to the end of the file, ensuring that you specify the absolute file path to your WordPress install.

*/5 * * * * cd /var/www/example.com/public; php -q wp-cron.php >/dev/null 2>&1

Alternatively, you can use cURL if PHP CLI causes you issues.

*/5 * * * * curl http://example.com/wp-cron.php?doing_wp_cron >/dev/null 2>&1

Hit CTRL-X followed by Y to save and exit. Cron will now fire every 5 minutes, regardless of whether or not your site receives any visits.

Testing Cron

You can test that cron is correctly configured by installing WP Crontrol. Once installed and activated head to Tools > Cron Events.

Each scheduled event will have it’s next run time displayed. An event which has now in the next run column is due for processing and should be handled on the next cron tick. If the next run time doesn’t update and the cron interval you have configured has elapsed, then cron jobs are not being processed. You need to check that your cron job is configured correctly, as detailed above.

Disable WP Cron

On high traffic sites, you may wish to disable WP Cron once you’ve configured a custom cron job and confirmed that it’s working correctly. To do so add the following line to your wp-config.php file:

define('DISABLE_WP_CRON', true);

Once set, WordPress will no longer fire off a request to wp-cron.php on every page request.

MEDIA ONLY::

Amazon S3 Quick Start Guide

CloudFront Setup for Media Offloaded to Amazon S3

Once the Status has changed from In Progress to Deployed, you can configure WP Offload Media to start serving from CloudFront using the cloudfront.net domain provided. However, we highly recommend that you configure a custom domain for your CloudFront distribution. Using the cloudfront.net domain name is not great for your site’s SEO. See the doc Configure a Custom Domain for CloudFront with HTTPS for step-by-step instructions.

CloudFront Setup for Media Offloaded to Amazon S3

This article assumes that you’ve already set up an Amazon S3 bucket and followed the steps within our Amazon S3 Quick Start Guide.

Once the above steps are taken care of, login into the AWS console and visit the CloudFront page. Click Create Distribution followed by Get Started under the Web delivery method.

Populate Origin Domain Name with the desired S3 bucket, however, leave Origin Path blank. Feel free to customize the remaining options if required, but the defaults are fine in most cases.

Click Create Distribution. CloudFront will begin copying all of your files from the specified bucket to its edge servers. This can take a while depending on the size of your Media Library.

Once the Status has changed from In Progress to Deployed, you can configure WP Offload Media to start serving from CloudFront using the cloudfront.net domain provided. However, we highly recommend that you configure a custom domain for your CloudFront distribution.

Our Cloudfront Distributions

Using the cloudfront.net domain name is not great for your site’s SEO. See the doc Configure a Custom Domain for CloudFront with HTTPS for step-by-step instructions.

If you are fine with using the cloudfront.net domain name, head over to your WordPress site. Visit the WP Offload Media screen and enter your cloudfront.net domain name into the Custom Domain (CNAME) option.

Click Save Changes. All of your media library files will now be served from the CloudFront distributed servers.

Offload Your Existing Media Library

Now that your site is configured to offload newly-uploaded media to Amazon S3, you might want to offload any existing media to the bucket too.

In the right-hand sidebar of WP Offload Media’s “Media Library” tab you should see what we call the “Offload Tool”, this can be used to offload existing media to the bucket.

If you have not offloaded anything yet, then you’ll see a “Your Media Library needs to be offloaded” message in the Offload Tool with an “Offload Now” button.

If you have already offloaded a few items, then you might see a message like “66% of your Media Library has been offloaded” in the Offload Tool, with an “Offload Remaining Now” button.

Either way, clicking the Offload Tool’s button will start an upload to the bucket of all existing media yet to be offloaded.

The media offload progresses in small batches in the background to keep things efficient. You can close the tab and just forget about it, when the tool finishes its job, you’ll get a notice in the WordPress dashboard, or you can monitor progress from WP Offload Media’s settings page.

Once complete, the Offload Tool displays a “100% of your Media Library has been offloaded, congratulations!” message and the “Download all files from bucket to server” and “Remove all files from bucket” tools are made available.

Sending our images up in versioned folder

Set up File CDN

Use the Same CloudFront Custom Domain with WP Offload Media and the Assets Pull Addon

If you use WP Offload Media with CloudFront and a custom domain with HTTPS, then you might want to use the same domain and CloudFront distribution for the Assets Pull addon as well to serve your site’s assets.

Using the same domain will have a performance benefit over two DNS entries, as there will be only one open connection which will utilize the multiplexing feature of HTTP/2 to make requests asynchronously.

This guide assumes you have already set up a CloudFront distribution with a custom domain running on HTTPS. The only origin of the distribution is currently pointing to the S3 bucket. The article covers setting up one CloudFront distribution to serve both your media library and site assets with the following steps:

1. Creating a custom origin to serve the assets
2. Edit the default cache behaviour of the distribution
3. Create a cache behaviour for the S3 origin

Assets Pull Addon – Steps shortened

NOTES::

assets.nakedsword.com – I searched my email – no mention of it before this? so not sure where it came from? It is possible it was not set up for this?

the site being on sworddevelop.wpengine.com – do you guys think this could be an issue?
– I asked if we could give it a non-wpengine ulr – the only way is to move this site into a new install and put it at production – that is the only site in wpengine that can have own url.

Origin Setting – aws docs – perhaps we will need to change this section – /DeveloperGuide/using-https-cloudfront-to-custom-origin.html – I do not know what we would use –

STEPS::

1. domain to pull assets – any sub-domain that points to sworddevelop.wpengine.com

2. cdn.thesword.com + set up a wildcard domain for assets
   – here we may have a problem – it is on wpengine.com –
   BUT can we use *thesword.com?
   for example – anything-at.thesword.com points to sworddevelop.wpengine.com? and will that work? (url issue) – not sure what the plugin will do honestly.

3. CloudFront distribution to “pull” and cache assets from site on-demand –

   1. For Origin Domain Name, enter sworddevelop.wpengine.com
   2. For Origin Protocol Policy, select Match Viewer.

   Everything else within the Origin Settings section can be left at the default value.

   AWS – other cloudfront custom origin settings

4. Default Cache Behavior Settings – control what CloudFront will do with the response it gets from sworddevelop.wpengine.com

   1. Query String Forwarding and Caching, select Forward all, cache based on all.
      This is necessary to ensure the proper version of an asset is served when WordPress, plugins and themes are updated.

   2. For Compress Objects Automatically, choose Yes.

5. Distribution Settings –

   1. Adjust the Price Class based on your requirements and budget. Use All Edge Locations – best performance, but will cost slightly more.
   2. For Alternate Domain Names, enter cdn.thesword.com into the field.
   3. SSL – says choose custom ssl
   4. Select your certificate from the dropdown menu (indicated by *.thesword.com).

6. Cloudfront Domain – CloudFront will begin to deploy your new distribution – can take some time depending on the Price Class chosen

7. Save your distribution’s unique CloudFront Domain Name for use in the next step. E.g. dXXXXXXXXXXXXX.cloudfront.net

8. CNAME Setup – Set DNS for your chosen subdomain so assets are served through CloudFront using your own domain –

   1. Create a CNAME record for the “cdn” subdomain to your CloudFront distribution’s domain name.
      NOTE FROM AWS IN SUBDOMAIN ARE
      You must list any custom domain names (for example, www.example.com) that you use in addition to the CloudFront domain name (for example, d1234.cloudfront.net) for the URLs for your files. Specify up to 100 CNAMEs separated with commas or put each on a new line. You also must create a CNAME record with your DNS service to route queries for www.example.com to d1234.cloudfront.net. For more information, see the Help.

9. ERROR CODES: (click for doc)

10. BONUS – additional pieces for coors

EXTENDED VERSION

The Assets addon will guide you through the set up of Amazon CloudFront. Once complete, it will start rewriting URLs to your enqueued styles and scripts so that they are served from CloudFront rather than your server.

If you prefer to use a CDN other than CloudFront, you can skip the set up, go directly to the settings page, and enter the domain name you’ve configured with your other CDN.

CloudFront Set Up

The following steps will walk you through the process of setting up the Assets addon with Amazon CloudFront.
Each step is clearly outlined with screenshots to guide you through these 3 basic processes:

• Choosing and configuring your domain name to use for serving your assets
• Requesting a free SSL certificate through the Amazon Certificate Manager
• Creating and configuring a new CloudFront distribution to serve your assets

The whole process usually takes about 15-30 minutes, depending on your pace, and external factors (such as DNS propagation).

When you’re ready to begin, click Get Started below.

What domain name would you like to serve your assets from?

could be any sub-domain that points to develop.wpengine.com

We highly recommend using a subdomain of wpengine.com for better
SEO.

Step 5 of 12

CloudFront Distribution

Now we will create a CloudFront distribution which will “pull” and cache assets from your site on-demand.

1. Navigate to the CloudFront service.
2. Click the Create Distribution button.
3. Under Web, click the Get Started button.

Step 6 of 12

The Origin settings control how the distribution will access your site for any assets which CloudFront does not have a cached version to serve.

1. For Origin Domain Name, enter sworddevelop.wpengine.com
2. For Origin Protocol Policy, select Match Viewer.

Everything else within the Origin Settings section can be left at the default value.

Step 7 of 12

The Behavior settings control what CloudFront will do with the response it gets from the Origin (your site).

1. Scroll to the Default Cache Behavior Settings section.
2. For Query String Forwarding and Caching, select Forward all, cache based on all.
   This is necessary to ensure the proper version of an asset is served when WordPress, plugins and themes are updated.
3. For Compress Objects Automatically, choose Yes.
   CloudFront will serve compressed versions of your assets to browsers that support it and uncompressed versions to browsers that don’t. This can significantly reduce the size of the data needed to be downloaded by the browser making for faster load times.

Step 8 of 12

1. Scroll to the Distribution Settings section.
2. Adjust the Price Class based on your requirements and budget.
   
   Choosing Use All Edge Locations will give you the best performance, but will
   cost slightly more.
   
3. For Alternate Domain Names, enter cdn.thesword.com
   into the field.
   NOTE FROM AWS – entering the origin domain again may not be necessary
   You must list any custom domain names (for example, www.example.com) that you use in addition to the CloudFront domain name (for example, d1234.cloudfront.net) for the URLs for your files. Specify up to 100 CNAMEs separated with commas or put each on a new line. You also must create a CNAME record with your DNS service to route queries for www.example.com to d1234.cloudfront.net. For more information, see the Help.
4. For SSL Certificate, select Custom SSL Certificate.
5. Select your certificate from the dropdown menu (indicated by *.thesword.com).
   If you do not see a certificate to select for your domain, revisit the steps for
   requesting a certificate and
   approving the certificate before continuing.
6. Complete the configuration by scrolling to the end of the page and clicking the Create Distribution button in the bottom right corner.

Step 9 of 12

CloudFront will begin to deploy your new distribution (indicated by the spinning arrows in the Status column), but this can take some time depending on the
Price Class chosen in the previous step.

1. Save your distribution’s unique CloudFront Domain Name for use in the next step.
   E.g. dXXXXXXXXXXXXX.cloudfront.net

Step 10 of 12

Now we are going to configure the DNS for your chosen subdomain so assets are served through CloudFront using your own domain.

1. Log into your DNS provider, and navigate to the DNS records for the thesword.com domain.
   
   Steps will vary by provider.
2. Create a CNAME record for the cdn subdomain to your CloudFront distribution’s domain name.

Step 11 of 12

If your theme uses web fonts you may experience issues with CORS.
If you notice that font files aren’t loading correctly, or receive the following error in the browser console then you need to modify your server configuration.

Please refer to this guide on how to configure your web server. https://deliciousbrains.com/wp-offload-media/doc/configure-cors-to-resolve-web-font-issues/

Step 12 of 12

Use the Same CloudFront Custom Domain with WP Offload Media and the Assets Pull Addon

If you use WP Offload Media with CloudFront and a custom domain with HTTPS, then you might want to use the same domain and CloudFront distribution for the Assets Pull addon as well to serve your site’s assets.

Using the same domain will have a performance benefit over two DNS entries, as there will be only one open connection which will utilize the multiplexing feature of HTTP/2 to make requests asynchronously.

This guide assumes you have already set up a CloudFront distribution with a custom domain running on HTTPS. The only origin of the distribution is currently pointing to the S3 bucket. The article covers setting up one CloudFront distribution to serve both your media library and site assets with the following steps:

1. Creating a custom origin to serve the assets
2. Edit the default cache behaviour of the distribution
3. Create a cache behaviour for the S3 origin

Creating a Custom origin to Serve the Assets

First create a new origin that will serve all the assets from the site as part of the Assets Pull addon. Edit your CloudFront distribution and select the Origins tab and click Create Origin. Set the Origin Domain Name to the domain of your site and set the Origin Protocol Policy to ‘Match Viewer’. Leave the other settings as they are:

Edit the Default Cache Behaviour of the Distribution

Next we need to edit the default behaviour for the distribution to point to the new origin. Select the Behavious tab, select the existing behaviour and click the Edit button:

Change the Origin to the new Custom origin we just created and ensure you turn on Compress Objects Automatically and set Query String Forwarding and Caching to ‘Forward all, cache based on all’.

Create a Cache Behaviour for the S3 Origin

Next add a new behaviour for the S3 origin for serving the site’s media files.

1. Click Create Behaviour
2. Set Origin as the S3 origin
3. Edit the Path Pattern to match the uploads directory of your site with a wildcard for all files under that path, eg. wp-content/uploads/*
4. Leave all the other settings as they are

Your behaviours should look like this:

Assets Pull: Configure CORS to Resolve Web Font Issues

If you notice that font files aren’t loading correctly, or receive the following error in the browser console then you need to modify your server configuration.

This is because web fonts are subject to Cross-Origin Resource Sharing (CORS). CORS is a way for a remote host to control access to certain types of resources.

To resolve this issue you need to ensure that your server is sending the correct Access-Control-Allow-Origin header when font files are requested. If you’re unable to modify your server configuration please contact your web host and link them to this article.

Apache

Add the following block to your .htaccess file:

<FilesMatch ".(eot|otf|ttf|woff|woff2)">
    Header always set Access-Control-Allow-Origin "*"
</FilesMatch>

Nginx

Add the following location block to your virtual host file (usually within the server directive) and reload Nginx:

location ~* .(eot|otf|ttf|woff|woff2)$ {
    add_header Access-Control-Allow-Origin *;
}

Invalidate CloudFront Distribution

Once the changes have been made you will need to invalidate the CloudFront cache with a path of “/*”.

Round 1 Steps – assets pull

Creating a Custom origin to Serve the Assets

First create a new origin that will serve all the assets from the site as part of the Assets Pull addon. Edit your CloudFront distribution and select the Origins tab and click Create Origin. Set the Origin Domain Name to the domain of your site and set the Origin Protocol Policy to ‘Match Viewer’. Leave the other settings as they are:

Edit the Default Cache Behaviour of the Distribution

Next we need to edit the default behaviour for the distribution to point to the new origin. Select the Behavious tab, select the existing behaviour and click the Edit button:

Change the Origin to the new Custom origin we just created and ensure you turn on Compress Objects Automatically and set Query String Forwarding and Caching to ‘Forward all, cache based on all’.

Create a Cache Behaviour for the S3 Origin

Next add a new behaviour for the S3 origin for serving the site’s media files.

1. Click Create Behaviour
2. Set Origin as the S3 origin
3. Edit the Path Pattern to match the uploads directory of your site with a wildcard for all files under that path, eg. wp-content/uploads/*
4. Leave all the other settings as they are

Your behaviours should look like this: