wget | create static site copy – host on aws *

/ SprUng, KEEPERS, CODING, Websites - more cheats, wordpress - more cheats, TOOLS / 2 minutes of reading

https://korays.com/create-static-copy-of-your-website/

Let's create a static version of our website using the magical wget.

wget -k -K -E -r -l 10 -p -N -F --restrict-file-names=windows -nH https://korays.com/

-k : convert links to relative

-K : keep an original versions of files without the conversions made by wget

-E : rename html files to .html (if they don’t already have an htm(l) extension)

-r : recursive… of course we want to make a recursive copy

-l 10 : the maximum level of recursion. if you have a really big website you may need to put a higher number, but 10 levels should be enough.

-p : download all necessary files for each page (css, js, images)

-N : Turn on time-stamping.

-F : When input is read from a file, force it to be treated as an HTML file.

-nH : By default, wget put files in a directory named after the site’s hostname. This will disabled creating of those hostname directories and put everything in the current directory.

–restrict-file-names=windows : may be useful if you want to copy the files to a Windows PC. (source: http://blog.jphoude.qc.ca/2007/10/16/creating-static-copy-of-a-dynamic-website/)

Possible problems

» wget download the homagepage, robots.txt then stops!
Your robots.txt file probably denies access to your site to search engines. Yes, in recursive mode, wget will respect the robots.txt file, so you will need to remove it before making the copy. Don’t forget to put it back in the static site if that’s what you want.

» Stylesheets : if you have @import stylesheet imports, wget won’t see them, and won’t download them. You might want to change them to <link rel=”stylesheet” … /> imports, which wget will see and download.

» Stylesheet images : wget won’t download background-images referenced in CSS files. For most websites that should not be too long to download those images manually.

» Be sure that you CSS files and with “.css”! Apache won’t send the correct mime-type if your file extension is not .css, and Firefox will not use the stylesheet.

(test.css?color=blue won’t work, change it to test.css?color=blue&ext=.css)
The same problem may happen with other files types that need to have a proper mimetype set (video files, for instance)

» LinguaPlone specific problems » To prevent having several duplicated files with the set_language parameter, you could setup one subdomain for each language, and force the set_language= in the Apache redirect rule.

» I also recommand to change the language link so it points to the main page instead of the current page.

» You have several possibilities here, but by just doing a wget without changing anything, you may end up with pages where languages are a bit fucked up.

» tag problem : If you pages contains a base tag (which is true for Plone sites), wget will empty it’s value but leave the base tag there ([base href="" /]). That works in Firefox, but it will confuse IE, which won’t load any images, CSS or links.To fix it, you can remove the base tag completely with this command :

 find | grep html$ | xargs perl -i -p -e 's/<base href=\"\" \/>//g'

 

Downsides » Most file names will change (bad for SEO) » May take some manual work to have a working static copy

After taking care of all the possible problems, you should have a working static site! Be sure to check with both IE and Firefox (at least), because some problems happen in only one browser. Then, you can shut down your CMS and server the static content using a standard webserver.

Don’t forget to put a nice 404 page pointing to your main page, since your URLs probably changed, and several visitors will get a 404 error if they come from search engines or bookmarks.

Alıntı : https://www.quora.com/How-do-you-export-a-WordPress-site-to-a-static-HTML

Show more
Show less

Once done we will use sync command sync all the static content to our S3 bucket using AWS CLI.

Let's make a new bucket (reminder: bucket names are unique and global)

aws s3 mb s3://newbucketweb --region us-west-1

Let's sync up all content to the new bucket

aws s3 sync . s3://newbucketweb

Sync up might take sometime depending on the objects we are syncing to our bucket.

Now tricky part is to enable public access to this new bucket we created.

aws s3api put-public-access-block \    --bucket newbucketweb \    --public-access-block-configuration "BlockPublicAcls=false,IgnorePublicAcls=false,BlockPublicPolicy=false,RestrictPublicBuckets=false"

If you go to AWS console, you should see the bucket settings as below:

bucketsettings

Now need to enable public read to the bucket using this JSON schema.

aws s3api put-bucket-policy --bucket newbucketweb --policy "{  \"Version\": \"2012-10-17\",  \"Statement\": [      {          \"Sid\": \"PublicReadGetObject\",          \"Effect\": \"Allow\",          \"Principal\": \"*\",          \"Action\": \"s3:GetObject\",          \"Resource\": \"arn:aws:s3:::newbucketweb/*\"      }  ] }"

This will add above rule to the bucket policy. You can check this via AWS console under bucket policy.

Let's enable our index.html page

aws s3 website "s3://newbucketweb" --index-document index.html --error-document index.html

The publicly accessible (as per our configuration) bucket endpoint would be constructed as per S3 guidelines as below:

http://<bucketname>.s3-website-<region>.amazonaws.com

Congratulations! you have a static copy of your site running under an S3 bucket. You can use this static copy for many purposes such as using as a static DR site.

Scroll to Top