Build Folder Control
scanning the project and the build fodler almost surely will cause duplicate results – but there also might be some pieces that are NOT in the build folder that could still cause issues.2 master scripts pass build argument – individual scrips simply need true / false.
This is the content that could be used for how_to.txt
Individual Script:
find_all_missing_images_check.sh
now takestrue
orfalse
to control the exclusion of thebuild
folder. Basic template to update all script for project_check_scripts, which includes ./seo_tag_check.sh ./unused_file_check.sh ./broken_link_check.sh ./js_console_errors_check.sh ./css_js_minification_check.sh ./duplicate_image_check.sh ./outdated_package_version_check.sh ./env_config_check.sh ./find_all_missing_images_check.shMaster Scripts
run_all_checks_exclude_build.sh
usestrue
to exclude thebuild
folder across all scripts.run_all_checks_include_build.sh
usesfalse
to include thebuild
folder across all scripts.
Output: Each master script generates a result file (
results_exclude_build.txt
orresults_include_build.txt
) summarizing the checks.
With these final versions, you’re ready to apply this logic to the other scripts and run either of the master scripts to complete all checks.
By running each master script separately (with or without the build
folder), you generate two separate result files, which you can then compare using diff
:
Running and Comparing Results
Run the Scripts:
To check without the build folder:
./run_all_checks_exclude_build.sh /path/to/project
This saves results to
results_exclude_build.txt
.To check with the build folder:
./run_all_checks_include_build.sh /path/to/project
This saves results to
results_include_build.txt
.
Compare the Results:
You can use
diff
to compare the two result files:diff results_exclude_build.txt results_include_build.txt
This will display differences between results from the main project and those from the
build
folder, helping you catch any discrepancies in assets, tags, or configurations across the two environments.
This approach ensures you can verify consistency or identify issues between the project source and its build output effectively.
Individual Scripts (example) + 2 master scripts
Here’s how to set up find_all_missing_images_check.sh
to use true
or false
as the parameter for excluding or including the build
folder. I’ll also provide examples of the master scripts run_all_checks_exclude_build.sh
and run_all_checks_include_build.sh
that pass true
or false
accordingly.
Updated – Individual Scrips Example using: find_all_missing_images_check.sh
This version uses true
or false
as the parameter for excluding or including the build
folder:
#!/bin/bash
# Check if a directory and an option are provided as arguments
if [ -z "$1" ] || [ -z "$2" ]; then
echo "Usage: $0 /path/to/project/directory [true|false]"
echo "Set the second argument to 'true' to exclude the build folder, or 'false' to include it."
exit 1
fi
PROJECT_DIR="$1"
EXCLUDE_BUILD="$2"
# Set the find command parameters based on the EXCLUDE_BUILD option
BUILD_PATH_CONDITION=""
if [ "$EXCLUDE_BUILD" == "true" ]; then
BUILD_PATH_CONDITION="-not -path '*/build/*'"
elif [ "$EXCLUDE_BUILD" != "false" ]; then
echo "Invalid option. Use 'true' or 'false' for the second argument."
exit 1
fi
# Define file types and initialize counter for missing images
FILE_TYPES="html css js"
missing_image_count=0
echo "Searching for missing images in $PROJECT_DIR (Excluding build folder: $EXCLUDE_BUILD)"
for filetype in $FILE_TYPES; do
echo "Checking .$filetype files for missing images..."
find "$PROJECT_DIR" -type f -name "*.$filetype" -not -path "*/node_modules/*" $BUILD_PATH_CONDITION -print0 | while IFS= read -r -d '' file; do
grep -oP '(?<=src=")[^"]+\.(jpg|jpeg|png|gif|svg)' "$file" | while read -r img; do
if [ ! -f "$PROJECT_DIR/$img" ]; then
((missing_image_count++))
echo "[$missing_image_count] Missing image in src: $img (referenced in $file)"
fi
done
# Check for image paths in import statements in JS files
if [[ "$filetype" == "js" ]]; then
grep -oP '(?<=import .* from [\'"])[^\'"]+\.(jpg|jpeg|png|gif|svg)' "$file" | while read -r img; do
if [ ! -f "$PROJECT_DIR/$img" ]; then
((missing_image_count++))
echo "[$missing_image_count] Missing image in import: $img (referenced in $file)"
fi
done
fi
done
done
echo "Total missing images found: $missing_image_count"
Master Scripts to Run All Checks
Now, let’s set up run_all_checks_exclude_build.sh
and run_all_checks_include_build.sh
to pass true
or false
as the second argument for each script.
run_all_checks_exclude_build.sh
This script passes true
to each check script, so the build
folder is excluded:
#!/bin/bash
# Path to the project
PROJECT_DIR="$1"
if [ -z "$PROJECT_DIR" ]; then
echo "Usage: $0 /path/to/project"
exit 1
fi
echo "Running all checks on $PROJECT_DIR (excluding build folder)..."
# Run each script with 'true' to exclude the build folder
for script in seo_tag_check.sh unused_file_check.sh broken_link_check.sh js_console_errors_check.sh css_js_minification_check.sh duplicate_image_check.sh outdated_package_version_check.sh env_config_check.sh find_all_missing_images_check.sh; do
if [ -x "$script" ]; then
echo "Running $script (excluding build)..."
"./$script" "$PROJECT_DIR" true >> results_exclude_build.txt
else
echo "Script $script not found or not executable"
fi
done
echo "All checks completed. Results saved to results_exclude_build.txt"
run_all_checks_include_build.sh
This script passes false
to each check script, so the build
folder is included:
#!/bin/bash
# Path to the project
PROJECT_DIR="$1"
if [ -z "$PROJECT_DIR" ]; then
echo "Usage: $0 /path/to/project"
exit 1
fi
echo "Running all checks on $PROJECT_DIR (including build folder)..."
# Run each script with 'false' to include the build folder
for script in seo_tag_check.sh unused_file_check.sh broken_link_check.sh js_console_errors_check.sh css_js_minification_check.sh duplicate_image_check.sh outdated_package_version_check.sh env_config_check.sh find_all_missing_images_check.sh; do
if [ -x "$script" ]; then
echo "Running $script (including build)..."
"./$script" "$PROJECT_DIR" false >> results_include_build.txt
else
echo "Script $script not found or not executable"
fi
done
echo "All checks completed. Results saved to results_include_build.txt"
Summary
- Control of Build Folder Exclusion: Each script has an option to include or exclude the
build
folder, giving flexibility to scan both environments. - Master Scripts for Clarity: Separate master scripts allow you to run all checks with or without the
build
folder, making results easier to manage and compare. - Clear Output: The saved results for each run (
results_exclude_build.txt
andresults_include_build.txt
) make it easy to find inconsistencies between development and production assets.
This setup should provide the thorough checks you’re looking for without cluttering the individual scripts, allowing you to keep things clean and efficient!
The Scripts
run these occasionally to check the status of important parts of the sites.
Find Missing Images
Here’s the modified "find missing images" script with a directory argument, allowing you to run it on any specified project directory.
excludes node modules AND CHECKS import statements in js files
Updated Script: find_all_missing_images.sh
#!/bin/bash
# Check if a directory is provided as an argument
if [ -z "$1" ]; then
echo "Usage: $0 /path/to/project/directory"
exit 1
fi
# Set the directory from the argument
PROJECT_DIR="$1"
# Define the file types you want to check for missing images (e.g., .html, .css, .js files)
FILE_TYPES="html css js"
# Step 1: Find image paths in specified files and check if the images exist
echo "Searching for missing images in directory: $PROJECT_DIR (excluding node_modules)"
# Loop through each file type and look for image references
for filetype in $FILE_TYPES; do
echo "Checking .$filetype files for missing images..."
# Find all files of the current type, excluding node_modules, and search for image paths
find "$PROJECT_DIR" -type f -name "*.$filetype" -not -path "*/node_modules/*" -print0 | while IFS= read -r -d '' file; do
# Extract image paths in src attributes
grep -oP '(?<=src=")[^"]+\.(jpg|jpeg|png|gif|svg)' "$file" | while read -r img; do
# Check if the image file exists
if [ ! -f "$PROJECT_DIR/$img" ]; then
echo "Missing image in src: $img (referenced in $file)"
fi
done
# Check for image paths in import statements in JS files
if [[ "$filetype" == "js" ]]; then
grep -oP '(?<=import .* from [\'"])[^\'"]+\.(jpg|jpeg|png|gif|svg)' "$file" | while read -r img; do
# Check if the imported image file exists
if [ ! -f "$PROJECT_DIR/$img" ]; then
echo "Missing image in import: $img (referenced in $file)"
fi
done
fi
done
done
How It Works
- Argument Check: The script checks if a directory argument is provided; if not, it displays usage information.
- Loop through File Types: You can specify the file types (
html
,css
,js
, etc.) to search for image references. - Image Path Extraction: It finds image paths within files and checks for the existence of each image file.
- Report Missing Images: For each missing image, the script outputs the image path and the file that references it.
Running the Script
Save the script as
find_missing_images.sh
.Make it executable:
chmod +x find_missing_images.sh
Run it on any project directory:
./find_missing_images.sh /path/to/project/directory or if in ~/scripts/ ~/scripts/find_missing_images.sh /path/to/project/directory
This updated script allows you to search for missing images across multiple projects by simply specifying the project directory.
1. Unused Files Check
#!/bin/bash
if [ -z "$1" ]; then
echo "Usage: $0 /path/to/project"
exit 1
fi
PROJECT_DIR="$1"
ASSETS_DIR="$PROJECT_DIR/assets" # adjust path as needed
find "$ASSETS_DIR" -type f \( -name "*.jpg" -o -name "*.png" -o -name "*.css" -o -name "*.js" \) | while read -r file; do
if ! grep -qr "$(basename "$file")" "$PROJECT_DIR"; then
echo "Unused file: $file"
fi
done
2. Broken Links Check
#!/bin/bash
if [ -z "$1" ]; then
echo "Usage: $0 /path/to/project"
exit 1
fi
PROJECT_DIR="$1"
find "$PROJECT_DIR" -name "*.html" -exec grep -oP 'href="\K[^"]+' {} \; | while read -r url; do
if ! curl -s --head "$url" | grep "200 OK" > /dev/null; then
echo "Broken link: $url"
fi
done
# usage
# make executable
# chmod +x broken_link_check.sh
# call and use
# ~/scripts/project_check_scripts/broken_link_check.sh /path/to/project
3. JavaScript Console Errors Check
#!/bin/bash
if [ -z "$1" ]; then
echo "Usage: $0 http://yourwebsite.com"
exit 1
fi
WEBSITE_URL="$1"
npx playwright test --browser=firefox <<EOF
const { firefox } = require('playwright');
(async () => {
const browser = await firefox.launch();
const page = await browser.newPage();
page.on('console', msg => {
if (msg.type() === 'error') console.log('JavaScript Error:', msg.text());
});
await page.goto('$WEBSITE_URL');
await browser.close();
})();
EOF
# usage
# make executable
# chmod +x js_console_errors_check.sh
# call and use
# ~/scripts/project_check_scripts/js_console_errors_check.sh /path/to/project
4. CSS and JavaScript Minification Check
#!/bin/bash
if [ -z "$1" ]; then
echo "Usage: $0 /path/to/project"
exit 1
fi
PROJECT_DIR="$1"
find "$PROJECT_DIR" -type f \( -name "*.css" -o -name "*.js" \) | while read -r file; do
if [ "$(grep -cP '\s' "$file")" -gt 100 ]; then
echo "Unminified file: $file"
fi
done
# usage
# make executable
# chmod +x css_js_minification_check.sh
# call and use
# ~/scripts/project_check_scripts/css_js_minification_check.sh /path/to/project
5. Duplicate Images Check
#!/bin/bash
if [ -z "$1" ]; then
echo "Usage: $0 /path/to/project"
exit 1
fi
PROJECT_DIR="$1"
find "$PROJECT_DIR" -type f \( -name "*.jpg" -o -name "*.png" \) -exec sha256sum {} \; | sort | uniq -D -w 64
# usage
# make executable
# chmod +x duplicate_image_check.sh
# call and use
# ~/scripts/project_check_scripts/duplicate_image_check.sh /path/to/project
6. Outdated Package Version Check
#!/bin/bash
if [ -z "$1" ]; then
echo "Usage: $0 /path/to/project"
exit 1
fi
PROJECT_DIR="$1"
cd "$PROJECT_DIR" || exit
npm outdated
# usage
# make executable
# chmod +x outdated_package_version_check.sh
# call and use
# ~/scripts/project_check_scripts/outdated_package_version_check.sh /path/to/project
7. Environment Configuration Check
#!/bin/bash
if [ -z "$1" ]; then
echo "Usage: $0 /path/to/project/.env"
exit 1
fi
ENV_FILE="$1"
REQUIRED_VARS=("REACT_APP_ENV_GA_MEASUREMENT_ID" "REACT_APP_SINGLE_WHITELABEL_COOKIE_DOMAIN" "REACT_APP_UNIVERSAL_GA_ID" "REACT_APP_WHITELABEL_GTM_ID" "REACT_APP_JOIN_NOW_URL" "REACT_APP_URL" "REACT_APP_CONTENT_REMOVAL" "REACT_APP_NEWSLETTER_URL" "REACT_APP_MODEL_URL" "REACT_APP_PROPERTY_ID" "REACT_APP_SITE_DESCRIPTION" "REACT_APP_SITE_TITLE" "REACT_APP_SITE_NAME" "REACT_APP_SUPPORT_EMAIL" "REACT_APP_BILLING_SUPPORT_SITE")
for var in "${REQUIRED_VARS[@]}"; do
if ! grep -q "^$var=" "$ENV_FILE"; then
echo "Missing environment variable: $var"
fi
done
# usage
# make executable
# chmod +x env_config_check.sh
# call and use
# ~/scripts/project_check_scripts/env_config_check.sh /path/to/project
8. SEO Checks
#!/bin/bash
if [ -z "$1" ]; then
echo "Usage: $0 /path/to/project"
exit 1
fi
PROJECT_DIR="$1"
# Check for essential SEO tags and tracking setup in HTML files
echo "Checking for SEO and tracking elements in HTML files within $PROJECT_DIR..."
# Function to check for a specific tag in HTML files
check_tag() {
local tag="$1"
local message="$2"
if ! grep -qr "$tag" "$PROJECT_DIR"; then
echo "$message"
fi
}
# Essential SEO tags
check_tag '<meta name="description"' "Missing meta description tag"
check_tag '<meta property="og:title"' "Missing Open Graph title (og:title) tag"
check_tag '<meta property="og:description"' "Missing Open Graph description (og:description) tag"
check_tag '<meta property="og:image"' "Missing Open Graph image (og:image) tag"
check_tag '<link rel="canonical"' "Missing canonical link tag"
check_tag '<meta name="twitter:card"' "Missing Twitter card meta tag"
check_tag '<meta name="viewport"' "Missing viewport meta tag for mobile responsiveness"
# Google Tag Manager script
check_tag 'https://www.googletagmanager.com/gtm.js' "Google Tag Manager is missing"
# Data layer variable
check_tag 'window.dataLayer' "Data layer variable (for GTM tracking) is missing"
# Check for image alt text
echo "Checking for missing alt text on images..."
find "$PROJECT_DIR" -name "*.html" -exec grep -o '<img [^>]*>' {} \; | grep -v 'alt=' && echo "Some images are missing alt text"
# Title and meta description length check
echo "Checking title and description lengths..."
find "$PROJECT_DIR" -name "*.html" -exec grep -E '<title>|<meta name="description"' {} \; | while read -r line; do
if [[ $line == *'<title>'* ]]; then
title="${line//<*title>/}"
[ ${#title} -lt 50 ] || [ ${#title} -gt 60 ] && echo "Title length is not optimal: $title"
elif [[ $line == *'<meta name="description"'* ]]; then
desc="${line//<*content="/}"
desc="${desc//">/}"
[ ${#desc} -lt 150 ] || [ ${#desc} -gt 160 ] && echo "Meta description length is not optimal: $desc"
fi
done
# usage
# make executable
# chmod +x seo_tag_check.sh
# call and use
# ~/scripts/project_check_scripts/seo_tag_check.sh /path/to/project
Usage for All Scripts
Save each script in your scripts directory.
Make them executable
chmod +x /path/to/each_script.sh ====================== chmod +x ~/scripts/project_check_scripts/find_missing_images_check.sh chmod +x ~/scripts/project_check_scripts/unused_file_check.sh chmod +x ~/scripts/project_check_scripts/broken_link_check.sh chmod +x ~/scripts/project_check_scripts/js_console_errors_check.sh chmod +x ~/scripts/project_check_scripts/css_js_minification_check.sh chmod +x ~/scripts/project_check_scripts/duplicate_image_check.sh chmod +x ~/scripts/project_check_scripts/outdated_package_version_check.sh chmod +x ~/scripts/project_check_scripts/env_config_check.sh chmod +x ~/scripts/project_check_scripts/seo_tag_check.sh +++++ chmod +x ~/scripts/compare_gz_assets.sh
Run with a directory argument
./find_unused_files.sh /path/to/project ./check_broken_links.sh /path/to/project // 2 exceptions ~/scripts/compare_gz_assets.sh /path/to/your/project/build_directory node ~/scripts/get_urls.js http://localhost:3000 # And so on for each script
This setup lets you use each script on any directory, making them powerful tools as your sites grow and evolve!
Master Script: run_all_checks.sh
This master script will take a directory argument, pass it to each individual check script, and then consolidate the output into a single report.
#!/bin/bash
if [ -z "$1" ]; then
echo "Usage: $0 /path/to/project"
exit 1
fi
PROJECT_DIR="$1"
REPORT_FILE="check_report.txt"
echo "Running all checks on $PROJECT_DIR..." | tee "$REPORT_FILE"
echo "---------------------------------------" | tee -a "$REPORT_FILE"
# Run each check script and append results to the report
for script in check_seo_tags.sh find_missing_images.sh compare_gz_assets.sh check_broken_links.sh; do
if [ -x "$(command -v "$script")" ]; then
echo "Running $script..." | tee -a "$REPORT_FILE"
./"$script" "$PROJECT_DIR" >> "$REPORT_FILE" 2>&1
echo "" | tee -a "$REPORT_FILE"
else
echo "Script $script not found or not executable" | tee -a "$REPORT_FILE"
fi
done
echo "All checks completed. Results saved to $REPORT_FILE"
# usage
# make executable
# cd ~/scripts/project_check_scripts
# then run
# chmod +x *.sh
# then run master
# ~/scripts/project_check_scripts/run_all_checks.sh /path/to/project
Steps to Use the Master Script
Save each individual script (
check_seo_tags.sh
,find_missing_images.sh
,compare_gz_assets.sh
,check_broken_links.sh
, etc.) in the same directory asrun_all_checks.sh
.Make all scripts executable:
cd ~/scripts/project_check_scripts chmod +x *.sh
Run the master script on a project directory:
~/scripts/project_check_scripts/run_all_checks.sh /path/to/project
This setup will run each individual check script on the specified directory and consolidate all results into check_report.txt
. This way, you have a single, comprehensive report on your project’s SEO, assets, and link integrity.
Updated geturls.js
Script
npm i puppeteer
// using old version of node - so need - npm install puppeteer@10
const puppeteer = require('puppeteer');
// Get the URL from the command line arguments
const targetUrl = process.argv[2];
if (!targetUrl) {
console.error("Usage: node geturls.js <URL>");
process.exit(1);
}
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
try {
await page.goto(targetUrl);
} catch (error) {
console.error('Navigation error:', error);
}
const urls = new Set();
page.on('request', request => {
const url = request.url();
// Only add external URLs
if (!url.startsWith(targetUrl)) {
urls.add(url);
}
});
// Wait for additional async requests to finish
await page.waitForTimeout(5000);
await browser.close();
// Output the connected URLs
console.log('Connected URLs:', Array.from(urls));
})();
Usage
Save this as
geturls.js
.Run the script by providing the URL you want to test as an argument:
node ~/scripts/geturls.js http://localhost:3000 //or https://www.sitename.com if script in in project
This setup allows you to specify any URL you’d like to crawl, making it more versatile for testing different environments or directories on your local server.
Compare .gz assets
####Step 1: Modify the Script to Accept a Directory Argument
Update the script to take a directory as an argument so you can specify any project directory when running it.
#!/bin/bash
# Check if a directory is provided as an argument
if [ -z "$1" ]; then
echo "Usage: $0 /path/to/project/build_directory"
exit 1
fi
# Set the directory from the argument
ASSET_DIR="$1"
CHECKSUM_FILE="$ASSET_DIR/checksums.txt"
PREV_CHECKSUM_FILE="$ASSET_DIR/checksums_prev.txt"
# Step 1: Move old checksum file to a backup
if [ -f "$CHECKSUM_FILE" ]; then
mv "$CHECKSUM_FILE" "$PREV_CHECKSUM_FILE"
fi
# Step 2: Generate new checksums for .gz files, filtering out directories and .DS_Store
find "$ASSET_DIR" -type f -name "*.gz" ! -name ".DS_Store.gz" -exec sha256sum {} \; > "$CHECKSUM_FILE"
# Step 3: Check if previous checksum file exists for comparison
if [ -f "$PREV_CHECKSUM_FILE" ]; then
# Compare previous checksums with the current checksums
echo "Comparing assets from the previous build..."
if diff "$CHECKSUM_FILE" "$PREV_CHECKSUM_FILE" > /dev/null; then
echo "No changes in .gz assets."
else
echo "Changes detected in .gz assets:"
diff "$CHECKSUM_FILE" "$PREV_CHECKSUM_FILE"
fi
else
echo "No previous checksums found. All current assets are considered new."
fi
Step 2: Make It Globally Accessible
Save the Script: Save it as
compare_gz_assets.sh
in a directory outside of your projects, such as~/scripts/
.Make It Executable:
chmod +x ~/scripts/compare_gz_assets.sh
Add the Script Directory to Your PATH (optional): Add
~/scripts/
to your PATH by adding this line to your~/.bashrc
or~/.zshrc
:export PATH="$PATH:$HOME/scripts"
Reload your shell:
source ~/.bashrc # or `source ~/.zshrc` if using zsh
Step 3: Run the Script on Any Project
Now, you can run this script from anywhere by specifying the directory for the build files:
~/scripts/compare_gz_assets.sh /path/to/your/project/build_directory
or if not in PATH
~/scripts/compare_gz_assets.sh /path/to/your/project/build_directory
This setup keeps the script outside your projects and allows you to use it flexibly across different builds.