GitLab
Git Basics | Cloud Deployment | Local to Ec2

/ Git - Gitlab | GitHub, AWS EC2, SprUng, CODING, Follow up, Websites - more cheats, TOOLS / 1 minute of reading

RESOURCES

 

Setup EC2 Gitlab Keys Setup Git SetupGIT Basics for GitLabAWS CLI 2Cloud Deployment Gitlab to EC2 using AWS CLI 2Deploy to AWS with GitLab

AWS SETUP STEPS

Get ssh access – create users – get current user credentials

$ ssh -i ~/.ssh/aws-xx-devteam.pem ubuntu@XX.XXX.XX.XX

 

update server

$ sudo apt-get update

$ sudo apt-get upgrade

install aws cli 2

https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html#cliv2-linux-install

$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

unzip awscliv2.zip 

(you might have to install unzip at this point)

sudo ./aws/install

set up sftp on aws

Server: ec2-34-203-195-89.compute-1.amazonaws.com
user: ubuntu
for key: aws-ns-devteam.pem

How to install Git in AWS EC2

Installing Git on Linux

If you want to install the basic Git tools on Linux via a binary installer, you can generally do so through the package management tool that comes with your distribution. If you’re on Fedora (or any closely-related RPM-based distribution, such as RHEL or CentOS), you can use dnf:

$ sudo dnf install git-all

If you’re on a Debian-based distribution, such as Ubuntu, try apt:

$ sudo apt install git-all

For more options, there are instructions for installing on several different Unix distributions on the Git website, at https://git-scm.com/download/linux.

Installing on macOS

There are several ways to install Git on a Mac. The easiest is probably to install the Xcode Command Line Tools. On Mavericks (10.9) or above you can do this simply by trying to run git from the Terminal the very first time.

$ git --version

If you don’t have it installed already, it will prompt you to install it.

If you want a more up to date version, you can also install it via a binary installer. A macOS Git installer is maintained and available for download at the Git website, at https://git-scm.com/download/mac.

2. Set-up Gitlab

Gitlab.com SETUP STEPS

copy pub key to clipboard – gitlab style

https://docs.gitlab.com/ee/ssh/README.html#generate-an-ssh-key-pair

result of: aws-ns-devteam.pub

 

ssh-rsa XXXXcRUg3KzxpIMHheSA8fASl17nJz3PCEQ0mQCd2HUbbQ8/Bn6Gz/1+lz5iVYQeot5VfLMVsHSWGLGKOlT3rgBFDBYWQJsxnUB59sfwSf6R4bjvQFBDp2fJljhJEKcAvu17KMixSaKQIcVr8jdkiU+LcuAjsOxU1z9c2qcv77tItjSCaF+cY7FTh4AujtdOOZRmKxkgx+vgrrLDXXXXXXXXX

 

Steps – create key – copy .pub

new key created with: 


ssh-keygen -t rsa -b 2048 -C "comment"

new key


ssh-keygen -t rsa -b 2048 -C "shaxxxxxtoec2"

then copy out the .pub

Gitlab wanted it done this way

tr -d '\n' < ~/.ssh/id_ed25519.pub | pbcopy

.pub key result

ssh-rsa XXXXXOsz9QN/ONnzqULpDMHPOM3j0ajh3P7Q3jgMn6nTADSBG0ktjz2hhNihv4yG0DuHtM7L+4lQeomzoDfPs/aFgp2hFigtqfmFWEeLGppo3JhbzgAjIPL3rURaB4qAtHdX8+ng9mBNpMoXXXXXXXX/pOVvG2ONCUfg+f XXXXXX
How to Configure Git Username and Email Address

Hopefully you have already configured a global Git user

GLOBAL USER CONFIG
git config --global user.name "Your Name Here"
git config --global user.email your@email.com

Git is a distributed version control system that’s being used by most software teams today. The first thing you should do after installing Git on your system is to configure your git username and email address. Git associate your identity with every commit you make.

Git allows you to set a global and per-project username and email address. You can set or change your git identity using the git config command. Changes only affect future commits. The name and email associated with the commits you made prior to the change are not affected.

Project Based Git Permissions

You can configure an individual repo to use a specific user / email address which overrides the global configuration. From the root of the repo, run

git config user.name "Your Name Here"
git config user.email your@email.com

so we use the info from gitlab to set it up user and email for this project

git config user.name "SpiffyGitLab"

git config user.email "git@shawneee.com"

Verify that the changes were made correctly:

git config --list
user.name=Your Name
user.email=youremail@yourdomain.com

The repository-specific setting are kept in the .git/config file under the root directory of the repository.
 

Debug connection


ssh -Tv git@gitlab.com

Add new connection to ~/.ssh/config


# GitLab.com
Host gitlab.com
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/aws-ns-xxxx.pem

(had to add .pem since it is part of that key name)

Verify you can connect

The instance URL of any GitLab install is basically the link to the GitLab you’re trying to connect to.

For example, if your project is hosted on gitlab.com/yourname/yourproject then the instance URL is gitlab.com‘

 


$ ssh -T git@gitlab.com

 

RESPONSE:::::
The authenticity of host ‘gitlab.com (XXX.XX.XX.XXX)’ can’t be established.
ECDSA key fingerprint is SHA256:HbW3g8zUjNSksFbqXXXXXXXrliXFzSnUw.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added ‘gitlab.com,XXX.XX.XX.XXX’ (ECDSA) to the list of known hosts.
Welcome to GitLab, @spiffydesign!

 

2. Set up a new project on gitlab.com/spiffydesign/stagesmp

After it is created GitLab gives you the following info:
The repository for this project is empty

You can get started by cloning the repository or start adding files to it with one of the following options.

Command line instructions

You can also upload existing files from your computer using the instructions below.

Create a new repository
git clone git@gitlab.com:spiffydesign/stagesmp.git
cd stagesmp
touch README.md
git add README.md
git commit -m "add README"
git push -u origin master
Push an existing folder
cd existing_folder
git init
git remote add origin git@gitlab.com:spiffydesign/stagesmp.git
git add .
git commit -m "Initial commit"
git push -u origin master
Push an existing Git repository
cd existing_repo
git remote rename origin old-origin
git remote add origin git@gitlab.com:spiffydesign/stagesmp.git
git push -u origin --all
git push -u origin --tags

Now that the project is ready on Gitlab – let’s move it from local machine up to Gitlab.com –

simple steps – if you have no errors

To get started with GitLab, you'll need to create a repository and push code to it. Here's how to push to GitLab.

https://zapier.com/blog/how-to-push-to-gitlab/

How to Create a Project (Repository) in GitLab

Because of their larger potential in GitLab, repositories are called projects. But you can use the words interchangeably because a project behaves the same way as a repository you might find on GitHub.

After creating your account in GitLab, go to your Projects page and click New project.

From the Blank project tab, give the project a name and add a description. If you want it to be a public repository, click the Public option. Make sure the Initialize repository with README option is left unchecked. Then click the Create project button.

How to Push to GitLab

Once a new project is created, it will be blank by default. There are a couple of ways to populate it. You can directly add files online using the Add Files button.

But if you're going to be using GitLab as an online versioning tool, it's best to upload your local project files using the command line. To get started, first install and set up Git on your Linux, Mac, or Windows PC.

Then, open the Terminal (Git Bash on Windows) and navigate to the local folder that you want to sync with your GitLab project. Use the cd command to switch to the folder in question.

Now you can start the Git process with this command:

git init

Now you'll connect the remote GitLab project with your local directory. Use the following command, pasting in your GitLab project URL at the end (make sure it ends in .git).

git remote add origin <project link>

After you press Enter or Return, you won't see any confirmation. That's because you haven't added or transferred the files to the Git process yet. To add all the files from the current directory to the Git process, use the following command.

git add .

Then you can check if all files were added correctly using the following command.

git status

Now you'll make a commit, so you know which files were included when the local directory was first uploaded to the GitLab project. You can add your comment between the quotes at the end of the next command. It can include what changes you made recently to the code and if you added any new files.

git commit -m "first commit"

Now you're ready to actually push the files and code to GitLab. Enter the following command.

git push -u origin master

After you press Enter or Return, you'll be asked to enter your GitLab username and password. Since you've created a private repository, this step makes sure that only the project owner can make changes to the repository.

Once the authentication is done, the upload process will begin. And shortly, you'll see a message saying that the upload is finished.

To check whether all the data was uploaded properly, you can open the project in your browser. Instead of the blank page, you'll now see a list of all the files and folders in the project.

 

Show more
Show less

Another Method: import project via url

Nothing seemed to register with Gitlab – so will make a new project and import from the beginning

I have a url that I can point at anything running on localhost:3000

https://localtransporter.space

let's try that?

https://clickety-clack.click/img/repo_by_url.html

Import project from repository by URL

You can import your existing repositories by providing the Git URL:

  1. From your GitLab dashboard click New project.
  2. Switch to the Import project tab.
  3. Click on the Repo by URL button.
  4. Fill in the “Git repository URL” and the remaining project fields.
  5. Click Create project to begin the import process.
  6. Once complete, you will be redirected to your newly created project.
  • The repository must be accessible over http://, https:// or git://.
  • When using the http:// or https:// protocols, please provide the exact URL to the repository. HTTP redirects will not be followed.
  • If your HTTP repository is not publicly accessible, add your credentials.
  • The import will time out after 180 minutes. For repositories that take longer, use a clone/push combination.
  • To import an SVN repository, check out this document.
  • Once imported, repositories can be mirrored over SSH. Read more here.

Show more
Show less

Add a remote repository

By “adding a remote repository” to your local directory you tell Git that the path to that specific project in GitLab corresponds to that specific folder you have in your computer. This way, your local folder is identified by Git as the local content for that specific remote project.

Gitlab gives you the info you need – but of course ERRORS (see below)

Push an existing folder
cd existing_folder
git init
- fatal error fix: git remote set-url origin git@gitlab.com:spiffydesign/stagesmp.git
git remote add origin 
git@gitlab.com:spiffydesign/smpstagetoec2.git
git add .
git commit -m "Initial commit"
git push -u origin master
  1. In GitLab, create a new project to hold your files.
  2. Visit this project’s homepage, scroll down to Push an existing folder, and copy the command that starts with git remote add.
  3. On your computer, open the terminal in the directory you’ve initialized, paste the command you copied, and press enter: 
    
git remote add origin git@gitlab.com:spiffydesign/stagesmp.git

ERROR: fatal: remote origin already exists

Solving the "Remote Origin Already Exists" Error in Four Different Ways

Having explained why the error message happens, we'll now cover some of the several potential solutions you can use to solve the problem. Keep in mind that the solution you'll use will depend on your specific situation because there are a few different scenarios that can cause this problem to happen.

1. Remove the Existing Remote

The first scenario we'll cover is the one in which there's already a remote called origin, but it's the wrong remote for some reason. Let's say, for the sake of the example, that you used to use GitLab for storing your repositories online and then decided to switch over to GitHub (or vice versa). To go about that, you could follow the steps below:

  1. Create a new repository online using GitHub or GitLab.
  2. Go to your local repository and remove the existing origin remote.
  3. Add the new online repository as the correct origin remote.
  4. Push your code to the new origin.

If, for some reason, you skip step #2, that will cause Git to display the "remote origin already exists" message. So a possible solution here would be simply removing the existing remote: 

git remote remove origin

As explained before, origin is just a name for a remote. It could be a different name for you. To make sure the remote is indeed deleted, you can use the Git remote command you saw earlier. Then, if everything is all right, you can go on to adding the desired remote.

2. Update the Existing Remote's URL

We've just shown you how to remove an existing remote, so you can hopefully add a new one, this time with the correct URL. However, you might be thinking that removing the remote and adding it again with a different link will have an eerily similar result as updating the URL of the existing remote. If that's the case, you're right, of course.

So let's see how to achieve the same result we got in the previous section but in a faster way. You just have to use a single command: 

git remote set-url <REMOTE-NAME> <NEW-URL>

As we've said before, we keep talking about origin throughout this post, but there's nothing preventing you from working with whatever remote names you feel like. So a complete example with origin as the remote name and a URL to a real repo could look like this: 

git remote set-url origin https://github.com/git/git.git

3. Rename the Existing Remote

Let's say that, for whatever reason, you already have a remote called origin. You want to add a new origin, but you also need to keep the old one. How would you go about it?

Easy. Rename the existing remote before adding the new one. Just run the following command and you're set: 

git remote rename <old-name> <new-name>

So let's say you want to rename your origin remote to backup. You'd simply run: 

git remote rename origin backup

Then you can add your new remote called origin normally, and you should no longer see the "remote origin already exists" error.

4. Do Nothing!

This is not a joke, I promise you. Here's the thing: Sometimes, you might get the "remote origin already exists" error when following a tutorial that has some step asking you to add a remote called origin. If you try to run the command and get the error message, it's possible that you've already executed that command and don't remember.

To check whether that's really the case, you can use the Git remote command with the verbose option, as we've covered before: 

git remote -v

That will allow you to see the existing remotes along with the URLs they point to. If the existing remote already has the same URL provided by the tutorial, that means your repo is ready to go and you don't need to do anything else.

Show more
Show less

All set – ok back to push

From Gitlab – project set up page – Push an existing Git repository
cd existing_repo
git remote rename origin old-origin
git remote add origin git@gitlab.com:spiffydesign/stagesmp.git
git push -u origin --all
git push -u origin --tags

 

git push -u origin master – new error

error: src refspec master does not match any
error: failed to push some refs to 'git@gitlab.com:spiffydesign/smpstagetoec2.git'
Project was started off a branch called: stagesmp

NOT ‘origin’ – This error means the branch you want to push in remote doesn’t exist! So trying again and using my working branch name: 


git push -u origin stagesmp


Enumerating objects: 4885, done.
Counting objects: 100% (4885/4885), done.
Delta compression using up to 4 threads
Compressing objects: 100% (1885/1885), done.
Writing objects: 100% (4885/4885), 54.04 MiB | 2.07 MiB/s, done.
Total 4885 (delta 2713), reused 4826 (delta 2691)
remote: Resolving deltas: 100% (2713/2713), done.
To gitlab.com:spiffydesign/smpstagetoec2.git
 * [new branch]      stagesmp -> stagesmp
Branch 'stagesmp' set up to track remote branch 'stagesmp' from 'origin'.

so project is moved up to gitlab – next connect gitlab to ec2

localtogitlabdotcom

 

3. how to get gitlab from git.quantox.tech to the ec2 instance –

(See cloud deployment tabs AWS Commands to connect Gitlab to EC2 | Deploy to AWS with GitLab )

 


https://docs.gitlab.com/ee/gitlab-basics/

https://docs.gitlab.com/ee/gitlab-basics/start-using-git.html

Git authentication methods

To connect your computer with GitLab, you need to add your credentials to identify yourself. You have two options:

  • Authenticate on a project-by-project basis through HTTPS, and enter your credentials every time you perform an operation between your computer and GitLab.
  • Authenticate through SSH once and GitLab no longer requests your credentials every time you pull, push, and clone.

To start the authentication process, we’ll clone an existing repository to our computer:

  • If you want to use SSH to authenticate, follow the instructions on the SSH documentation to set it up before cloning.
  • If you want to use

    HTTPS

    , GitLab requests your user name and password:

    • If you have 2FA enabled for your account, you must use a Personal Access Token with read_repository or write_repository permissions instead of your account’s password. Create one before cloning.
    • If you don’t have 2FA enabled, use your account’s password.

Authenticating via SSH is the GitLab recommended method. You can read more about credential storage in the Git Credentials documentation.

Git terminology

If you’re familiar with the Git terminology, you may want to jump directly into the basic commands.

Namespace

A namespace is either a user name or a group name.

For example, suppose Jo is a GitLab.com user and they chose their user name as jo. You can see Jo’s profile at https://gitlab.com/jo. jo is a namespace.

Jo also created a group in GitLab, and chose the path test-group for their group. The group can be accessed under https://gitlab.com/test-group. test-group is a namespace.

Repository

Your files in GitLab live in a repository, similar to how you have them in a folder or directory in your computer. Remote repository refers to the files in GitLab and the copy in your computer is called local copy. A project in GitLab is what holds a repository, which holds your files.

Often, the word “repository” is shortened to “repo”.

Fork

When you want to copy someone else’s repository, you the project. By forking it, you create a copy of the project into your own namespace to have read and write permissions to modify the project files and settings.

For example, if you fork this project, https://gitlab.com/gitlab-tests/sample-project/ into your namespace, you create your own copy of the repository in your namespace (https://gitlab.com/your-namespace/sample-project/). From there, you can clone it into your computer, work on its files, and (optionally) submit proposed changes back to the original repository if you’d like.

Download vs clone

To create a copy of a remote repository’s files on your computer, you can either download or clone. If you download, you cannot sync it with the remote repository on GitLab.

Cloning a repository is the same as downloading, except it preserves the Git connection with the remote repository. This allows you to modify the files locally and upload the changes to the remote repository on GitLab.

Pull and push

After you saved a local copy of a repository and modified its files on your computer, you can upload the changes to GitLab. This is referred to as pushing to GitLab, as this is achieved by the command .

When the remote repository changes, your local copy is behind it. You can update it with the new changes in the remote repository. This is referred to as pulling from GitLab, as this is achieved by the command .

Basic Git commands

For the purposes of this guide, we use this example project on GitLab.com: https://gitlab.com/gitlab-tests/sample-project/.

To use it, log into GitLab.com and fork the example project into your namespace to have your own copy to playing with. Your sample project is available under https://gitlab.com/<your-namespace>/sample-project/.

You can also choose any other project to follow this guide. Then, replace the example URLs with your own project’s.

If you want to start by copying an existing GitLab repository onto your computer, see how to clone a repository. On the other hand, if you want to start by uploading an existing folder from your computer to GitLab, see how to convert a local folder into a Git repository.

Clone a repository

To start working locally on an existing remote repository, clone it with the command git clone <repository path>. You can either clone it via HTTPS or SSH, according to your preferred authentication method.

You can find both paths (HTTPS and SSH) by navigating to your project’s landing page and clicking Clone. GitLab prompts you with both paths, from which you can copy and paste in your command line. You can also clone and open directly in Visual Studio Code.

For example, considering our sample project:

  • To clone through HTTPS, use https://gitlab.com/gitlab-tests/sample-project.git.
  • To clone through SSH, use git@gitlab.com:gitlab-tests/sample-project.git.

To get started, open a terminal window in the directory you wish to add the repository files into, and run one of the git clone commands as described below.

Both commands download a copy of the files in a folder named after the project’s name and preserve the connection with the remote repository. You can then navigate to the new directory with cd sample-project and start working on it locally.

Clone via HTTPS

To clone https://gitlab.com/gitlab-tests/sample-project/ via HTTPS:

git clone https://gitlab.com/gitlab-tests/sample-project.git

On Windows, if you entered incorrect passwords multiple times and GitLab is responding Access denied, you may have to add your namespace (user name or group name) to clone through HTTPS: git clone https://namespace@gitlab.com/gitlab-org/gitlab.git.

Clone via SSH

To clone git@gitlab.com:gitlab-org/gitlab.git via SSH:

git clone git@gitlab.com:gitlab-org/gitlab.git

Convert a local directory into a repository

When you have your files in a local folder and want to convert it into a repository, you must initialize the folder through the git init command. This instructs Git to begin to track that directory as a repository. To do so, open the terminal on the directory you’d like to convert and run:

git init

This command creates a .git folder in your directory that contains Git records and configuration files. We advise against editing these files directly.

Then, on the next step, add the path to your remote repository so that Git can upload your files into the correct project.

Add a remote repository

By “adding a remote repository” to your local directory you tell Git that the path to that specific project in GitLab corresponds to that specific folder you have in your computer. This way, your local folder is identified by Git as the local content for that specific remote project.

To add a remote repository to your local copy:

  1. In GitLab, create a new project to hold your files.
  2. Visit this project’s homepage, scroll down to Push an existing folder, and copy the command that starts with git remote add.
  3. On your computer, open the terminal in the directory you’ve initialized, paste the command you copied, and press enter: 
    git remote add origin git@gitlab.com:username/projectpath.git

After you’ve done that, you can stage your files and upload them to GitLab.

Download the latest changes in the project

To work on an up-to-date copy of the project (it is important to do this every time you start working on a project), you pull to get all the changes made by users since the last time you cloned or pulled the project. Use master for the <name-of-branch> to get the main branch code, or the branch name of the branch you are currently working in.

git pull <REMOTE> <name-of-branch>

When you clone a repository, REMOTE is typically origin. This is where the repository was cloned from, and it indicates the SSH or HTTPS URL of the repository on the remote server. <name-of-branch> is usually master, but it may be any existing branch. You can create additional named remotes and branches as necessary.

You can learn more on how Git manages remote repositories in the Git Remote documentation.

View your remote repositories

To view your remote repositories, type:

git remote -v

The -v flag stands for verbose.

Branching

If you want to add code to a project but you’re not sure if it works properly, or you’re collaborating on the project with others, and don’t want your work to get mixed up, it’s a good idea to work on a different branch.

When you create a branch in a Git repository, you make a copy of its files at the time of branching. You’re free to do whatever you want with the code in your branch without impacting the main branch or other branches. And when you’re ready to bring your changes to the main codebase, you can merge your branch into the default branch used in your project (such as master).

A new branch is often called feature branch to differentiate from the default branch.

Create a branch

To create a new feature branch and work from without affecting the master branch:

git checkout -b <name-of-branch>

Note that Git does not accept empty spaces and special characters in branch names, so use only lowercase letters, numbers, hyphens (-), and underscores (_). Do not use capital letters, as it may cause duplications.

Switch to the master branch

You are always in a branch when working with Git. The main branch is the master branch, but you can use the same command to switch to a different branch by changing master to the branch name.

git checkout master

Work on an existing branch

To switch to an existing branch, so you can work on it:

git checkout <name-of-branch>

View the changes you’ve made

It’s important to be aware of what’s happening and the status of your changes. When you add, change, or delete files/folders, Git knows about it. To check the status of your changes:

git status

View differences

To view the differences between your local, unstaged changes and the repository versions that you cloned or pulled, type:

git diff

Add and commit local changes

Local changes are shown in red when you type git status. These changes may be new, modified, or deleted files/folders. Use git add to first stage (prepare) a local file/folder for committing. Then use git commit to commit (save) the staged files:

git add <file-name OR folder-name>
git commit -m "COMMENT TO DESCRIBE THE INTENTION OF THE COMMIT"

Add all changes to commit

To add and commit (save) all local changes quickly:

git add .
git commit -m "COMMENT TO DESCRIBE THE INTENTION OF THE COMMIT"

The . character means all file changes in the current directory and all subdirectories.

Send changes to GitLab.com

To push all local commits (saved changes) to the remote repository:

git push <remote> <name-of-branch>

For example, to push your local commits to the master branch of the origin remote:

git push origin master

On certain occasions, Git disallows pushes to your repository, and then you must force an update.

To create a merge request from a fork to an upstream repository, see the forking workflow.

Delete all changes in the branch

To delete all local changes in the branch that have not been added to the staging area, and leave unstaged files/folders, type:

git checkout .

Note that this removes changes to files, not the files themselves.

Unstage all changes that have been added to the staging area

To undo the most recently added, but not committed, changes to files/folders:

git reset .

Undo most recent commit

To undo the most recent commit, type:

git reset HEAD~1

This leaves the changed files and folders unstaged in your local repository.

A Git commit should not usually be reversed, particularly if you already pushed it to the remote repository. Although you can undo a commit, the best option is to avoid the situation altogether by working carefully.

Merge a branch with master branch

When you are ready to make all the changes in a branch a permanent addition to the master branch, you merge the two together:

git checkout <name-of-branch>
git merge master

Advanced use of Git through the command line

For an introduction of more advanced Git techniques, see Git rebase, force-push, and merge conflicts.

Synchronize changes in a forked repository with the upstream

Forking a repository lets you create a copy of a repository in your namespace. Changes made to your copy of the repository are not synchronized automatically with the original. Your local fork (copy) contains changes made by you only, so to keep the project in sync with the original project, you need to pull from the original repository.

You must create a link to the remote repository to pull changes from the original repository. It is common to call this remote the upstream.

You can now use the [code]<code>upstream</code>[/code] as a to[code]</code>pull<code>[/code] new updates] https://clickety-clack.click/img/start-using-git.html#download-the-latest-changes-in-the-project from the original repository, and use theorigin` to push local changes and create merge requests.

working with a bucket set up by another user

getting some info about bucket

 aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                             None    None
access_key                             None    None
secret_key                             None    None
    region                us-east-1             imds

aws ec2 describe-instances
{
    "Reservations": []
}

aws ec2 describe-instances \
>     --query 'Reservations[*].Instances[*].{Instance:InstanceId,Subnet:SubnetId}' \
>     --output json
[]
Bucket info

All we know is that it is in us-east-1 – everything else is blank.

Getting an IAM user attached and adding an Instance Profile to bucket

– so we can use it to pull gitlab

Previously created an IAM user that should be able to hop into that bucket

I also uploaded the .pub key for ns-xx-webteam – that I am trying to use everywhere to that user

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html

About Profiles

A collection of settings is called a profile. By default, the AWS CLI uses the default profile. You can create and use additional named profiles with varying credentials and settings by specifying the --profile option and assigning a name.

The following example creates a profile named produser.

$ aws configure --profile produser
AWS Access Key ID [None]: AKIAI44QH8DHBEXAMPLE
AWS Secret Access Key [None]: je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
Default region name [None]: us-east-1
Default output format [None]: text

You can then specify a --profile profilename and use the credentials and settings stored under that name.

$ aws s3 ls --profile produser

To update these settings, run aws configure again (with or without the --profile parameter, depending on which profile you want to update) and enter new values as appropriate. The next sections contain more information about the files that aws configure creates, additional settings, and named profiles.

For more information on named profiles, see Named profiles.

Show more
Show less

AWS CONFIGURE

https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config

Quick configuration with aws configure

For general use, the aws configure command is the fastest way to set up your AWS CLI installation. When you enter this command, the AWS CLI prompts you for four pieces of information:

The AWS CLI stores this information in a profile (a collection of settings) named default in the credentials file. By default, the information in this profile is used when you run an AWS CLI command that doesn't explicitly specify a profile to use. For more information on the credentials file, see Configuration and credential file settings

The following example shows sample values. Replace them with your own values as described in the following sections.

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json

Access key ID and secret access key

Access keys consist of an access key ID and secret access key, which are used to sign programmatic requests that you make to AWS. If you don't have access keys, you can create them from the AWS Management Console. As a best practice, do not use the AWS account root user access keys for any task where it's not required. Instead, create a new administrator IAM user with access keys for yourself.

The only time that you can view or download the secret access key is when you create the keys. You cannot recover them later. However, you can create new access keys at any time. You must also have permissions to perform the required IAM actions. For more information, see Permissions required to access IAM resources in the IAM User Guide.

To create access keys for an IAM user

  1. Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.

  2. In the navigation pane, choose Users.

  3. Choose the name of the user whose access keys you want to create, and then choose the Security credentials tab.

  4. In the Access keys section, choose Create access key.

  5. To view the new access key pair, choose Show. You will not have access to the secret access key again after this dialog box closes. Your credentials will look something like this:

    • Access key ID: AKIAIOSFODNN7EXAMPLE
    • Secret access key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

  6. To download the key pair, choose Download .csv file. Store the keys in a secure location. You will not have access to the secret access key again after this dialog box closes.

    Keep the keys confidential in order to protect your AWS account and never email them. Do not share them outside your organization, even if an inquiry appears to come from AWS or Amazon.com. No one who legitimately represents Amazon will ever ask you for your secret key.

  7. After you download the .csv file, choose Close. When you create an access key, the key pair is active by default, and you can use the pair right away.

Related topics

Show more
Show less

(using my "web" iam user)

aws configure

AWS Access Key ID [None]: AKIXXXXXXLOR7L2IANL
AWS Secret Access Key [None]: QXXXXXXX9mof1Js1x1MCSTQm+rs
Default region name [None]: us-east-1
Default output format [None]: json

RESULT::

ubuntu@ip-1CX-XX-3-82:~/SmpNsDevTeamUOne$ aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                             None    None
access_key     ****************IANL shared-credentials-file
secret_key     ****************m+rs shared-credentials-file
    region                us-east-1      config-file    ~/.aws/config

then add that user to ~/.aws/config – this is set up as default user
[default]
aws_access_key_id=AXXXWBLOLLLLLLL
aws_secret_access_key=YYYYYYfXXXXsfaefasefwefefefef
region = us-east-1
output = json

Attach an IAM user to a EC2 Instance

but make sure all of that exists first

From using : aws ec2 describe-instances / aws configure list / aws ec2 describe-instances we know that there was none of that info – so let’s add it

Set up an Instance ID on the EC2

https://aws.amazon.com/premiumsupport/knowledge-center/attach-replace-ec2-instance-profile/

How do I attach or replace an instance profile on an Amazon EC2 instance?

Last updated: 2021-01-12

How do I attach or replace an instance profile on an Amazon Elastic Compute Cloud (Amazon EC2) instance?

Resolution

Follow these instructions to attach or replace an instance profile on an EC2 instance.

Note:

  • If you create the AWS Identity and Access Management (IAM) role using the AWS Command Line Interface (AWS CLI), you must also create the instance profile using the AWS CLI. The IAM role name and instance profile name can be different because multiple steps are used to create and add the role to the instance profile, and then attach that role to the EC2 instance. However, if you create the role using the AWS Management Console and choose EC2 as the AWS service that the role is used for, the instance profile and IAM role names are the same.
  • If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.

AWS Management Console

  1. Open the Amazon EC2 console, and then choose Instances.
  2. Choose the instance that you want to attach an IAM role to.
  3. Check the IAM role under the Description view of the Launch Instance pane to confirm if an IAM role is attached to the Amazon EC2 instance. If an IAM role is attached, be sure that changing the role attached to this Amazon EC2 instance doesn't affect your applications or access to AWS services.
  4. Choose Actions, choose Instance Settings, and then choose Attach/Replace IAM role.
  5. On the Attach/Replace IAM role page, under IAM role, choose the instance profile that you want to attach from the drop-down list.
  6. Choose Apply.

For more information, see Creating an IAM role (Console).

 

Attempt to add instanceid – error

Add the role to an instance profile before attaching the instance profile to the EC2 instance.

1. If you haven't already created an instance profile, run the following AWS CLI command:

aws iam create-instance-profile --instance-profile-name EXAMPLEPROFILENAME

ATTEMPT::
aws iam create-instance-profile --instance-profile-name awsnsdevteamsmpinstance

ERROR::

An error occurred (AccessDenied) when calling the CreateInstanceProfile operation: User: arn:aws:iam::910029097693:user/web-clickety is not authorized to perform: iam:CreateInstanceProfile on resource: arn:aws:iam::910029097693:instance-profile/awsnsdevteamsmpinstance


https://aws.amazon.com/premiumsupport/knowledge-center/troubleshoot-iam-policy-issues/

Evaluate the IAM policy

Verify if the proper permissions are granted to the API caller by checking the IAM policies attached. For more information, see Determining whether a request is allowed or denied within an account.

After you verify that your API caller has the proper IAM permissions, review the resource-level permissions. If the API actions don't support resource-level permissions, specify the wildcard * in the resource element of the IAM statement. You can also view the Policy summary to troubleshoot resource-level permission issues. To view the policy summary:

  1. Open the IAM console.
  2. In the navigation pane, choose Policies.
  3. Choose the arrow next to the policy name to expand the policy details view.
  4. Choose Policy summary.

In the following example, the policy only provides permission to StartInstances, but the policy doesn't grant permission to DeleteSubnet. Because the DeleteSubnet action doesn't support resource-level permission, you can't specify the Amazon Resource Name (ARN) in the resource element. Use an asterisk instead: "Resource": "*"

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PermissionNotGrantedAsAPIDoesNotSupportResourceLevelPermission",
            "Effect": "Allow",
            "Action": "ec2:DeleteSubnet",
            "Resource": "arn:aws:ec2:us-east-1:123456789012:subnet/subnet-254cf47c"
        },
        {
            "Sid": "PermissionGrantedAsAPISupportResourceLevelPermission",
            "Effect": "Allow",
            "Action": "ec2:StartInstances",
            "Resource": "arn:aws:ec2:us-east-1:123456789012:instance/i-0e58cfa95b858b1ce"
        }
    ]
}

After you have confirmed your resource-level permissions, review the conditions used in your IAM policies. AWS global condition context keys can be used with all AWS services that support IAM for access control. Service specific condition keys can only be used within the specific service—EC2 conditions to EC2 API actions. For more information, see Actions, resources, and condition context keys for AWS services. If you use multiple conditions, see Multiple values in a condition.

In this example policy, the condition is matched if an IAM API request is called by IAM user admin and the source IP address is from 1.1.1.0/24 or 2.2.2.0/24.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "iam:*",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:username": "admin"
                },
                "IpAddress": {
                    "aws:SourceIp": [
                        "1.1.1.0/24",
                        "2.2.2.0/24"
                    ]
                }
            }
        }
    ]
}

Review any service-specific controls that may affect the permissions. For example, VPC endpoint policies and AWS Organizations Service Control Policies (SCP) can also affect the permission evaluation. For more information, see Controlling access to services with VPC endpoints.

IAM policy error message troubleshooting examples

See the following examples to identify the error message, the API caller, the API, and the resources being called:

Example error message API caller API Resources When
A: "An error occurred (UnauthorizedOperation) when calling the DeleteKeyPair operation: You are not authorized to perform this operation." unknown DeleteKeyPair unknown time of the error occurs
B: "An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:iam::123456789012:user/test is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::123456789012:role/EC2-FullAccess" arn:aws:iam::123456789012:user/test AssumeRole arn:aws:iam::123456789012:role/EC2-FullAccess time of the error occurs
C: "An error occurred (AccessDenied) when calling the GetSessionToken operation: Cannot call GetSessionToken with session credentials" unknown GetSessionToken unknown time of the error occurs
D: "An error occurred (UnauthorizedOperation) when calling the AssociateIamInstanceProfile operation: You are not authorized to perform this operation. Encoded authorization failure message: …." unknown AssociateIamInstanceProfile unknown time of the error occurs

Using this evaluation method, you can identify the cause of the error messages you can receive for permission issues for different AWS services. For more details, see the following common error messages and troubleshooting steps:

Example error message A:

This error message indicates that you don't have permission to call the DeleteKeyPair API.

  1. Identify the API caller.
  2. Confirm that the ec2:DeleteKeyPair API action isn't included in any deny statements.
  3. Confirm that the ec2:DeleteKeyPair API action is included in the allow statements.
  4. Confirm that there's no resource specified for this API action. Note: This API action doesn't support resource-level permissions.
  5. Confirm that all IAM conditions specified in the allow statement are supported by the delete-key-pair action and that the conditions are matched.

Example error message B:

This error message includes the API name, API caller, and target resource. Be sure that the IAM identity that called the API has the correct access to the resources. Review the IAM policies using the previous evaluation method and evaluate the following:

The trust policy of IAM role: EC2-FullAccess:

  1. Confirm arn:aws:iam::123456789012:user/test or arn:aws:iam::123456789012:root isn't included in any deny statement of the trust policy.
  2. Confirm arn:aws:iam::123456789012:user/test or arn:aws:iam::123456789012:root is included in the allow statement of the trust policy.
  3. Confirm all IAM conditions specified in that allow statement are supported by sts:AssumeRole API action and matched.

The IAM policies attached to the API caller (arn:aws:iam::123456789012:user/test):

  1. Confirm arn:aws:iam::123456789012:role/EC2-FullAccess isn't included in any deny statement with sts:AssumeRole API action.
  2. If arn:aws:iam::123456789012:root is in the allow statement of the trust policy, confirm arn:aws:iam::123456789012:role/EC2-FullAccess is included in the allow statement of the IAM policies with sts:AssumeRole API action.
  3. Confirm all IAM conditions specified in that allow statement are supported by sts:AssumeRole API action and match.

Example error message C:

This error message indicates that get-session-token isn't supported by temporary credentials. For more information, see Comparing the AWS STS API operations.

Example error message D:

This error message returns an encoded message that can provide details about the authorization failure. To decode the error message and get the details of the permission failure, see DecodeAuthorizationMessage. After decoding the error message, identify the API caller and review the resource-level permissions and conditions.

Review the IAM policy permissions:

  1. If the error message indicates that the API is explicitly denied, remove ec2:AssociateIamInstanceProfile or iam:PassRole API actions from the matched statement.
  2. Confirm that ec2:AssociateIamInstanceProfile and iam:PassRole are in the allow statement with supported and correct resource targets. For example, confirm that the resource targets of ec2:AssociateIamInstanceProfile API action are EC2 instances and the resource targets of iam:PassRole are IAM roles.
  3. If ec2:AssociateIamInstanceProfile and iam:PassRole API actions are in the same allow statement, confirm that all conditions are supported by ec2:AssociateIamInstanceProfile and iam:PassRole API action and that the conditions match.
  4. If ec2:AssociateIamInstanceProfile and iam:PassRole API actions are in separate allow statements, confirm that all conditions in each allow statement are supported by an action and that the conditions match.

Show more
Show less

2. Run the following AWS CLI command to add the role to the instance profile:

$ aws iam add-role-to-instance-profile --instance-profile-name EXAMPLEPROFILENAME --role-name EXAMPLEROLENAME

3. Run the following AWS CLI command to attach the instance profile to the EC2 instance:

$ aws ec2 associate-iam-instance-profile --iam-instance-profile Name=EXAMPLEPROFILENAME --instance-id i-012345678910abcde

Note: If you already have an instance profile associated with the EC2 instance, then the associate-iam-instance-profile command fails. To resolve this issue, run the describe-iam-instance-profile-associations command to get the associated instance ID. 

aws ec2 describe-iam-instance-profile-associations

Then, do one of the following:

Run the replace-iam-instance-profile-association command to replace the instance profile.

-or-

Run the disassociate-iam-instance-profile command to detach the instance profile, and then then rerun the associate-iam-instance-profile command.

\4. Run the following AWS CLI command to verify that the IAM role is attached to the instance:

$ aws ec2 describe-iam-instance-profile-associations --filters Name=instance-id,Values=i-012345678910abcde

Using an IAM role to grant permissions to applications running on Amazon EC2 instances
Using instance profiles
Troubleshooting Amazon EC2 and IAM

Attach an AWS IAM Role to an Existing Amazon EC2 Instance by Using the AWS CLI

https://aws.amazon.com/blogs/security/new-attach-an-aws-iam-role-to-an-existing-amazon-ec2-instance-by-using-the-aws-cli/

https://aws.amazon.com/blogs/security/new-attach-an-aws-iam-role-to-an-existing-amazon-ec2-instance-by-using-the-aws-cli/

AWS Identity and Access Management (IAM) roles enable your applications running on Amazon EC2 to use temporary security credentials that AWS creates, distributes, and rotates automatically. Using temporary credentials is an IAM best practice because you do not need to maintain long-term keys on your instance. Using IAM roles for EC2 also eliminates the need to use long-term AWS access keys that you have to manage manually or programmatically. Starting today, you can enable your applications to use temporary security credentials provided by AWS by attaching an IAM role to an existing EC2 instance. You can also replace the IAM role attached to an existing EC2 instance.

In this blog post, I show how you can attach an IAM role to an existing EC2 instance by using the AWS CLI.

Overview of the solution

In this blog post’s solution, I:

  1. Create an IAM role.
  2. Attach the IAM role to an existing EC2 instance that was originally launched without an IAM role.
  3. Replace the attached IAM role.

For the purpose of this post, I will use the placeholder, YourNewRole, to denote the newly created IAM role; the placeholder, YourNewRole-Instance-Profile, to denote the instance profile associated with this role; and the placeholder, YourInstanceId, to denote the existing instance. Be sure to replace these placeholders with the resource names from your account.

This post assumes you have set up version 1.11.46 or higher of the AWS Command Line Interface (CLI), have permissions to create an IAM role, and have permissions to call EC2 APIs.

Create an IAM role

Note: If you want to attach an existing IAM role, skip ahead to the “Attach the IAM role to an existing EC2 instance that was originally launched without an IAM role” section of this post. You also can create an IAM role using the console, and then skip ahead to the same section.

Before you can create an IAM role from the AWS CLI, you must create a trust policy. A trust policy permits AWS services such as EC2 to assume an IAM role on behalf of your application. To create the trust policy, copy the following policy and paste it in a text file that you save with the name, YourNewRole-Trust-Policy.json.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

JSON

Now that you have created the trust policy, you are ready to create an IAM role that you can then attach to an existing EC2 instance.

To create an IAM role from the AWS CLI:

  1. Open the AWS CLI and call the

    create-role command to create the IAM role, YourNewRole, based on the trust policy, YourNewRole-Trust-Policy.json

    $aws iam create-role --role-name YourNewRole --assume-role-policy-document file://YourNewRole-Trust-Policy.json


$ aws iam create-role --role-name SmpNsDevTeamUOne --assume-role-policy-document file:///Users/spiffy/Documents/projectssmp/SmpNsDevTeamUOne/SmpNsDevTeamUOne-Trust-Policy.json

- NOPE

$ aws iam create-role --role-name SmpNsDevTeamUOne --assume-role-policy-document https://clickety-clack.click/aws/SmpNsDevTeamUOne/SmpNsDevTeamUOne-Trust-Policy.json

- USING CURRENT BUCKET URL
$ aws iam create-role --role-name SmpNsDevTeamUOne --assume-role-policy-document http://ec2-34-203-195-89.compute-1.amazonaws.com/SmpNsDevTeamUOne/SmpNsDevTeamUOne-Trust-Policy.json



- USING CURRENT BUCKET
$ aws iam create-role --role-name SmpNsDevTeamUOne --assume-role-policy-document file://SmpNsDevTeamUOne/SmpNsDevTeamUOne-Trust-Policy.json



- USING CURRENT BUCKET - first cd / to the folder where you uploaded the .json
$ aws iam create-role --role-name SmpNsDevTeamUOne --assume-role-policy-document file://SmpNsDevTeamUOne-Trust-Policy.json

  2. Call the

    attach-role-policy

    https://docs.aws.amazon.com/cli/latest/userguide/cli-services-iam-policy.html

    Attaching an IAM managed policy to an IAM user

    This topic describes how to use AWS Command Line Interface (AWS CLI) commands to attach an AWS Identity and Access Management (IAM) policy to an IAM user. The policy in this example provides the user with "Power User Access". For more information on the IAM service, see the AWS Identity and Access Management User Guide.

    Before you run any commands, set your default credentials. For more information, see Configuring the AWS CLI.

    To attach an IAM managed policy to an IAM user

    1. Determine the Amazon Resource Name (ARN) of the policy to attach. The following command uses list-policies to find the ARN of the policy with the name PowerUserAccess. It then stores that ARN in an environment variable.

      $ export POLICYARN=$(aws iam list-policies --query 'Policies[?PolicyName==`PowerUserAccess`].{ARN:Arn}' --output text)       ~
$ echo $POLICYARN
arn:aws:iam::aws:policy/PowerUserAccess

    2. To attach the policy, use the command, and reference the environment variable that holds the policy ARN.

      $ aws iam attach-user-policy --user-name MyUser --policy-arn $POLICYARN

    3. Verify that the policy is attached to the user by running the command.

      $ aws iam list-attached-user-policies --user-name MyUser
{
    "AttachedPolicies": [
        {
            "PolicyName": "PowerUserAccess",
            "PolicyArn": "arn:aws:iam::aws:policy/PowerUserAccess"
        }
    ]
}

    For more information, see Access Management Resources. This topic provides links to an overview of permissions and policies, and links to examples of policies for accessing Amazon S3, Amazon EC2, and other services.

    command to grant this IAM role permission to access resources in your account. In this example, I assume your application requires read-only access to all Amazon S3 buckets in your account and objects inside the buckets. Therefore, I will use the

    AmazonS3ReadOnlyAccess AWS managed policy. For more information about AWS managed policies, see Working with Managed Policies.

    $aws iam attach-role-policy --role-name YourNewRole --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess

  3. Call the

    create-instance-profile command, followed by the add-role-to-instance-profile command to create the IAM instance profile,YourNewRole-Instance-Profile. The instance profile allows EC2 to pass the IAM role, YourNewRole, to an EC2 instance. To learn more, see

    Using Instance Profiles

    $aws iam create-instance-profile --instance-profile-name YourNewRole-Instance-Profile
$aws iam add-role-to-instance-profile --role-name YourNewRole --instance-profile-name YourNewRole-Instance-Profile

You have successfully created the IAM role, YourNewRole.

Show more
Show less

Attach the IAM role to an existing EC2 instance that was originally launched without an IAM role

You are now ready to attach the IAM role, YourNewRole, to the EC2 instance, YourInstanceId. To attach the role:

  1. Call the associate-iam-instance-profile command to attach the instance profile,

    YourNewRole-Instance-Profile, for the newly created IAM role, YourNewRole, to your EC2 instance, YourInstanceId

    
$aws ec2 associate-iam-instance-profile --instance-id YourInstanceId --iam-instance-profile Name=YourNewRole-Instance-Profile

  2. You can verify that the IAM role is now attached to the instance by calling the

    describe-iam-instance-profile-associationcommand.

    https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html

    $aws ec2 describe-iam-instance-profile-associations

  3. Now, you can update your application to use the IAM role to access AWS resources and delete the long-term keys from your instance.

Replace the attached IAM role

If your role requirements change and you need to modify the permissions you granted your EC2 instance via the IAM role, you can replace the policy attached to the IAM role. However, this will also modify permissions for other EC2 instances that use this IAM role.

Instead, you could call replace-iam-instance-profile-association to replace the currently attached IAM role, YourNewRole, with another IAM role without terminating your EC2 instance. In the following example, I use the placeholder, YourCurrentAssociation-id, to denote the current iam-instance-profile-association instance, and the placeholder, YourReplacementRole-Instance-Profile, to denote the replacement instance profile you want to associate with that instance. Be sure to replace these placeholders with the appropriate association-id and the IAM instance profile name from your account.

$aws ec2 replace-iam-instance-profile-association --association-id YourCurrentAssociation-id --iam-instance-profile Name=YourReplacementRole-Instance-Profile

Note: You can get YourCurrentAssociation-id by making the describe-iam-instance-profile-associations call.

Conclusion

As I have shown in this post, you can enable your applications to use temporary security credentials provided by AWS by attaching an IAM role to an existing EC2 instance, without relaunching the instance. You can also replace the IAM role attached to an EC2 instance, without terminating mission-critical workloads.

If you have comments about this post, submit them in the “Comments” section below. If you have questions or suggestions, please start a new thread on the IAM forum.

– Apurv

https://clickety-clack.click/img/cli-usage-parameters-file.html

Loading AWS CLI parameters from a file

Some parameters expect file names as arguments, from which the AWS CLI loads the data. Other parameters enable you to specify the parameter value as either text typed on the command line or read from a file. Whether a file is required or optional, you must encode the file correctly so that the AWS CLI can understand it. The file's encoding must match the reading system's default locale. You can determine this by using the Python locale.getpreferredencoding() method.

Note

By default, Windows PowerShell outputs text as UTF-16, which conflicts with the UTF-8 encoding used by many Linux systems. We recommend that you use -Encoding ascii with your PowerShell Out-File commands to ensure the AWS CLI can read the resulting file.

Sometimes it's convenient to load a parameter value from a file instead of trying to type it all as a command line parameter value, such as when the parameter is a complex JSON string. To specify a file that contains the value, specify a file URL in the following format.

file://complete/path/to/file

The first two slash '/' characters are part of the specification. If the required path begins with a '/', the result is three slash characters: file:///folder/file.

The URL provides the path to the file that contains the actual parameter content.

Note

This behavior is disabled automatically for parameters that already expect a URL, such as parameter that identifies a AWS CloudFormation template URL.

You can also disable this behavior by adding the following line to your AWS CLI configuration file.

cli_follow_urlparam = false

The file paths in the following examples are interpreted to be relative to the current working directory.

Linux or macOS

// Read from a file in the current directory
$ aws ec2 describe-instances --filters file://filter.json

// Read from a file in /tmp
$ aws ec2 describe-instances --filters file:///tmp/filter.json

Windows

// Read from a file in C:\temp
C:\> aws ec2 describe-instances --filters file://C:\temp\filter.json

The file:// prefix option supports Unix-style expansions, including "~/", "./", and "../". On Windows, the "~/" expression expands to your user directory, stored in the %USERPROFILE% environment variable. For example, on Windows 10 you would typically have a user directory under C:\Users\User Name\.

You must still escape JSON documents that are embedded as the value of another JSON document.

$ aws sqs create-queue --queue-name my-queue --attributes file://attributes.json

attributes.json

{
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-west-2:0123456789012:deadletter\", \"maxReceiveCount\":\"5\"}"
}

Binary files

For commands that take binary data as a parameter, specify that the data is binary content by using the fileb:// prefix. Commands that accept binary data include:

  • aws ec2 run-instances – --user-data parameter.
  • aws s3api put-object – --sse-customer-key parameter.
  • aws kms decrypt – --ciphertext-blob parameter.

The following example generates a binary 256-bit AES key using a Linux command line tool, and then provides it to Amazon S3 to encrypt an uploaded file server-side.

$ dd if=/dev/urandom bs=1 count=32 > sse.key
32+0 records in
32+0 records out
32 bytes (32 B) copied, 0.000164441 s, 195 kB/s
$ aws s3api put-object --bucket my-bucket --key test.txt --body test.txt --sse-customer-key fileb://sse.key --sse-customer-algorithm AES256
{
    "SSECustomerKeyMD5": "iVg8oWa8sy714+FjtesrJg==",
    "SSECustomerAlgorithm": "AES256",
    "ETag": "\"a6118e84b76cf98bf04bbe14b6045c6c\""
}

Remote files

The AWS CLI also supports loading parameters from a file hosted on the internet with an http:// or https:// URL. The following example references a file stored in an Amazon S3 bucket. This allows you to access parameter files from any computer, but it does require that the container is publicly accessible.

$ aws ec2 run-instances --image-id ami-12345678 --block-device-mappings http://my-bucket.s3.amazonaws.com/filename.json

The preceding example assumes that the file filename.json contains the following JSON data.

[
  {
    "DeviceName": "/dev/sdb",
    "Ebs": {
      "VolumeSize": 20,
      "DeleteOnTermination": false,
      "VolumeType": "standard"
    }
  }
]
Launching, listing, and terminating Amazon EC2 instances

https://docs.aws.amazon.com/cli/latest/userguide/cli-services-ec2-instances.html

Using an IAM role in the AWS CLI

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html

IAM roles for Amazon EC2

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

Show more
Show less

https://docs.gitlab.com/ee/ci/cloud_deployment/#run-aws-commands-from-gitlab-cicd

using aws-ns-dev-team.pub key everywhere

ssh-rsa XXXXXXXt4cvEXNHBCvRkKfwndbGuCu4aYDN8pwEPh8PvO9iSA79jzSl/FO2aP3ik28dBLuhOFkGBjbwWo+vn8gwbe2gH/bXQbPI+I8ysga5yHo9Y1FqW00hxTXS6xO+Z+TAIcEN96kNVcRUg3KzxpIMHheSA8fAXXXXXXXXXd2HUbbQ8/Bn6Gz/1+lz5iVYQeot5VfLMVsHSWGLGKOlT3rgBFDBYWQJsxnUB59sfwSf6R4bjvQFBDp2fJljhJEKcAvu17KMixSaKQIcXXXXXXX

 

https://docs.gitlab.com/ee/ci/cloud_deployment/

Run AWS commands from GitLab CI/CD

Introduced in GitLab 12.6.

The GitLab AWS Docker image provides the AWS Command Line Interface, which enables you to run aws commands. As part of your deployment strategy, you can run aws commands directly from .gitlab-ci.yml by specifying the GitLab AWS Docker image.

Some credentials are required to be able to run aws commands:

  1. Sign up for an AWS account if you don’t have one yet.
  2. Log in onto the console and create a new IAM user.
  3. Select your newly created user to access its details. Navigate to Security credentials > Create a new access key.

    A new Access key ID and Secret access key are generated. Please take a note of them right away.

  4. In your GitLab project, go to Settings > CI/CD. Set the following as CI/CD variables (see table below):
    Environment variable name Value
    AWS_ACCESS_KEY_ID Your Access key ID
    AWS_SECRET_ACCESS_KEY Your Secret access key
    AWS_DEFAULT_REGION Your region code
  5. You can now use aws commands in the .gitlab-ci.yml file of this project: 
    deploy:
  stage: deploy
  image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest  # see the note below
  script:
    - aws s3 ...
    - aws create-deployment ...

    The image used in the example above (registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest) is hosted on the GitLab Container Registry and is ready to use. Alternatively, replace the image with one hosted on AWS ECR.

Use an AWS Elastic Container Registry (ECR) image in your CI/CD

Instead of referencing an image hosted on the GitLab Registry, you can reference an image hosted on any third-party registry, such as the Amazon Elastic Container Registry (ECR).

To do so, push your image into your ECR repository. Then reference it in your .gitlab-ci.yml file and replace the image path to point to your ECR image.

Deploy your application to the AWS Elastic Container Service (ECS)

Version history

GitLab provides a series of CI templates that you can include in your project. To automate deployments of your application to your Amazon Elastic Container Service (AWS ECS) cluster, you can include the AWS/Deploy-ECS.gitlab-ci.yml template in your .gitlab-ci.yml file.

GitLab also provides Docker images that can be used in your gitlab-ci.yml file to simplify working with AWS:

  • Use registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest to use AWS CLI commands.
  • Use registry.gitlab.com/gitlab-org/cloud-deploy/aws-ecs:latest to deploy your application to AWS ECS.

Before getting started with this process, you need a cluster on AWS ECS, as well as related components, like an ECS service, ECS task definition, a database on AWS RDS, etc. Read more about AWS ECS.

The ECS task definition can be:

  • An existing task definition in AWS ECS
  • A JSON file containing a task definition. Create the JSON file by using the template provided in the AWS documentation. Copy the task definition into a new file in your project, for example <project-root>/ci/aws/task-definition.json. Available in GitLab 13.3 and later.

After you have these prerequisites ready, follow these steps:

  1. Make sure your AWS credentials are set up as CI/CD variables for your project. You can follow the steps above to complete this setup.
  2. Add these variables to your project’s .gitlab-ci.yml file, or in the project’s CI/CD settings:
    • CI_AWS_ECS_CLUSTER: The name of the AWS ECS cluster that you’re targeting for your deployments.
    • CI_AWS_ECS_SERVICE: The name of the targeted service tied to your AWS ECS cluster.
    • CI_AWS_ECS_TASK_DEFINITION: The name of an existing task definition in ECS tied to the service mentioned above.
    variables:
  CI_AWS_ECS_CLUSTER: my-cluster
  CI_AWS_ECS_SERVICE: my-service
  CI_AWS_ECS_TASK_DEFINITION: my-task-definition

    You can find these names after selecting the targeted cluster on your AWS ECS dashboard:

    Alternatively, if you want to use a task definition defined in a JSON file, use CI_AWS_ECS_TASK_DEFINITION_FILE instead:

    variables:
  CI_AWS_ECS_CLUSTER: my-cluster
  CI_AWS_ECS_SERVICE: my-service
  CI_AWS_ECS_TASK_DEFINITION_FILE: ci/aws/my_task_definition.json

    You can create your CI_AWS_ECS_TASK_DEFINITION_FILE variable as a file-typed CI/CD variable instead of a regular CI/CD variable. If you choose to do so, set the variable value to be the full contents of the JSON task definition. You can then remove the JSON file from your project.

    In both cases, make sure that the value for the containerDefinitions[].name attribute is the same as the Container name defined in your targeted ECS service.

    CI_AWS_ECS_TASK_DEFINITION_FILE takes precedence over CI_AWS_ECS_TASK_DEFINITION if both these variables are defined within your project.

    If the name of the task definition you wrote in your JSON file is the same name as an existing task definition on AWS, then a new revision is created for it. Otherwise, a brand new task definition is created, starting at revision 1.

  3. Include this template in .gitlab-ci.yml: 
    include:
  - template: AWS/Deploy-ECS.gitlab-ci.yml

    The AWS/Deploy-ECS template ships with GitLab and is available on GitLab.com.

  4. Commit and push your updated .gitlab-ci.yml to your project’s repository, and you’re done!

    Your application Docker image is rebuilt and pushed to the GitLab registry. If your image is located in a private registry, make sure your task definition is configured with a repositoryCredentials attribute.

    Then the targeted task definition is updated with the location of the new Docker image, and a new revision is created in ECS as result.

    Finally, your AWS ECS service is updated with the new revision of the task definition, making the cluster pull the newest version of your application.

The template includes both the and “sub-templates”. Do not include these “sub-templates” on their own, and only include the main AWS/Deploy-ECS.gitlab-ci.yml template. The “sub-templates” are designed to only be used along with the main template. They may move or change unexpectedly causing your pipeline to fail if you didn’t include the main template. Also, the job names within these templates may change. Do not override these jobs names in your own pipeline, as the override stops working when the name changes.

Alternatively, if you don’t wish to use the AWS/Deploy-ECS.gitlab-ci.yml template to deploy to AWS ECS, you can always use our aws-base Docker image to run your own AWS CLI commands for ECS.

deploy:
  stage: deploy
  image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest
  script:
    - aws ecs register-task-definition ...

Provision and deploy to your AWS Elastic Compute Cloud (EC2)

Introduced in GitLab 13.5.

You can use the AWS/CF-Provision-and-Deploy-EC2 CI template to perform the following actions within the same pipeline:

  1. Create stack: Provision your own infrastructure by leveraging the AWS CloudFormation API.
  2. Push to S3: Push your previously-built artifact to an AWS S3 bucket.
  3. Deploy to EC2: Deploy this pushed content onto an AWS EC2 instance.

Run the AWS/CF-Provision-and-Deploy-EC2.gitlab-ci.yml template

To run the AWS/CF-Provision-and-Deploy-EC2.gitlab-ci.yml template, you must pass three JSON input objects, based on existing templates:

  1. The AWS documentation provides templates for the Create stack and Deploy to EC2 steps (links below). We provide the template for the remaining step, Push to S3:
  2. After you have completed these three templates based on your requirements, you have two ways to pass in these JSON objects:
    • They can be three actual files located in your project. You must specify their path relative to your project root in your .gitlab-ci.yml file, using the following CI/CD variables. For example, if your files are in a <project_root>/aws folder: 
      variables:
  CI_AWS_CF_CREATE_STACK_FILE: 'aws/cf_create_stack.json'
  CI_AWS_S3_PUSH_FILE: 'aws/s3_push.json'
  CI_AWS_EC2_DEPLOYMENT_FILE: 'aws/create_deployment.json'
    • Alternatively, you can provide these JSON objects as file-typed CI/CD variables. In your project, go to Settings > CI/CD > Variables and add the three variables listed above as file-typed CI/CD variables. For each variable, set the value to its corresponding JSON object.
  3. Provide the name of the stack you’re creating and/or targeting, as a CI/CD variable: 
    variables:
  CI_AWS_CF_STACK_NAME: 'YourStackName'
  4. Add this CI template to your .gitlab-ci.yml: 
    include:
  - template: AWS/CF-Provision-and-Deploy-EC2.gitlab-ci.yml

When running your project pipeline at this point:

  • Your AWS CloudFormation stack is created based on the content of your CI_AWS_CF_CREATE_STACK_FILE file/variable. If your stack already exists, this step is skipped, but the provision job it belongs to still runs.
  • Your built application is pushed to your S3 bucket then and deployed to your EC2 instance, based on the related JSON object’s content. The deployment job finishes whenever the deployment to EC2 is done or has failed.

Custom build job for Auto DevOps

Introduced in GitLab 13.6.

To leverage Auto DevOps for your project when deploying to AWS EC2, first you must define your AWS credentials as CI/CD variables.

Next, define a job for the build stage. To do so, you must reference the Auto-DevOps.gitlab-ci.yml template and include a job named build_artifact in your .gitlab-ci.yml file. For example:

# .gitlab-ci.yml

include:
  - template: Auto-DevOps.gitlab-ci.yml

variables:
  AUTO_DEVOPS_PLATFORM_TARGET: EC2

build_artifact:
  stage: build
  script:
    - <your build script goes here>
  artifacts:
    paths:
      - <built artifact>

For a video walkthrough of this configuration process, see Auto Deploy to EC2.

Deploy to Amazon EKS

Deploy to Google Cloud

https://about.gitlab.com/blog/2020/12/15/deploy-aws/

Cloud computing services are replacing traditional hardware technologies at an extremely fast pace. The majority of businesses worldwide are already moving to the cloud or plan to in the near future. Over a short period of time, this technology took over the market as businesses preferred remote access to data as well as the cloud’s scalability, economy, and reach.

AWS Deployment: deploying to the cloud

COVID-19 and the resulting trend toward remote work forced organizations to adopt cloud technologies even if they hadn’t planned to originally. Software deployment to the cloud has also increased. Cloud is no longer just virtual machines, organizations are driving the use of Containers as a Service (CaaS) due to their growing interest in leveraging containers to ease development and testing, speed up deployment, scale operations, and increase the efficiency of workloads running in the cloud.

Since deployment to the cloud has become a standard practice, at GitLab we want to make this repeatable and boring. In this blog post, we explain how we’ve made deploying to Amazon Web Services (AWS) easier and we invite users to replicate this example to deploy to other cloud providers in a similar way.

Since we want cloud deployment to be as flexible as possible (similar to a microservices architecture), we constructed atomic Docker images that function as building blocks. Users can use these images as part of their custom gitlab-ci.yml file or use our predefined .gitlab-ci.yml templates. We also added the ability to use Auto DevOps with the new AWS deployment targets.

AWS Deployment: how to use GitLab’s official AWS Docker Images

AWS CLI Docker image

In GitLab 12.6, we provided an official GitLab AWS cloud-deploy Docker image that downloads and installs the AWS CLI. This allows users to run aws commands directly from their pipelines. For more information, see Run AWS commands from GitLab CI/CD.

Cloud Formation stack creation Docker image

In . With this type of JSON file, the command creates the infrastructure on AWS, including an EC2 instance directly from the .gitlab-ci.yml file. The script exists once we get confirmation that the stack setup is complete or has failed (through periodic polling).

Push to S3 and Deploy to EC2 Docker image

In . This code can be whatever artifact is built from a preceding build job. The gl-ec2 deploy-to-ec2 script uses aws deploy create-deployment behind the scenes to create a deployment to an EC2 instance directly from the .gitlab-ci.yml file. For an example of the JSON template to pass, see . The script ends once we get confirmation that the deployment has succeeded or failed (via polling).

AWS Deployment: using GitLab CI templates to deploy to AWS

How to deploy to Elastic Container Service (ECS) with GitLab

In that deploys to ECS and extends support for Fargate. Users can include the template in their configuration, specify a few variables, and their application will be deployed and ready to go in no time. This template can be customized for your specific needs. For example: Replacing the selected container registry, changing the path of the file location, etc.

How to deploy to Elastic Cloud Compute (EC2) with GitLab

In that provisions the infrastructure by leveraging AWS CloudFormation. It then pushes your previously-built artifact to an AWS S3 bucket and deploys the pushed content to AWS EC2.

AWS Deployment: security considerations

Predefined AWS CI/CD variables

In order to deploy to AWS, you must use AWS security keys to connect to to your AWS instance. Users can define this security keys as CI/CD environment variables that can be used by the deployment pipeline.

In GitLab 12.9, we added support for predefined AWS variables. This support function helps users know which variables are required for deploying to AWS and also prevents typos and spelling mistakes.

Env. variable name Value
AWS_ACCESS_KEY_ID Your Access key ID
AWS_SECRET_ACCESS_KEY Your Secret access key
AWS_DEFAULT_REGION Your region code

“Just-in-time” guidance for AWS deployments

GitLab 13.1 provides just-in-time guidance for users who wish to deploy to AWS. Setting up AWS deployments isn’t always as easy as we’d like it to be, so we’ve added in-product links to our AWS templates and documentation when you start adding AWS CI/CD variables to make it easier for you to use our AWS features. This will help you get up and running faster.

AWS guide from CI/CD variables

Added security for the GitLab’s official AWS Docker images

In GitLab 13.5, we changed the image identifier from the release version number to the Docker image digest. Docker supports immutable image identifiers and we adopted this best practice to update our cloud-deploy images. When a new image is tagged, we also programmatically retrieve the image digest upon its build and create a release note to effectively communicate this digest to users. This guarantees that every instance of the service runs exactly the same code. You can roll back to an earlier version of the image, even if that version wasn’t tagged (or is no longer tagged). This can even prevent race conditions if a new image is pushed while a deploy is in progress.

Docker image digest or release tag

AWS Deployment: auto DevOps support

GitLab already supports Kubernetes users deploying to AWS EKS cluster. Click the link to read instructions about how to deploy an application to a GitLab-managed Amazon EKS cluster with Auto DevOps.

We also expanded Auto DevOps to support non-Kubernetes users. Users can specify their deployment target by adding the AUTO_DEVOPS_PLATFORM_TARGET variable under the CI/CD variables settings. Specifying the deployment target platform builds a full CI/CD pipeline that deploys to AWS targets.

We currently support:

  • AUTO_DEVOPS_PLATFORM_TARGET: ECS (added in GitLab 13.0)
  • AUTO_DEVOPS_PLATFORM_TARGET: FARGATE (added in GitLab 13.2)
  • AUTO_DEVOPS_PLATFORM_TARGET: EC2 (added in GitLab 13.6)

For more information about Auto DevOps for AWS targets, see requirements for Auto DevOps documentation.

Here’s a quick recording for how to use Auto Deploy to ECS:

Speed run on how to use auto deploy to EC2 (animation):

AWS Deployment: Future plans to extend deployment support via GitLab

Check out some of the open issues below to see our plans are for the future of deploying to AWS using GitLab.

More material on deploying to EKS and Lambda

We invite you to contribute to our other cloud provider solutions:

At GitLab, everyone can contribute. If you want to deploy to a target that isn’t mentioned in this post, please let us know by adding an issue and linking it to our Natively support hypercloud deployments epic.

Cover image by SpaceX on Unsplash

Scroll to Top