HowTo: Install AWS CLI - Security Credentials

Here we talk about how to setup our security credentials to use the various command line tools used with AWS.

Where To Find Your Security Credentials

Assuming you’re not working with an account created with IAM, you can find your security credentials here.

If you are using an account created through IAM, there is no way to get the secret access key again. If you don’t have it, a new one set of credentials can and will need to be generated. With IAM accounts, Amazon will not generate X.509 certificates, however you can generate your own and upload the public key.

Regarding Environment Variables

To keep sensitive information out of ~/.bashrc we’ll create a new file ~/.bash_aws and put the following into ~/.bashrc:

~/.bashrc (Excerpt)

1
2
# AWS Credentials
[[ -f ~/.bash_aws ]] && . ~/.bash_aws

This will check if ~/.bash_aws exists, and if so, source it.

AWS Credential Environment Variables (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY)

For use with the new AWS Command Line Interface Tool and for use with python programs using boto, we can set our credentials using the following environment variables:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

Set them in ~/.bash_aws, replacing the example values with your access key id and secret access key:

~/.bash_aws (Excerpt)

1
2
export AWS_ACCESS_KEY_ID=C99F5C7EE00F1EXAMPLE
export AWS_SECRET_ACCESS_KEY=a63xWEj9ZFbigxqA7wI3Nuwj3mte3RDBdEXAMPLE

AWS Configuration File (AWS_CONFIG_FILE)

For this we will need our Access Key ID and Secret Access Key, and optionally which region we want to default to. We will then put them in a file with the following format:

[default]
aws_access_key_id=Your Access Key ID
aws_secret_access_key=Your Secret Access Key
region=Optional, the default region to use for this profile

You can define multiple profiles in this file, just put --profile profile_name

For this example we will use the file ~/.aws/aws_config_file and change the permissions so only your account can read the file.

Console - user@hostname ~ $

1
mkdir -p ~/.aws

~/.aws/aws_config_file

1
2
3
4
5
6
7
8
9
[default]
aws_access_key_id=C99F5C7EE00F1EXAMPLE
aws_secret_access_key=a63xWEj9ZFbigxqA7wI3Nuwj3mte3RDBdEXAMPLE
region=us-east-1

[profile test]
aws_access_key_id=DAAG6D8FF11G2EXAMPLE
aws_secret_access_key=b74yXFkAaGcjhyrB8xm4Ovxk4nuf4SECeEXAMPLE
region=us-west-2

Console - user@hostname ~ $

1
chmod 600 ~/.aws/aws_config_file

We then need to set the environment variable AWS_CONFIG_FILE to the path of the file.

We will want to set AWS_CONFIG_FILE as a user specific environment variable, so not everyone on the machine will use your credentials. We will put it into ~/.bash_aws for this example:

~/.bash_aws (Excerpt)

1
export AWS_CONFIG_FILE=~/.aws/aws_config_file

Boto Config File (/etc/boto.cfg, ~/.boto)

For programs that use boto that aren’t the AWS Command Line Tool, you can still set your credentials in a configuration file.

  • /etc/boto.cfg is used for global settings on the system
  • ~/.boto is used for user-specific settings.

The layout is the similar AWS_CONFIG_FILE except only one set of credentials, [Credentials], can be set:

[Credentials]
aws_access_key_id=Your Access Key ID
aws_secret_access_key=Your Secret Access Key

~/.boto

1
2
3
[Credentials]
aws_access_key_id=C99F5C7EE00F1EXAMPLE
aws_secret_access_key=a63xWEj9ZFbigxqA7wI3Nuwj3mte3RDBdEXAMPLE

See BotoConfig for more options to set in the configuration file.

Also with boto, you can use the environment variables:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

as described in AWS Credential Environment Variables (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY). Also IAM roles can be used if you are running on an EC2 instance that has an IAM role set.

AWS Credential Environment Variables (AWS_ACCESS_KEY_ID/AWS_SECRET_KEY)

This approach for setting your credentials that replaced X.509 certificates for EC2 API tools and can be used by the AWS Java SDK. We will need our Access Key ID and Secret Access Key and we will use them to set the following environment variables:

  • AWS_ACCESS_KEY_ID (for the Access Key ID)
  • AWS_SECRET_KEY (for the Secret Access Key)

The following example has you setting these environment variables in your ~/.bash_aws file

~/.bash_aws (Excerpt)

1
2
export AWS_ACCESS_KEY_ID=C99F5C7EE00F1EXAMPLE
export AWS_SECRET_KEY=a63xWEj9ZFbigxqA7wI3Nuwj3mte3RDBdEXAMPLE

AWS Credential File (AWS_CREDENTIAL_FILE)

For this we need our Access Key ID and Secret Access Key, and we will put them into a file with the following format:

AWSAccessKeyId=Your Access Key ID
AWSSecretKey=Your Secret Access Key

For this example we will put it into ~/.aws/aws_credential_file and change the permissions so only your account can read the file.

Console - user@hostname ~ $

1
mkdir -p ~/.aws

~/.aws/aws_credential_file

1
2
AWSAccessKeyId=C99F5C7EE00F1EXAMPLE
AWSSecretKey=a63xWEj9ZFbigxqA7wI3Nuwj3mte3RDBdEXAMPLE

Console - user@hostname ~ $

1
chmod 600 ~/.aws/aws_credential_file

We then need to set the environment variable AWS_CREDENTIAL_FILE to the path of the file.

We will want to set AWS_CREDENTIAL_FILE as a user specific environment variable, so not everyone on the machine will use your credentials. We will put it into ~/.bash_aws for this example.

~/.bash_aws (Excerpt)

1
export AWS_CREDENTIAL_FILE=~/.aws/aws_credential_file

X.509 certificates (EC2_CERT/EC2_PRIVATE_KEY)

For this we will need our X.509 certificates. If Amazon created them for you, should be named something like: cert-3F4CCOPFPLFTBZ2KRFLQXQYMYEXAMPLE.pem, which is the public key; and pk-3F4CCOPFPLFTBZ2KRFLQXQYMYEXAMPLE.pem, which is the private key. If you are using an Amazon IAM user, or if you wish to create your own X.509 certificate, see our HowTo: Generate an X.509 Certificate for an Amazon IAM User article.

We also need to set 2 environment variables:

  • EC2_CERT
  • EC2_PRIVATE_KEY

Set EC2_CERT to the cert.pem file, and EC2_PRIVATE_KEY to the pk.pem file.

For this example, save the cert.pem file and pk.pem file into the ~/.aws directory; set the environment variables in ~/.bash_aws:

~/.bash_aws (Excerpt)

1
2
export EC2_CERT=~/.aws/cert-3F4CCOPFPLFTBZ2KRFLQXQYMYEXAMPLE.pem
export EC2_PRIVATE_KEY=~/.aws/pk-3F4CCOPFPLFTBZ2KRFLQXQYMYEXAMPLE.pem

~/.s3curl and ~/.aws_secrets

The S3 command line tool, s3curl.pl keeps its security credentials in ~/.s3curl. The Route53 command line tool dnscurl.pl, and the CloudFront command line tool cfcurl.pl, keep their security credentials in ~/.aws_secrets. These two files have the format, so if you so desired you can populate one and have the other be a symbolic link. The format is like so:

%awsSecretAccessKeys = (
friendly_name => {
id => 'Your Access Key ID',
key => 'Your Secret Access Key',
},
another_name => {
id => 'Another Access Key ID',
key => 'Another Secret Access Key',
},
);

For this example we will save into ~/.s3curl and we will have the friendly name be main.

~/.s3curl

1
2
3
4
5
6
%awsSecretAccessKeys = (
main => {
id => 'C99F5C7EE00F1EXAMPLE',
key => 'a63xWEj9ZFbigxqA7wI3Nuwj3mte3RDBdEXAMPLE',
},
);

Set the permissions to 600, otherwise it will complain when you try running the programs.

Console - user@hostname ~ $

1
chmod 600 ~/.s3curl

Create the symbolic link for ~/.aws_secrets.

Console - user@hostname ~ $

1
ln -s ~/.s3curl ~/.aws_secrets

credentials.json

For the elasticmapreduce ruby command line tool, you will need to put your credentials into a json file. By default it looks in the install directory for credentials.json, but there is a flag (-c) that will let you use whatever credentials.json file you want. For example:

Console - user@hostname ~ $

1
2
3
4
5
6
7
8
9
elastic-mapreduce \
--create \
--name "Hive Query" \
--instance-type m1.small \
--num-instances 2 \
--hive-script \
s3n://your-bucket/hive/command.hiveql \
--args "-d","OUTPUT=s3n://my-bucket/output/`date --utc +%Y%m%d'T'%H%M%S'Z'`/" \
-c ~/.aws/credentials.json

The format for credentials.json is like so:

{
"access_id": "AWS Access Key ID",
"private_key": "AWS Secret Access Key",
"keypair": "Key pair to start the EMR instances with",
"key-pair-file": "The path the the private key (.pem) file for the keypair",
"log_uri": "Path to S3 bucket, (ex: s3n://my-bucket/logs/)",
"region": "Region to run the job in. One of the following: us-east-1, us-west-1, us-west-2, eu-west-1, ap-northeast-1, ap-southeast-1, ap-southeast-2, sa-east-1"
}

For this example we will save into ~/.aws/credentials.json.

~/.aws/credentials.json

1
2
3
4
5
6
7
8
{
"access_id": "C99F5C7EE00F1EXAMPLE",
"private_key": "a63xWEj9ZFbigxqA7wI3Nuwj3mte3RDBdEXAMPLE",
"keypair": "my-key",
"key-pair-file": "~/.ssh/my-key.pem",
"log_uri": "s3n://my-bucket/hadoop/",
"region": "us-east-1"
}

Parts in this series