Part 2 - Expanding Across Accounts

Quick recap of Part 1:

In Part 1 we setup secure MFA authentication into a single account from the CLI/SDK, using nothing more than out of the box AWS features and tools (IAM users, Roles, CLI, Python SDK).

New Story:

Expand on Part 1’s MFA access model and use it for cross-account authentication.

Background:

All of AWS cross-account access is based on the AssumeRole API. Indeed the biggest hurdle for most folks moving from a single AWS account to multiple accounts is that leap from using IAM user accounts alone to using AssumeRole. It’s a very big shift; In Concept, in Complexity, in Setup, in Management, in End User Support.

Thankfully we’ve already made that leap in Part 1, so expanding it to cross-account is relatively simple.

First the User:

When we last left test-user her IAM user only had permission to assume a single Role in a single AWS Account ID, that which we specified in our Resource block:

"Resource": [
    "arn:aws:iam::111111111111:role/TestRole"
]

We could continue this theme and add the full ARN for each account’s Role to that Resource block, but after a few accounts that quickly becomes unwieldy to manage. Imagine you have dozens or hundreds of accounts…and dozens of cross-account roles for each various level of access. Some organizations spin up new accounts on the fly for development sandboxes, etc.

Instead we’re going to take a page from convention over configuration and at least from the User’s side, allow them to AssumeRole into any TestRole in any account by using the * syntax:

"Resource": [
    "arn:aws:iam::*:role/TestRole"
]

Also in the interests of management, we’re going to take a moment to organize our IAM structure a bit while we’re here to conform to IAM Best Practices:

  1. We’ll move this User’s inline policy to a managed policy.
  2. Create a new Group
  3. Attach the new managed policy to the new Group
  4. Assign the User to the new Group

1: New AssumeTestRole Managed Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AssumeTestRole",
            "Effect": "Allow",
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": [
                "arn:aws:iam::*:role/TestRole"
            ],
            "Condition": {
                "Bool": {
                    "aws:MultiFactorAuthPresent": true
                }
            }
        }
    ]
}

In the Console that will look something like: Console example of IAM Managed Policy

2, 3: New Group with attached AssumeTestRole Policy: Console example of IAM Group with Managed Policy Attached

4: And the User added to the Group: Console example of IAM User added to Group

Management

We manage who has access to the TestRole by assigning them to (or removing them from) the group TestGroup. In turn each AWS account controls for themselves what if any permissions those TestGroup users should get by adjusting the Policies on their own TestRole.

Next the new Roles

The TestRole we configured in Part 1 for account 111111111111 (aka “cheddar”) is already fine as it is. Up to this point nothing has changed for our user.

What we need is a new TestRole in the other accounts we want to grant access to:

  1. 222222222222 (aka profile brie)
  2. 333333333333 (aka profile havarti)

Follow the instructions in Part1 for the original TestRole, the setup and policies are identical at the account level.

Update the ~/.aws/config file for test-user

With the two new accounts we have access to we’ll need two new profiles in our config. Our updated config will look like this:

[profile login]
region = us-east-1
output = json
[profile cheddar]
source_profile = login
role_arn = arn:aws:iam::111111111111:role/TestRole
mfa_serial = arn:aws:iam::111111111111:mfa/test-user
role_session_name = test-user
region = us-east-1
output = json
[profile brie]
source_profile = login
role_arn = arn:aws:iam::222222222222:role/TestRole
mfa_serial = arn:aws:iam::111111111111:mfa/test-user
role_session_name = test-user
region = us-east-1
output = json
[profile havarti]
source_profile = login
role_arn = arn:aws:iam::333333333333:role/TestRole
mfa_serial = arn:aws:iam::111111111111:mfa/test-user
role_session_name = test-user
region = us-east-1
output = json

Our brie and harvarti sections are almost identical to our cheddar section, only the role_arn values changed to target the appropriate accounts. The mfa_serial stays the same because we’re authenticating against our login account when we issue these AssumeRole API calls.

Take it for a spin

And we’re done, that’s it. Arguably that was more work than we needed to do because we included some cleanup at the top of this. As I mentioned all the hard work so far happened in Part 1.

Lets try our new profiles out. I’ll assume you gave your new TestRoles in the new accounts the same SecurityAudit policy which among other permissions will allow us to call the ListUsers API:

$ aws --profile cheddar iam list-users
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
{
    "Users": [
        {
            "UserName": "test-user",
            "PasswordLastUsed": "2018-02-13T01:12:29Z",
            "CreateDate": "2018-01-29T23:58:26Z",
            "UserId": "AIDAI5VR6DPR4O4MI7BDE",
            "Path": "/",
            "Arn": "arn:aws:iam::111111111111:user/test-user"
        }
    ]
}
$ aws --profile brie iam list-users
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
{
    "Users": []
}
$ aws --profile havarti iam list-users
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
{
    "Users": []
}

Everything works! We’re done! Happy users! Happy admins! Impenetrable MFA security!

Well, not quite.

Cons:

While this works and frankly is WAY better than the tedious approach the AWS documentation offers (not to mention inadequate especially when it comes to enforcing MFA security), this cross-account approach still has all the same cons we went over in Part 1. It’s even worse than that, because now those cons apply seperately to each new profile. Every profile has its own temporary credentials, each of which require a unique MFA token, which only generate new every 30 seconds, and each seperately expire in just 1 hour.

It’s only saving grace is like before, it’s entirely done in ~/.aws/config from the user’s perspective. No helper scripts or manually copying and pasting values into environment variables for each session.

But personally, I find myself frequently running calls across “all accounts” and “all regions” with a pair of loops that looks like this:

for profile in cheddar brie havarti; do
    for region in $(aws --profile $profile ec2 describe-regions --query "Regions[][RegionName]" --output text); do
        aws --profile $profile --region $region ec2 describe-instances
    done
done

If you run that loop with our current setup you’ll get this:

$ for profile in cheddar brie havarti; do
>    for region in $(aws --profile $profile ec2 describe-regions --query "Regions[][RegionName]" --output text); do
>        aws --profile $profile --region $region ec2 describe-instances
>    done
>done
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
{
    "Reservations": []
}
{
    "Reservations": []
}
{
    "Reservations": []
}
    ...etc for each $region

Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
{
    "Reservations": []
}
    ...etc for each $region
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
{
    "Reservations": []
}
    ...etc for each $region

If that wasn’t bad enough, if it’s a Python/boto3 script we’re calling in the middle of that loop it will prompt for a new MFA token with each and every region!!

$ for profile in cheddar brie havarti; do
>    for region in $(aws --profile $profile ec2 describe-regions --query "Regions[][RegionName]" --output text); do
>        export AWS_PROFILE="$profile"
>        export AWS_DEFAULT_REGION="$region"
>        ./some_python_boto3_script.py
>    done
>done
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
Enter MFA code for arn:aws:iam::111111111111:mfa/test-user:
...

Kick In The Head

Ok sure, you can alleviate some of that by coding the loops directly in the Python script so it can take advantage of memory-cached session tokens, but it’s still going to prompt you for an MFA token at least once for every profile you use. This despite the fact that they are all using the same MFA device and so you haven’t really increased security one iota. And if you run the script multiple times? It’s almost enough to make an admin push for moving to MS Azure, the horror of horrors!

I also prefer to code my Python utilities against “default” credentials rather than assume what naming the user has used for their profiles or what regions they which to run against.

While this model does function and is better than a kick in the head, it’s only really going to be satisfactory for light use and highly sensitive accounts. It’s completely untenable for power users and serious admin work. We still need to find a Better Way™

Part 3 - Finding a Better Way™

(Comments and questions will be enabled at the end of this journey on Part 3)