Service Accounts & AWS IAM Roles¶

Description¶

Theory¶

In Kubernetes on AWS, there are two complementary access control regimes at work. AWS Identity and Access Management (IAM) allows you to assign permissions to AWS services: for example, an app can access an S3 bucket. In the context of Kubernetes, the complementary system to define permissions towards Kubernetes resources is Kubernetes Role-based Access Control (RBAC).

OIDC federation access allows you to assume IAM roles via the Secure Token Service (STS), enabling authentication with an OIDC provider, receiving a JSON Web Token (JWT), which in turn can be used to assume an IAM role. Kubernetes, on the other hand, can issue so-called projected service account tokens, which happen to be valid OIDC JWTs for pods. Our setup equips each pod with a cryptographically-signed token that can be verified by STS against the OIDC provider to establish the pod’s identity. 

Simple explanation¶

We can associate k8s Service Account with AWS IAM Role using OIDC. Then we can create AWS IAM Policy with desired set of permissions (access to S3, Athena, etc.) and assign it to this role.

This would allow application's PODs to access specific AWS resources. 

Why we did this?¶

In the previous access model you had to make a call from your application to the Vault Server, providing Vault's Token to get a credentials which will allow your app to access required resources in AWS.

By switching to IAM Role/Service Account model we are eliminating Vault from the equation, reducing the number of the moving parts and possible points of failure. At the same time we are keeping the same level of security, because we can granularly manage access policies per application.

Active Roles and Service Accounts¶

Guide for Developers¶

Assigning Service Account to Pods¶

You want your application to be able to access some resources in AWS, how should you proceed?

First, you need to check if Service Account is already created in the namespace of your application:

Info

Here we will use blue-raven as an example, that's why all command are coming with '-n blue-raven'. Replace it with the namespace of your application.

$kubectl get serviceAccounts -n blue-raven

NAME         SECRETS   AGE
blue-raven   1         43h
default      1         55d

$kubectl describe serviceAccounts/blue-raven -n blue-raven

***
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::003250186609:role/bln-fairbid-production-blue-raven
***

In the example above you see that Service Account is created in blue-raven namespace and it's annotated with the corresponding IAM Role. If this is the case for you, then this is all you need: pods with your application can access resources in AWS (the ones which are mentioned in the policies attached to the IAM Role).

To assign POD to specific Service Account you need to add following spec to the POD's yaml (Deployment, Stateful Set, Replica Set) document:

apiVersion: v1
kind: Pod
metadata:
  name: my_perfect_pod
  namespace: blue-raven
spec:
  serviceAccountName: blue-raven
...
...

Usually we deploy pods with deployments, so instead add this to add this to deployment specification:

     spec:
+      {{- if .Values.serviceAccount.enabled }}
+      serviceAccountName: {{ .Values.serviceAccount.name }}
+      {{- end }}
       containers:

Last thing you should do is to recreate/redeploy your PODs/Deployments to apply the changes of the spec.

To validate that the POD has the correct settings you can enter the POD and execute following commands:

$kubectl exec --stdin --tty pod/my_perfect_pod /bin/bash -n blue-raven

root@my_perfect_pod/# env | grep AWS
AWS_ROLE_ARN=arn:aws:iam::003250186609:role/bln-fairbid-production-blue-raven
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

If you did everything correctly you will see information about Role and Identity Token.

Creating Service Account for existing Application¶

If there's no service account created for your application (you see only default account) then refer to the table "Active Roles and Service Accounts" above. If your application listed there that means that we already took care of all configurations and all you need is to create Service Account in the namespace of your app.

You can do it by adding this helm template to your application:

{{- if .Values.serviceAccount.enabled }}
apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ .Values.serviceAccount.name }}
  annotations:
    eks.amazonaws.com/role-arn: {{ .Values.serviceAccount.iam_role }}
{{- end }}

And adjusting the values:

serviceAccount:
  enabled: true
  name: blue-raven
  iam_role: arn:aws:iam::003250186609:role/bln-fairbid-production-blue-raven

Now, when the Service Account is created you can assign it to the pods as it described in the previous step.

Creating Service Account for new Application¶

If you are working on a completely new application and it is not listed in the table "Active Roles and Service Accounts" then please create a ticket for DevOps and describe the requirements: application name, namespace, required permissions in AWS.

Adding & Changing Permissions provided by Service Account¶

If you have everything set for your application, i.e. Service Account is configured and assigned to the PODs, and you want to change the permissions it provide to your app - please create a ticket for DevOps and describe what kind of a change you desire. We will apply this change to the IAM Policy and it will take action immediately without the need of redeploying an application.

Libraries¶

Guide for DevOps¶

Configuration as code¶

IAM Roles and corresponding policies were created with terraform, using the bln-fairbid-iac/production/ocean/us-east-1state.

Terraform resources can be found here: iam_serviceaccounts.tf (Bitbucket)

All applications policies are being stored in the policies/ directory.

In case if you need to add a service account for a new app, here's the drill:

1. Edit iam_serviceaccounts.tf (copy existing block and change the variables accordingly)
2. Add policies with the permissions requested by developers
3. Edit terragrunt.hcl state (add the new variables and values)
4. terragrunt apply

Articles¶

If you are interested in more information about implemented solution, please take a look at the following articles:

Introducing fine-grained IAM roles for service accounts | AWS Open Source Blog

IAM roles for service accounts - Amazon EKS

With any additional questions - don't hesitate to reach out to #devops-berlin in slack.

Based on: DEVOPSBLN-1593