Today I have a Interesting concept, where I was migrating all my secret storing methodology from Init Containers to CSI Driver to pick our passwords from AWS Secrets Manager and mount them on our containers.
But the interesting part of AWS security comes into picture. Even though the service is running in EKS, it won’t allow us to pick or change any AWS resource directly. Here an important concept comes into picture called OIDC.
What basically is OIDC?
OpenID Connect, where it will generate an endpoint used to unlock or authenticate with the key produced by it. Here I was working with EKS, where it has a default behavior. We can get this endpoint and place it in our AWS IAM identity by using the command:
eksctl utils associate-iam-oidc-provider \
--cluster my-cluster \
--region ap-south-1 \
--approve
This command will extract the endpoint and place it in AWS, meaning the requests coming from that particular EKS cluster are valid and trusted.
Now how come our pods get the EKS secrets or make any AWS changes?
This is through Service Accounts and IRSA (IAM Roles for Service Accounts).
We need to create a Service Account and attach it to the pod. Since we gave a particular Service Account, EKS will automatically provide a token/key to that Service Account.
apiVersion: v1
kind: ServiceAccount
metadata:
name: payment-sa
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/payment-role
This Service Account is attached to a class which is responsible to call the CSI Driver installed, where it runs as a pod inside kube-system to fetch the secrets.
CSI Driver official link: https://secrets-store-csi-driver.sigs.k8s.io/
AWS Provider for CSI Driver: https://github.com/aws/secrets-store-csi-driver-provider-aws
Now here, since we have to take care of authentication and authorization, we need an IAM Role attached through ARN which has permissions to fetch the secrets from AWS Secrets Manager.
Example policy:
{
"Effect": "Allow",
"Action": [
"secretsmanager:GetSecretValue"
],
"Resource": "*"
}
When the pod gets started, it will trigger the CSI Driver. The CSI Driver uses the Service Account token to authenticate with AWS through the OIDC trust relationship. AWS uses the endpoint to verify the information and provide temporary credentials.
Then the CSI Driver fetches the secrets from AWS Secrets Manager and mounts them directly inside the container.
volumeMounts:
- name: secrets-store mountPath: "/mnt/secrets" readOnly: true
Architecture flow:
Pod
↓
Service Account
↓
OIDC Trust
↓
IAM Role
↓
CSI Driver
↓
AWS Secrets Manager
↓
Secrets Mounted
Why this method when we already have Init Containers?
With Init Containers we had additional responsibilities and troubleshooting areas became more. To reduce this scope, we moved to CSI Driver.
This looks simple, and now the only areas to manage are:
AWS Secrets
IAM permissions
Mount points
A very clean cloud-native implementation in EKS.
IRSA (IAM Roles for Service Accounts) is an EKS feature that maps Kubernetes Service Accounts to AWS IAM Roles, allowing pods to securely access AWS services without storing static AWS credentials inside containers.
