Security Groups with EKS pods

By Default EKS assigns security groups to node group. In case one wanted to have multiple security group defined within a EKS cluster, they have to use multiple nodegroups. This requires over provisioned & hence not a cost effective solution. AWS has provided feature to assign security groups ie ENI directly to the pods since 2020. Let’s see how we can do same.

Architecture:

Requirements:

Amazon VPC CNI plugin version 1.7.7 +

Steps

1. Enable pod networking environment variable:

kubectl set env daemonset -n kube-system aws-node ENABLE_POD_ENI=true

2. Set DISABLE_TCP_EARLY_DEMUX to true

kubectl set env daemonset -n kube-system aws-node  -c "aws-vpc-cni-init" DISABLE_TCP_EARLY_DEMUX=true

3. upgrade cni version to latest possible

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.9/config/v1.9/aws-k8s-cni.yaml

4. Enable OIDC provider (if not done already)

cluster_name=<cluster Name>
eksctl utils associate-iam-oidc-provider — cluster=$cluster_name

5. Get VPC details

set VPCID (aws eks describe-cluster --name $cluster_name  --query "cluster.resourcesVpcConfig.vpcId"  --output text)
echo $VPCID

6. Create security Group for same VPC

set RDSSG (aws ec2 create-security-group --group-name RDSDbAccessSG  --description "Security group to apply to apps that need access to RDS" --vpc-id $VPCID --query "GroupId"  --output text )
aws ec2 describe-security-groups --filters "Name=vpc-id,Values=vpc-096b8b9f0cb3f605a" --query "SecurityGroups[].[GroupId,GroupName]"

7. Create service account for the pod

Metadata, we have added labels, this will be used to associate them with correct security groups.

kubectl apply -f service-account.yaml

8. Create Security Policy associated with labels

kubectl apply -f security-group.yaml

9. Create pod / Deployment with service account

apiVersion: v1
kind: Pod
metadata:
  name: postgres-test
spec:
  serviceAccountName: rds-db-access
containers:
  - name: postgres-test
    image: <>.dkr.ecr.eu-west-1.amazonaws.com/postgres-test-demo:new

Error:

Error1: Unable to allocate IP

Error Details: “Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “4110c06553f32e31b55a681577b11e3533c1d99ab4461f99c928fe8aaef3f6de” network for pod “postgres-test”: networkPlugin cni failed to set up pod “postgres-test_default” network: add cmd: failed to assign an IP address to container”

Solution: upgrade aws cni version

Reference:

Introducing security groups for pods | Amazon Web Services