The AWS Provider for Managed Kubernetes is designed to facilitate the setup of Kubernetes clusters on AWS Cloud. It automates key cloud component management, enabling easy and rapid creation and maintenance of scalable, production-grade Kubernetes clusters. This provider simplifies various cloud infrastructure tasks, including node autoscaling, load balancer provisioning, version upgrades, and storage management. It enables centralized management of a fleet of Kubernetes clusters across single or multiple AWS accounts.

​
Requirements

​
Supported add-ons

  • Dashboard: Provides a Kubernetes dashboard UI for the cluster.
  • AWS Workload Identity: Allows your pods to assume AWS IAM Roles.
  • AWS ECR: Allows pulling images from private ECR registries.
  • AWS EFS: Provides support for persistent volumes using AWS Elastic File System.
  • Local Path Storage: Create PVCs backed by local volumes.
  • Logs: Enable logging for pods and cluster auditing.

​
Step 1 - Preparing the AWS environment

Ensure your AWS environment includes:

Tailored instructions for IAM Role creation are available in the Control Plane Console when creating an AWS Kubernetes cluster using the UI. Alternatively, you can follow the instructions below:

  • IAM Role:
    1. Create an IAM Role with the naming pattern cpln-mk8s-${ORG}. For example, cpln-mk8s-testing if the organization name is testing.
    2. Create an IAM Policy with the naming pattern cpln-mk8s-${ORG}. For example, cpln-mk8s-testing if the organization name is testing.
Control Plane Connector Policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CreateIamResourcesForCplnPrefix",
      "Effect": "Allow",
      "Action": [
        "iam:Get*",
        "iam:CreateRole",
        "iam:DeleteRole",
        "iam:PassRole",
        "iam:CreateInstanceProfile",
        "iam:DeleteInstanceProfile",
        "iam:AddRoleToInstanceProfile",
        "iam:RemoveRoleFromInstanceProfile",
        "iam:CreatePolicy",
        "iam:DeletePolicy",
        "iam:CreatePolicyVersion",
        "iam:DeletePolicyVersion",
        "iam:AttachRolePolicy",
        "iam:DetachRolePolicy",
        "iam:UpdateRoleDescription",
        "iam:UpdateAssumeRolePolicy",
        "sts:AssumeRole"
      ],
      "Resource": [
        "arn:aws:iam::*:instance-profile/cpln-*",
        "arn:aws:iam::*:policy/cpln-*",
        "arn:aws:iam::*:role/cpln-*"
      ]
    },
    {
      "Sid": "ValidateConfiguration",
      "Effect": "Allow",
      "Action": [
        "kms:ListKeys",
        "kms:DescribeKey",
        "iam:ListPolicies",
        "iam:ListPolicyVersions",
        "iam:ListRolePolicies",
        "ec2:DescribeKeyPairs",
        "ec2:DescribeSubnets",
        "ec2:DescribeVpcs",
        "ec2:DescribeSecurityGroups",
        "iam:ListAttachedRolePolicies",
        "iam:ListInstanceProfilesForRole",
        "iam:SimulatePrincipalPolicy"
      ],
      "Resource": "*"
    },
    {
      "Sid": "PrepareCluster",
      "Effect": "Allow",
      "Action": [
        "autoscaling:CreateAutoScalingGroup",
        "ec2:CreateLaunchTemplate",
        "ec2:ModifyLaunchTemplate",
        "ec2:CreateSecurityGroup",
        "ec2:CreateLaunchTemplate",
        "ec2:CreateTags",
        "autoscaling:CreateOrUpdateTags",
        "ec2:RunInstances",
        "ec2:DescribeLaunchTemplates",
        "ec2:DescribeLaunchTemplateVersions",
        "ec2:DescribeImages",
        "autoscaling:UpdateAutoScalingGroup",
        "autoscaling:StartInstanceRefresh",
        "ec2:CreateLaunchTemplateVersion",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:AuthorizeSecurityGroupEgress",
        "ec2:DescribeSecurityGroups"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Cleanup",
      "Effect": "Allow",
      "Action": [
        "autoscaling:DeleteAutoScalingGroup",
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DeleteLaunchConfiguration",
        "ec2:DeleteLaunchTemplate",
        "ec2:DeleteSecurityGroup",
        "ec2:DeleteVolume",
        "ec2:DescribeLaunchTemplates",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeVolumes",
        "elasticloadbalancing:DeleteLoadBalancer",
        "elasticloadbalancing:DescribeLoadBalancers",
        "elasticloadbalancing:DescribeTags"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AmiLookupInSSM",
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameters",
        "ssm:DescribeParameters",
        "ssm:GetParameter"
      ],
      "Resource": [
        "arn:aws:ssm:*::parameter/aws/service/ami-amazon-linux-latest/*",
        "arn:aws:ssm:*::parameter/aws/service/canonical/*",
        "arn:aws:ssm:*::parameter/aws/service/eks/*"
      ]
    }
  ]
}

  1. Attach the connector policy to the cpln-mk8s-${ORG} IAM Role.

  2. Edit the trust policy of the cpln-mk8s-${ORG} IAM Role.

    Important: Replace ${ACCOUNT_ID} with your own account id.

Trust Policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/oidc.mk8s.cpln.io/federate"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.mk8s.cpln.io/federate:aud": "sts.amazonaws.com",
          "oidc.mk8s.cpln.io/federate:sub": "observer-org"
        }
      }
    }
  ]
}
  1. Save the updates and copy the ARN of the IAM Role for use in the next step.

​
Step 2 - Create a Managed Kubernetes Cluster Using a Manifest File

  1. Update the manifest below: Modify the following aws-mk8s-template.yaml YAML manifest below with actual values. Customize the file as needed. Replace placeholders for ${ROLE_ARN}, ${SSH_KEY_NAME}, ${SUBNET_1} and ${SUBNET_2} and ${VPC_ID}. Use the Role ARN of the IAM Role created in the previous step in place of ${ROLE_ARN}.
YAML
kind: mk8s
name: aws-mk8s-example
description: aws-mk8s-example cluster
tags: {}
spec:
  provider:
    aws:
      deployRoleArn: ${ROLE_ARN}
      image:
        recommended: ubuntu/jammy-22.04
      # keyPair: ${SSH_KEY_NAME} # Optional
      nodePools:
        - name: general
          bootDiskSize: 25
          extraSecurityGroupIds: []
          instanceTypes:
            - m7g.large
            - m6gd.large
            - c7g.xlarge
          labels: {}
          maxSize: 4
          minSize: 1
          onDemandBaseCapacity: 0
          onDemandPercentageAboveBaseCapacity: 0
          # overrideImage:
          #   recommended: amazon/amzn2
          spotAllocationStrategy: lowest-price
          subnetIds:
            - ${SUBNET_1}
            - ${SUBNET_2}
          taints: []
      region: eu-central-1
      securityGroupIds: []
      skipCreateRoles: false
      vpcId: ${VPC_ID}
  addOns:
    awsECR: {}
    dashboard: {}
    localPathStorage: {}
    awsWorkloadIdentity: {}
  firewall:
    - description: default
      sourceCIDR: 0.0.0.0/0
  version: 1.30.3

This example creates a managed Kubernetes cluster in AWS with the following configurations:

  • Add-ons: Includes Dashboard, Local Path Storage, AWS Workload Identity and awsECR.
  • Location: The cluster’s Kubernetes control plane is managed in the eu-central-1 region. Placing worker nodes close to the control plane is recommended for optimal performance. Full list of supported regions: [af-south-1, ap-east-1, ap-northeast-1, ap-northeast-2, ap-northeast-3, ap-south-1, ap-south-2, ap-southeast-1, ap-southeast-2, ap-southeast-3, ap-southeast-4, ca-central-1, eu-central-1, eu-central-2, eu-north-1, eu-south-1, eu-south-2, eu-west-1, eu-west-2, eu-west-3, me-central-1, me-south-1, sa-east-1, us-east-1, us-east-2, us-west-1, us-west-2].
  • Kubernetes API Firewall: Utilizes the Default rule, allowing public access to the Kubernetes API. It is advisable to restrict API access to a known IP range for security purposes.
  • Kubernetes Version: 1.30.3.
  • Node Pool: A single general node pool, scaling on-demand between 1 and 4 nodes.
  • Server Image: ubuntu/jammy-22.04.

  1. Create the Cluster: Deploy the aws-mk8s-example cluster by applying the manifest.

    • Console: Apply the aws-mk8s-template.yaml file using the cpln apply >_ option in the upper right corner.
    • CLI: Execute cpln apply -f aws-mk8s-template.yaml --org YOUR_ORG_HERE.

    Wait until the cluster is initialized.

​
Step 3 - Accessing the Cluster

​
1. Using the Terminal

  1. Obtain the Cluster’s Kubeconfig File: Execute the command cpln mk8s kubeconfig aws-mk8s-example -f /tmp/aws-mk8s-example-conf.
  2. Access the Cluster with kubectl: Use the obtained kubeconfig file by running export KUBECONFIG=/tmp/aws-mk8s-example-conf for the current shell session.

​
2. Using Kubernetes Dashboard

  1. Navigate to Control Plane Console: Visit the Control Plane Console.
  2. Access the Dashboard: In the Control Plane Console, navigate to Kubernetes in the left sidebar panel and click on Open under Dashboard for the cluster aws-mk8s-example.

​
Advanced Configuration Options

​
Deploy Role Chain

AWS accounts with high security requirements may not allow a direct trust policy to allow direct access from Control Plane for setup and management. In this suituation, you can configure the initial login from Control Plane into a different intermediary account and add a chain of roles that will be assumed to reach the target account.

​
.spec.provider.aws.deployRoleChain

Each record in the deployRoleChain array is assumed after the initial login into the deployRole. The chain of roles is processed in the order listed in the array, from top to bottom. Each step in the chain must have a trust policy in place to allow sts:assumeRole from the previous step.

PropertyTypeDescription
roleArnstringThe role that will be assumed using sts:assumeRole.
externalIdstring (optional)The externalId to use during the assume role.
sessionNamePrefixstring (optional)A prefix to use for the session name. The rest of the session will be generated to make it unique.
example trust policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::333377778888:role/ExampleTrustedRole"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "ExampleExternalId"
        }
      }
    }
  ]
}