EKS + Fargate = Extensibility of Kubernetes + Serverless Benefits
One of the fundamental advantages of cloud computing is the major reduction in the complexity of managing server hardware. With the rise of application containerization, AWS has come up with a multitude of container-specific platforms. However, provisioning and managing virtual servers was not an option and always an operational burden until AWS introduced Fargate (ECS launch type), where container workloads are deployed on AWS managed compute resources without having to manage any EC2 instances by the user. Fargate relies on docker containers to run applications. Containers run as ECS Tasks (similar to a Kubernetes Pod), tasks are managed by Services (similar to a Kubernetes Deployment), and services can be set up with an Elastic Load Balancer to receive external traffic over HTTP.
Choosing a container execution strategy includes managing EC2 instances where the most flexibility, control over configuration and price comes into play. AWS offers three major models for running containers: ECS, Fargate and EKS.
- ECS was the first container orchestration tool offering by AWS. It essentially consists of EC2 instances which have docker already installed, and run a Docker container which talks with AWS backend.
- Fargate was the second service offering and is intended to abstract all everything below the container (EC2 instances where they run) from the user.
- EKS is the latest offering. With EKS some of the complexities of launching a Kubernetes Cluster are abstracted from the user as AWS manages the Master Nodes — the control plane.
AWS Fargate is one of the two ECS launch types available and Fargate is not an additional container or offering in itself, it is a launch type. AWS Fargate presents Containers as a Services (CaaS) as compared to Infrastructure as a Service (IaaS) that EC2 is, in simpler terms “Serverless”. That means the container has already been set up, including the networking, security and most importantly the scaling. These major operational burdens are abstracted away providing users with the ability to run containers directly on the cloud. With the service, developers simply have to specify the resources for each container instance and let Fargate seamlessly takes care of all other needs.
ECS (Container Instances) vs Fargate
An ECS container instance is nothing more than an EC2 instance that runs the ECS Container Agent. The EC2 instance is owned and managed by the user. The instance appears in the list of EC2 instances like any other EC2 instance. On the other hand AWS Fargate manages the task execution. No EC2 instances to manage anymore. Each task that runs in Fargate comes with a dedicated Elastic Network Interface (ENI) with a private IP address. All containers of the same task can communicate with each other via localhost.
Serverless Kubernetes Pods Using Amazon EKS and AWS Fargate
At AWS re:Invent in Las Vegas, the company announced that Elastic Kubernetes Service (EKS) is available on Fargate. Users can use this feature to enable EKS to run Kubernetes pods on AWS Fargate. With AWS Fargate there is no longer a need to worry about patching, scaling, or securing a cluster of Amazon EC2 instances to run Kubernetes containers in the cloud.
Then & Now
User launches EKS control plane and adds Amazon EC2 to the EKS cluster. The required compute capacity to run Kubernetes based workloads is provided by EC2. Charge of $ 0.20/cluster/hr for managing the master node and worker nodes are charged at the regular EC2 instance costs. Other complimentary charges are applicable for launching complementary resources like EBS Volumes, Network/Elastic Loadbalancers, etc.
Now, user launches EKS control plane and configures a fargate-profile and may also add EC2 to the cluster. Networking on Fargate use the same subnets configured with AWS CNI for EKS, where each Fargate pod is allocated with an ENI from the specified set of subnets. With this new addition, users can leverage serverless/nodeless Kubernetes where they only pay for the amount of vCPU and memory resources that pod needs to run.
With AWS Fargate, users only pay for the amount of vCPU and memory resources that pod needs to run. This includes the resources the pod requests in addition to a small amount of memory needed to run Kubernetes components alongside the pod.
AWS Fargate on Amazon EKS
AWS Fargate with EKS can be seamlessly deployed with “eksctl” where all required entities are auto-created. AWS also enables users to associate Fargate functionality with the existing EKS clusters (having EC2 as Kubernetes Nodes and version=1.14).
In existing clusters using Managed Node Groups (used to provision or register the instances that provide compute capacity) all cluster security groups are automatically configured to the Fargate based workloads or users can add security groups to node group’s or auto-scaling group to enable communication between pods running on existing EC2 instances with pods running on Fargate. Pods running on Fargate are automatically configured to use the cluster security group for the cluster that they are associated with.
There are multiple ways to create a EKS cluster with Fargate. Users can use eksctl, Terraform or AWS Console to create Fargate on EKS. Users can use “fargate-profiles” to control the scheduling of Kubernetes pods on Fargate or existing EC2 Kubernetes nodes. Fargate profiles facilitate usage of selectors to scope deployment of pods based on namespace or a key:value tag that can be added to Kubernetes deployment/pod labels.
Creating a Cluster using eksctl
A new option “ — fargate” is added to eksctl which auto-creates required IAM policies and fargate-profiles to enable usage of Fargate with EKS.
Creating a Cluster using Terraform
Terraform added support to create required fargate-profiles and attach the same to EKS cluster.
creating a fargate-profile:
resource "aws_eks_fargate_profile" "example" {
cluster_name = aws_eks_cluster.example.name
fargate_profile_name = "example"
pod_execution_role_arn = aws_iam_role.example.arn
subnet_ids = aws_subnet.example[*].id selector {
namespace = "example"
}}
creating IAM Role for EKS Fargate Profile:
resource "aws_iam_role" "example" {
name = "eks-fargate-profile-example" assume_role_policy = jsonencode({
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "eks-fargate-pods.amazonaws.com"
}
}]
Version = "2012-10-17"
})
}
resource "aws_iam_role_policy_attachment" "example-AmazonEKSFargatePodExecutionRolePolicy" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSFargatePodExecutionRolePolicy"
role = aws_iam_role.example.name
}
Creating a Cluster using AWS Console
Fargate profiles can be added to EKS cluster once it is created from AWS console “EKS” section.
Configuration Components
Fargate Pod Execution Role
Pods on AWS Fargate need to make calls to AWS APIs on like pull container images from Amazon ECR. The Amazon EKS pod execution role provides the IAM permissions to perform the actions on behalf of the user.
trust-relationship for IAM role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"eks.amazonaws.com",
"eks-fargate-pods.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}Associating trust-relationship with a IAM role:
Adding AmazonEKSFargatePodExecutionRolePolicy to the role above:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage"
],
"Resource": "*"
}
]
}The Kubernetes agent or kubelet makes calls to the Kubernetes API server on the users behalf, so it requires an IAM policy and role for the cluster to know that the kubelet belongs to a specific user. This IAM role is referred to as a pod execution IAM role. Users can have multiple pod execution roles for different purposes associated with your account.
Fargate Profiles
Fargate profiles allow users to declare which pods to run on Fargate. This enables users to have workloads running on both Nodes (EC2 instances) and Fargate. At least one Fargate profile should be specified to determine pods that should use Fargate when they are launched. Fargate profiles are immutable, a new updated profile should be created to replace an existing profile followed by the deletion of the original profile.
User creates a profile that matches the namespace and labels defined as part of Kubernetes pods. When new pods are started, they will run on Fargate using the execution role and subnets defined in the profile.
{
"fargateProfileName": "", "clusterName": "", "podExecutionRoleArn": "", "subnets": [
"subnet-1", "subnet-2"
], "selectors": [
{
"namespace": "", "key-1": "label", "key-2": "label"
}
]
}
- fargateProfileName: Unique name to identify fargate profile object
- clusterName: Name of the cluster that the profile to be attached
- podExecutionRoleArn: Amazon Resource Name (example: arn:aws:iam::xxx:role/AmazonEKSFargatePodExecutionRole) of a pod execution role to use with pods on Fargate. This role is added to the cluster’s Kubernetes RBAC for authorization so that kubelet that is running on the Fargate infrastructure can register with Amazon EKS cluster and appear in the cluster as a node.
- subnets: Subnet IDs to be associated with the profile. In case of existing clusters users can choose the Subnet IDs of the active cluster. As AWS CNI uses IP’s from a definitive set of subnets allocated while creation, the same subnets can be chosen to cater Fargate allowing communication between pods and services in the cluster. As of now only private subnets (with no direct route to an Internet Gateway) are accepted for this parameter.
- selectors: A Selector is a ‘key:value’ pair to match pods to use a Fargate profile. Each selector must have an associated namespace. Optionally, users can also specify labels for a namespace. Users may specify up to five selectors in a Fargate profile. A pod only needs to match one selector to run using the Fargate profile.
For example, a user wants to deploy Fargate on EKS where all Kubernetes pods created in namespaces: kube-system and default are provisioned on Fargate. In this scenario CoreDNS — the only EKS control plane component deployed on user infrastructure will also be created on Fargate as user has selected ‘kube-system’ namespace.
Fargate Profile Configuration:
{
"fargateProfileName": "fargate-profile-1", "clusterName": "eks-kube-fargate", "podExecutionRoleArn": "arn:aws:iam::xxx:role/AmazonEKSFargatePodExecutionRole", "subnets": [
"subnet-0968a124a4e4b0afe", "subnet-0723bbe802a360eb9"
], "selectors": [
{
"namespace": "kube-system",
"namespace": "default"
}
]
}
Optionally in the pod selector menu, users can add specific tags that allow to specify which pods from a specific namespace run on Fargate using this profile.
Multiple fargate-profiles can be attached to a EKS Cluster.
Deploying Pods on Fargate
Two Fargate profiles are created, first specifying namespaces: kube-system and default and the other specifying a namespace “eks-fargate” and a label “environment:fargate”. A pod execution role is selected for the profile (ARN).
As ‘kube-system’ is selected in profile-1, CoreDNS pods will be scheduled on Fargate. Users can control deployment of CoreDNS pods on a EC2 cluster removing the namespace from the profile above.
As seen above, two CoreDNS pods (deployment-replicas:2) are created on Fargate and provided with IP’s from the subnets configured in the profile. As in Fargate AWS manages the underlying infrastructure the compute capacity to run the pods is distributed across random nodes managed by AWS.
EKS auto-assigns specific labels required to segregate Fargate based pods from others (if exist’s) which includes fargate-profile used. Optionally users can provide labels to a fargate profile which will be added to pod labels.
Fargate-profile “fargate-profile-2” created above, selector configuration includes selection of a namespace “eks-fargate” and a custom label “environment:fargate”. A sample nginx deployment is deployed using the manifest below which only includes “namespace:eks-fargate”:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-aws
namespace: eks-fargate
spec:
selector:
matchLabels:
app: nginx-aws
replicas: 1
template:
metadata:
labels:
app: nginx-aws
spec:
containers:
- name: nginx-aws
image: nginx
ports: - containerPort: 80
As “fargate-profile-2” include a label “environment:fargate” along with “namespace:eks-fargate”, the deployment above will remain in pending state as there are no other EC2 Kubernetes nodes in the cluster.
This shows the ability to control scheduling of pods on Fargate using custom labels along with a namespace. Pods from the same namespace can run on EC2 or Faragate. As shown below the pod is scheduled on Fargate once a label “environment:fargate” is added.
Limitations
There are certain limitations in using native EKS features with Fargate on EKS. Most of these can be mitigated having a mixed cluster (EC2+Fargate), where Kubernetes objects with complex requirements can use EC2-Nodes and other pods having minimal requirements can be deployed on Fargate which can significantly reduce the number of EC2 instances in a cluster.
Usecases
- If a workload is consistent for CPU or memory requirements (in and around: 4 vCPU and 30Gb) and has no complex requirements (storage, LB, etc.), users can schedule these specific set of workloads on Fargate rather using EC2 for compute capacity.
- When the workload is made up of consistent periodic tasks, such as a cron job or occasional jobs that come from a queue then AWS Fargate can be leveraged. Instead of paying for an EC2 instance, AWS Fargate can be used to run containers based on requirements.
- In case, the workload is small and simple (website or a tier in an application ) where the application has more traffic during the day and low traffic at night then AWS Fargate is again the best choice. Users may scale down to tiny containers on Fargate at night that costs little and scale up again in the daytime on EC2 when infrastructure requirements are higher.
- For development and testing teams, Fargate can be used to launch pods internally in the organization.
- For tasks that run on-demand or on a schedule and don’t need a dedicated EC2 instance.
*****************************************************************
AWS Fargate on EKS will allow customers to remove undifferentiated heavy lifting of operating worker nodes, whilst also being able to use the Kubernetes API. With Fargate, users pay only for the amount of vCPU and memory and resources that the pod needs to run, which means developers won’t have to worry about overprovisioning because Fargate should run the exact number of resources needed to run that pod at any given moment, and no more.
Amazon EKS and AWS Fargate provide the serverless benefits of Fargate, the best practices of Amazon EKS, and the extensibility of Kubernetes out of the box.
