test-mcp-glama

--- title: "NodeClasses" linkTitle: "NodeClasses" weight: 2 description: > Configure AWS-specific settings with EC2NodeClasses --- Node Classes enable configuration of AWS specific settings. Each NodePool must reference an EC2NodeClass using `spec.template.spec.nodeClassRef`. Multiple NodePools may point to the same EC2NodeClass. ```yaml apiVersion: karpenter.sh/v1beta1 kind: NodePool metadata: name: default spec: template: spec: nodeClassRef: apiVersion: karpenter.k8s.aws/v1beta1 kind: EC2NodeClass name: default --- apiVersion: karpenter.k8s.aws/v1beta1 kind: EC2NodeClass metadata: name: default spec: # Required, resolves a default ami and userdata amiFamily: AL2 # Required, discovers subnets to attach to instances # Each term in the array of subnetSelectorTerms is ORed together # Within a single term, all conditions are ANDed subnetSelectorTerms: # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}" # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921" - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" environment: test - id: subnet-09fa4a0a8f233a921 # Required, discovers security groups to attach to instances # Each term in the array of securityGroupSelectorTerms is ORed together # Within a single term, all conditions are ANDed securityGroupSelectorTerms: # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag # AND the "environment: test" tag OR any security group with the "my-security-group" name # OR any security group with ID "sg-063d7acfb4b06c82c" - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" environment: test - name: my-security-group - id: sg-063d7acfb4b06c82c # Optional, IAM role to use for the node identity. # The "role" field is immutable after EC2NodeClass creation. This may change in the # future, but this restriction is currently in place today to ensure that Karpenter # avoids leaking managed instance profiles in your account. # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes role: "KarpenterNodeRole-${CLUSTER_NAME}" # Optional, IAM instance profile to use for the node identity. # Must specify one of "role" or "instanceProfile" for Karpenter to launch nodes instanceProfile: "KarpenterNodeInstanceProfile-${CLUSTER_NAME}" # Optional, discovers amis to override the amiFamily's default amis # Each term in the array of amiSelectorTerms is ORed together # Within a single term, all conditions are ANDed amiSelectorTerms: # Select on any AMI that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag # AND the "environment: test" tag OR any AMI with the "my-ami" name # OR any AMI with ID "ami-123" - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" environment: test - name: my-ami - id: ami-123 # Optional, overrides autogenerated userdata with a merge semantic userData: | echo "Hello world" # Optional, propagates tags to underlying EC2 resources tags: team: team-a app: team-a-app # Optional, configures IMDS for the instance metadataOptions: httpEndpoint: enabled httpProtocolIPv6: disabled httpPutResponseHopLimit: 2 httpTokens: required # Optional, configures storage devices for the instance blockDeviceMappings: - deviceName: /dev/xvda ebs: volumeSize: 100Gi volumeType: gp3 iops: 10000 encrypted: true kmsKeyID: "1234abcd-12ab-34cd-56ef-1234567890ab" deleteOnTermination: true throughput: 125 snapshotID: snap-0123456789 # Optional, configures detailed monitoring for the instance detailedMonitoring: true status: # Resolved subnets subnets: - id: subnet-0a462d98193ff9fac zone: us-east-2b - id: subnet-0322dfafd76a609b6 zone: us-east-2c - id: subnet-0727ef01daf4ac9fe zone: us-east-2b - id: subnet-00c99aeafe2a70304 zone: us-east-2a - id: subnet-023b232fd5eb0028e zone: us-east-2c - id: subnet-03941e7ad6afeaa72 zone: us-east-2a # Resolved security groups securityGroups: - id: sg-041513b454818610b name: ClusterSharedNodeSecurityGroup - id: sg-0286715698b894bca name: ControlPlaneSecurityGroup-1AQ073TSAAPW # Resolved AMIs amis: - id: ami-01234567890123456 name: custom-ami-amd64 requirements: - key: kubernetes.io/arch operator: In values: - amd64 - id: ami-01234567890123456 name: custom-ami-arm64 requirements: - key: kubernetes.io/arch operator: In values: - arm64 # Generated instance profile name from "role" instanceProfile: "${CLUSTER_NAME}-0123456778901234567789" ``` Refer to the [NodePool docs]({{<ref "./nodepools" >}}) for settings applicable to all providers. To explore various `EC2NodeClass` configurations, refer to the examples provided [in the Karpenter Github repository](https://github.com/aws/karpenter/blob/v0.32.0/examples/v1beta1/). ## spec.amiFamily AMIFamily is a required field, dictating both the default bootstrapping logic for nodes provisioned through this `EC2NodeClass` but also selecting a group of recommended, latest AMIs by default. Currently, Karpenter supports `amiFamily` values `AL2`, `Bottlerocket`, `Ubuntu`, `Windows2019`, `Windows2022` and `Custom`. GPUs are only supported by default with `AL2` and `Bottlerocket`. The `AL2` amiFamily does not support ARM64 GPU instance types unless you specify custom [`amiSelectorTerms`]({{<ref "#specamiselectorterms" >}}). Default bootstrapping logic is shown below for each of the supported families. ### AL2 ```bash MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="//" --// Content-Type: text/x-shellscript; charset="us-ascii" #!/bin/bash -xe exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1 /etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \ --dns-cluster-ip '10.100.0.10' \ --use-max-pods false \ --container-runtime containerd \ --kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110' --//-- ``` ### Bottlerocket ```toml [settings] [settings.kubernetes] api-server = 'https://test-cluster' cluster-certificate = 'ca-bundle' cluster-name = 'test-cluster' cluster-dns-ip = '10.100.0.10' max-pods = 110 [settings.kubernetes.node-labels] 'karpenter.sh/capacity-type' = 'on-demand' 'karpenter.sh/nodepool' = 'test' ``` ### Ubuntu ```bash MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="//" --// Content-Type: text/x-shellscript; charset="us-ascii" #!/bin/bash -xe exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1 /etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \ --dns-cluster-ip '10.100.0.10' \ --use-max-pods false \ --kubelet-extra-args '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' --//-- ``` ### Windows2019 ```powershell <powershell> [string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1" & $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10' </powershell> ``` ### Windows2022 ```powershell <powershell> [string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1" & $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test" --max-pods=110' -DNSClusterIP '10.100.0.10' </powershell> ``` {{% alert title="Note" color="primary" %}} Karpenter will automatically query for the appropriate [EKS optimized AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-amis.html) via AWS Systems Manager (SSM). In the case of the `Custom` AMIFamily, no default AMIs are defined. As a result, `amiSelectorTerms` must be specified to inform Karpenter on which custom AMIs are to be used. {{% /alert %}} ### Custom The `Custom` AMIFamily ships without any default userData to allow you to configure custom bootstrapping for control planes or images that don't support the default methods from the other families. ## spec.subnetSelectorTerms Subnet Selector Terms allow you to specify selection logic for a set of subnet options that Karpenter can choose from when launching an instance from the `EC2NodeClass`. Karpenter discovers subnets through the `EC2NodeClass` using ids or [tags](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html). When launching nodes, a subnet is automatically chosen that matches the desired zone. If multiple subnets exist for a zone, the one with the most available IP addresses will be used. This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It's possible that you may want to select on two different subnets that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled. ```yaml subnetSelectorTerms: # Select on any subnet that has the "karpenter.sh/discovery: ${CLUSTER_NAME}" # AND the "environment: test" tag OR any subnet with ID "subnet-09fa4a0a8f233a921" - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" environment: test - id: subnet-09fa4a0a8f233a921 ``` {{% alert title="Tip" color="secondary" %}} Subnets may be specified by any tag, including `Name`. Selecting tag values using wildcards (`*`) is supported. {{% /alert %}} #### Examples Select all with a specified tag key: ```yaml spec: subnetSelectorTerms: - tags: karpenter.sh/discovery/MyClusterName: '*' ``` Select by name and tag (all criteria must match): ```yaml spec: subnetSelectorTerms: - tags: Name: my-subnet MyTag: '' # matches all resources with the tag ``` Select using multiple tag terms: ```yaml spec: subnetSelectorTerms: - tags: Name: "my-subnet-1" - tags: Name: "my-subnet-2" ``` Select using wildcards: ```yaml spec: subnetSelectorTerms: - tags: Name: "*Public*" ``` Select using ids: ```yaml spec: subnetSelectorTerms: - id: "subnet-09fa4a0a8f233a921" - id: "subnet-0471ca205b8a129ae" ``` ## spec.securityGroupSelectorTerms Security Group Selector Terms allow you to specify selection logic for all security groups that will be attached to an instance launched from the `EC2NodeClass`. The security group of an instance is comparable to a set of firewall rules. [EKS creates at least two security groups by default](https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html). This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It's possible that you may want to select on two different security groups that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled. ```yaml securityGroupSelectorTerms: # Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag # AND the "environment: test" tag OR any security group with the "my-security-group" name # OR any security group with ID "sg-063d7acfb4b06c82c" - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" environment: test - name: my-security-group - id: sg-063d7acfb4b06c82c ``` {{% alert title="Tip" color="secondary" %}} Security groups may be specified by any tag, including "Name". Selecting tags using wildcards (`*`) is supported. {{% /alert %}} {{% alert title="Note" color="primary" %}} When launching nodes, Karpenter uses all the security groups that match the selector. If you choose to use the `kubernetes.io/cluster/$CLUSTER_NAME` tag for discovery, note that this may result in failures using the AWS Load Balancer controller. The Load Balancer controller only supports a single security group having that tag key. See [this issue](https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/2367) for more details. To verify if this restriction affects you, run the following commands. ```bash CLUSTER_VPC_ID="$(aws eks describe-cluster --name $CLUSTER_NAME --query cluster.resourcesVpcConfig.vpcId --output text)" aws ec2 describe-security-groups --filters Name=vpc-id,Values=$CLUSTER_VPC_ID Name=tag-key,Values=kubernetes.io/cluster/$CLUSTER_NAME --query 'SecurityGroups[].[GroupName]' --output text ``` If multiple securityGroups are printed, you will need more specific securityGroupSelectorTerms. We generally recommend that you use the `karpenter.sh/discovery: $CLUSTER_NAME` tag selector instead. {{% /alert %}} #### Examples Select all assigned to a cluster: ```yaml spec: securityGroupSelectorTerms: - tags: kubernetes.io/cluster/$CLUSTER_NAME: "owned" ``` Select all with a specified tag key: ```yaml spec: securityGroupSelectorTerms: - tags: MyTag: '*' ``` Select by name and tag (all criteria must match): ```yaml spec: securityGroupSelectorTerms: - name: my-security-group tags: MyTag: '*' # matches all resources with the tag ``` Select using multiple tag terms: ```yaml spec: securityGroupSelectorTerms: - tags: Name: "my-security-group-1" - tags: Name: "my-security-group-2" ``` Select by name using a wildcard: ```yaml spec: securityGroupSelectorTerms: - name: "*Public*" ``` Select using ids: ```yaml spec: securityGroupSelectorTerms: - id: "sg-063d7acfb4b06c82c" - id: "sg-06e0cf9c198874591" ``` ## spec.amiSelectorTerms AMI Selector Terms are used to configure custom AMIs for Karpenter to use, where the AMIs are discovered through ids, owners, name, and [tags](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html). **When you specify `amiSelectorTerms`, you fully override the default AMIs that are selected on by your EC2NodeClass [`amiFamily`]({{< ref "#specamifamily" >}}).** {{% alert title="Note" color="primary" %}} [`amiFamily`]({{< ref "#specamifamily" >}}) determines the bootstrapping mode, while `amiSelectorTerms` specifies specific AMIs to be used. Therefore, you need to ensure consistency between [`amiFamily`]({{< ref "#specamifamily" >}}) and `amiSelectorTerms` to avoid conflicts during bootstrapping. {{% /alert %}} This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It's possible that you may want to select on two different AMIs that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled. ```yaml amiSelectorTerms: # Select on any AMI that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag # AND the "environment: test" tag OR any AMI with the "my-ami" name # OR any AMI with ID "ami-123" - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" environment: test - name: my-ami - id: ami-123 ``` This field is optional, and Karpenter will use the latest EKS-optimized AMIs for the AMIFamily if no amiSelectorTerms are specified. To select an AMI by name, use the `name` field in the selector term. To select an AMI by id, use the `id` field in the selector term. To ensure that AMIs are owned by the expected owner, use the `owner` field - you can use a combination of account aliases (e.g. `self` `amazon`, `your-aws-account-name`) and account IDs. If owner is not set for `name`, it defaults to `self,amazon`, preventing Karpenter from inadvertently selecting an AMI that is owned by a different account. Tags don't require an owner as tags can only be discovered by the user who created them. {{% alert title="Tip" color="secondary" %}} AMIs may be specified by any AWS tag, including `Name`. Selecting by tag or by name using wildcards (`*`) is supported. {{% /alert %}} {{% alert title="Note" color="primary" %}} If `amiSelectorTerms` match more than one AMI, Karpenter will automatically determine which AMI best fits the workloads on the launched worker node under the following constraints: * When launching nodes, Karpenter automatically determines which architecture a custom AMI is compatible with and will use images that match an instanceType's requirements. * Note that Karpenter **cannot** detect any requirement other than architecture. If you need to specify different AMIs for different kind of nodes (e.g. accelerated GPU AMIs), you should use a separate `EC2NodeClass`. * If multiple AMIs are found that can be used, Karpenter will choose the latest one. * If no AMIs are found that can be used, then no nodes will be provisioned. {{% /alert %}} #### Examples Select all with a specified tag: ```yaml amiSelectorTerms: - tags: karpenter.sh/discovery/MyClusterName: '*' ``` Select by name: ```yaml amiSelectorTerms: - name: my-ami ``` Select by `Name` tag: ```yaml amiSelectorTerms: - tags: Name: my-ami ``` Select by name and owner: ```yaml amiSelectorTerms: - name: my-ami owner: self - name: my-ami owner: "0123456789" ``` Select by name using a wildcard: ```yaml spec: amiSelectorTerms: - name: "*EKS*" ``` Select by all under an owner: ```yaml spec: amiSelectorTerms: - name: "*" owner: self ``` Specify using ids: ```yaml amiSelectorTerms: - id: "ami-123" - id: "ami-456" ``` ## spec.role `Role` is an optional field and tells Karpenter which IAM identity nodes should assume. You must specify one of `role` or `instanceProfile` when creating a Karpenter `EC2NodeClass`. If using the [Karpenter Getting Started Guide]({{<ref "../getting-started/getting-started-with-karpenter" >}}) to deploy Karpenter, you can use the `KarpenterNodeRole-$CLUSTER_NAME` role provisioned by that process. ```yaml spec: role: "KarpenterNodeRole-$CLUSTER_NAME" ``` ## spec.instanceProfile `InstanceProfile` is an optional field and tells Karpenter which IAM identity nodes should assume. You must specify one of `role` or `instanceProfile` when creating a Karpenter `EC2NodeClass`. If you use the `instanceProfile` field instead of `role`, Karpenter will not manage the InstanceProfile on your behalf; instead, it expects that you have pre-provisioned an IAM instance profile and assigned it a role. You can provision and assign a role to an IAM instance profile using [CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-instanceprofile.html) or by using the [`aws iam create-instance-profile`](https://docs.aws.amazon.com/cli/latest/reference/iam/create-instance-profile.html) and [`aws iam add-role-to-instance-profile`](https://docs.aws.amazon.com/cli/latest/reference/iam/add-role-to-instance-profile.html) commands in the CLI. {{% alert title="Note" color="primary" %}} For [private clusters](https://docs.aws.amazon.com/eks/latest/userguide/private-clusters.html) that do not have access to the public internet, using `spec.instanceProfile` is required. `spec.role` cannot be used since Karpenter needs to access IAM endpoints to manage a generated instance profile. IAM [doesn't support private endpoints](https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-privatelink-support.html) to enable accessing the service without going to the public internet. {{% /alert %}} ## spec.tags Karpenter adds tags to all resources it creates, including EC2 Instances, EBS volumes, and Launch Templates. The default set of tags are listed below. ```yaml Name: <node-name> karpenter.sh/nodeclaim: <nodeclaim-name> karpenter.sh/nodepool: <nodepool-name> karpenter.k8s.aws/ec2nodeclass: <ec2nodeclass-name> kubernetes.io/cluster/<cluster-name>: owned ``` Additional tags can be added in the tags section, which will be merged with the default tags specified above. ```yaml spec: tags: InternalAccountingTag: 1234 dev.corp.net/app: Calculator dev.corp.net/team: MyTeam ``` {{% alert title="Note" color="primary" %}} Karpenter allows overrides of the default "Name" tag but does not allow overrides to restricted domains (such as "karpenter.sh", "karpenter.k8s.aws", and "kubernetes.io/cluster"). This ensures that Karpenter is able to correctly auto-discover nodes that it owns. {{% /alert %}} ## spec.metadataOptions Control the exposure of [Instance Metadata Service](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) on EC2 Instances launched by this EC2NodeClass using a generated launch template. Refer to [recommended, security best practices](https://aws.github.io/aws-eks-best-practices/security/docs/iam/#restrict-access-to-the-instance-profile-assigned-to-the-worker-node) for limiting exposure of Instance Metadata and User Data to pods. If metadataOptions are omitted from this EC2NodeClass, the following default settings are applied: ```yaml spec: metadataOptions: httpEndpoint: enabled httpProtocolIPv6: disabled httpPutResponseHopLimit: 2 httpTokens: required ``` ## spec.blockDeviceMappings The `blockDeviceMappings` field in an `EC2NodeClass` can be used to control the [Elastic Block Storage (EBS) volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html#instance-block-device-mapping) that Karpenter attaches to provisioned nodes. Karpenter uses default block device mappings for the AMIFamily specified. For example, the `Bottlerocket` AMI Family defaults with two block device mappings, one for Bottlerocket's control volume and the other for container resources such as images and logs. ```yaml spec: blockDeviceMappings: - deviceName: /dev/xvda ebs: volumeSize: 100Gi volumeType: gp3 iops: 10000 encrypted: true kmsKeyID: "1234abcd-12ab-34cd-56ef-1234567890ab" deleteOnTermination: true throughput: 125 snapshotID: snap-0123456789 ``` The following blockDeviceMapping defaults are used for each `AMIFamily` if no `blockDeviceMapping` overrides are specified in the `EC2NodeClass` ### AL2 ```yaml spec: blockDeviceMappings: - deviceName: /dev/xvda ebs: volumeSize: 20Gi volumeType: gp3 encrypted: true ``` ### Bottlerocket ```yaml spec: blockDeviceMappings: # Root device - deviceName: /dev/xvda ebs: volumeSize: 4Gi volumeType: gp3 encrypted: true # Data device: Container resources such as images and logs - deviceName: /dev/xvdb ebs: volumeSize: 20Gi volumeType: gp3 encrypted: true ``` ### Ubuntu ```yaml spec: blockDeviceMappings: - deviceName: /dev/sda1 ebs: volumeSize: 20Gi volumeType: gp3 encrypted: true ``` ### Windows2019/Windows2022 ```yaml spec: blockDeviceMappings: - deviceName: /dev/sda1 ebs: volumeSize: 50Gi volumeType: gp3 encrypted: true ``` ### Custom The `Custom` AMIFamily ships without any default `blockDeviceMappings`. ## spec.userData You can control the UserData that is applied to your worker nodes via this field. This allows you to run custom scripts or pass-through custom configuration to Karpenter instances on start-up. ```yaml apiVersion: karpenter.k8s.aws/v1beta1 kind: EC2NodeClass metadata: name: bottlerocket-example spec: ... amiFamily: Bottlerocket userData: | [settings.kubernetes] "kube-api-qps" = 30 "shutdown-grace-period" = "30s" "shutdown-grace-period-for-critical-pods" = "30s" [settings.kubernetes.eviction-hard] "memory.available" = "20%" ``` This example adds SSH keys to allow remote login to the node (replace *my-authorized_keys* with your key file): {{% alert title="Note" color="primary" %}} Instead of using SSH as set up in this example, you can use Session Manager (SSM) or EC2 Instance Connect to gain shell access to Karpenter nodes. See [Node NotReady]({{< ref "../troubleshooting/#node-notready" >}}) troubleshooting for an example of starting an SSM session from the command line or [EC2 Instance Connect](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-set-up.html) documentation to connect to nodes using SSH. {{% /alert %}} ```yaml apiVersion: karpenter.k8s.aws/v1beta1 kind: EC2NodeClass metadata: name: al2-example spec: ... amiFamily: AL2 userData: | #!/bin/bash mkdir -p ~ec2-user/.ssh/ touch ~ec2-user/.ssh/authorized_keys cat >> ~ec2-user/.ssh/authorized_keys <<EOF {{ insertFile "../my-authorized_keys" | indent 4 }} EOF chmod -R go-w ~ec2-user/.ssh/authorized_keys chown -R ec2-user ~ec2-user/.ssh ``` For more examples on configuring fields for different AMI families, see the [examples here](https://github.com/aws/karpenter/blob/v0.32.0/examples/v1beta1/). Karpenter will merge the userData you specify with the default userData for that AMIFamily. See the [AMIFamily]({{< ref "#specamifamily" >}}) section for more details on these defaults. View the sections below to understand the different merge strategies for each AMIFamily. ### AL2/Ubuntu * Your UserData can be in the [MIME multi part archive](https://cloudinit.readthedocs.io/en/latest/topics/format.html#mime-multi-part-archive) format. * Karpenter will transform your custom user-data as a MIME part, if necessary, and then merge a final MIME part to the end of your UserData parts which will bootstrap the worker node. Karpenter will have full control over all the parameters being passed to the bootstrap script. * Karpenter will continue to set MaxPods, ClusterDNS and all other parameters defined in `spec.kubeletConfiguration` as before. Consider the following example to understand how your custom UserData will be merged - #### Passed-in UserData (bash) ```bash #!/bin/bash echo "Running custom user data script" ``` #### Passed-in UserData (MIME) ```bash MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="BOUNDARY" --BOUNDARY Content-Type: text/x-shellscript; charset="us-ascii" #!/bin/bash echo "Running custom user data script" --BOUNDARY-- ``` #### Merged UserData ```bash MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="//" --// Content-Type: text/x-shellscript; charset="us-ascii" #!/bin/bash echo "Running custom user data script" --// Content-Type: text/x-shellscript; charset="us-ascii" #!/bin/bash -xe exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1 /etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \ --use-max-pods false \ --container-runtime containerd \ --kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110' --//-- ``` {{% alert title="Note" color="primary" %}} You can also set kubelet-config properties by modifying the kubelet-config.json file before the EKS bootstrap script starts the kubelet: ```yaml apiVersion: karpenter.k8s.aws/v1beta1 kind: EC2NodeClass metadata: name: kubelet-config-example spec: ... amiFamily: AL2 userData: | #!/bin/bash echo "$(jq '.kubeAPIQPS=50' /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json ``` {{% /alert %}} ### Bottlerocket * Your UserData must be valid TOML. * Karpenter will automatically merge settings to ensure successful bootstrap including `cluster-name`, `api-server` and `cluster-certificate`. Any labels and taints that need to be set based on pod requirements will also be specified in the final merged UserData. * All Kubelet settings that Karpenter applies will override the corresponding settings in the provided UserData. For example, if you've specified `settings.kubernetes.cluster-name`, it will be overridden. * If MaxPods is specified via the binary arg to Karpenter, the value will override anything specified in the UserData. * If ClusterDNS is specified via `spec.kubeletConfiguration`, then that value will override anything specified in the UserData. * Unknown TOML fields will be ignored when the final merged UserData is generated by Karpenter. Consider the following example to understand how your custom UserData settings will be merged in. #### Passed-in UserData ```toml [settings.kubernetes.eviction-hard] "memory.available" = "12%" [settings.kubernetes] "unknown-setting" = "unknown" [settings.kubernetes.node-labels] 'field.controlled.by/karpenter' = 'will-be-overridden' ``` #### Merged UserData ```toml [settings] [settings.kubernetes] api-server = 'https://cluster' cluster-certificate = 'ca-bundle' cluster-name = 'cluster' [settings.kubernetes.node-labels] 'karpenter.sh/capacity-type' = 'on-demand' 'karpenter.sh/nodepool' = 'default' [settings.kubernetes.node-taints] [settings.kubernetes.eviction-hard] 'memory.available' = '12%%' ``` ### Windows2019/Windows2022 * Your UserData must be specified as PowerShell commands. * The UserData specified will be prepended to a Karpenter managed section that will bootstrap the kubelet. * Karpenter will continue to set ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before. Consider the following example to understand how your custom UserData settings will be merged in. #### Passed-in UserData ```powershell Write-Host "Running custom user data script" ``` #### Merged UserData ```powershell <powershell> Write-Host "Running custom user data script" [string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1" & $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=spot,karpenter.sh/nodepool=windows2022" --max-pods=110' -DNSClusterIP '10.0.100.10' </powershell> ``` {{% alert title="Windows Support Notice" color="warning" %}} Currently, Karpenter does not specify `-ServiceCIDR` to [EKS Windows AMI Bootstrap script](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-windows-ami.html#bootstrap-script-configuration-parameters). Windows worker nodes will use `172.20.0.0/16` or `10.100.0.0/16` for Kubernetes service IP address ranges based on the IP address of the primary interface. The effective ServiceCIDR can be verified at `$env:ProgramData\Amazon\EKS\cni\config\vpc-bridge.conf` on the worker node. Support for the Windows ServiceCIDR argument can be tracked in a [Karpenter Github Issue](https://github.com/aws/karpenter/issues/4088). Currently, if the effective ServiceCIDR is incorrect for your windows worker nodes, you can add the following userData as a workaround. ```yaml spec: userData: | $global:EKSCluster = Get-EKSCluster -Name my-cluster ``` {{% /alert %}} ### Custom * No merging is performed, your UserData must perform all setup required of the node to allow it to join the cluster. ## spec.detailedMonitoring Enabling detailed monitoring controls the [EC2 detailed monitoring](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html) feature. If you enable this option, the Amazon EC2 console displays monitoring graphs with a 1-minute period for the instances that Karpenter launches. ```yaml spec: detailedMonitoring: true ``` ## status.subnets [`status.subnets`]({{< ref "#statussubnets" >}}) contains the resolved `id` and `zone` of the subnets that were selected by the [`spec.subnetSelectorTerms`]({{< ref "#specsubnetselectorterms" >}}) for the node class. The subnets will be sorted by the available IP address count in decreasing order. #### Examples ```yaml spec: subnetSelectorTerms: - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" status: subnets: - id: subnet-0a462d98193ff9fac zone: us-east-2b - id: subnet-0322dfafd76a609b6 zone: us-east-2c - id: subnet-0727ef01daf4ac9fe zone: us-east-2b - id: subnet-00c99aeafe2a70304 zone: us-east-2a - id: subnet-023b232fd5eb0028e zone: us-east-2c - id: subnet-03941e7ad6afeaa72 zone: us-east-2a ``` ## status.securityGroups [`status.securityGroups`]({{< ref "#statussecuritygroups" >}}) contains the resolved `id` and `name` of the security groups that were selected by the [`spec.securityGroupSelectorTerms`]({{< ref "#specsecuritygroupselectorterms" >}}) for the node class. The subnets will be sorted by the available IP address count in decreasing order. #### Examples ```yaml spec: securityGroupSelectorTerms: - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" status: securityGroups: - id: sg-041513b454818610b name: ClusterSharedNodeSecurityGroup - id: sg-0286715698b894bca name: ControlPlaneSecurityGroup-1AQ073TSAAPW ``` ## status.amis [`status.amis`]({{< ref "#statusamis" >}}) contains the resolved `id`, `name`, and `requirements` of either the default AMIs for the [`spec.amiFamily`]({{< ref "#specamifamily" >}}) or the AMIs selected by the [`spec.amiSelectorTerms`]({{< ref "#specamiselectorterms" >}}) if this field is specified. #### Examples Default AMIs resolved from the AL2 AMIFamily: ```yaml spec: amiFamily: AL2 status: amis: - id: ami-03c3a3dcda64f5b75 name: amazon-linux-2-gpu requirements: - key: kubernetes.io/arch operator: In values: - amd64 - key: karpenter.k8s.aws/instance-gpu-count operator: Exists - id: ami-03c3a3dcda64f5b75 name: amazon-linux-2-gpu requirements: - key: kubernetes.io/arch operator: In values: - amd64 - key: karpenter.k8s.aws/instance-accelerator-count operator: Exists - id: ami-06afb2d101cc4b8bd name: amazon-linux-2-arm64 requirements: - key: kubernetes.io/arch operator: In values: - arm64 - key: karpenter.k8s.aws/instance-gpu-count operator: DoesNotExist - key: karpenter.k8s.aws/instance-accelerator-count operator: DoesNotExist - id: ami-0e28b76d768af234e name: amazon-linux-2 requirements: - key: kubernetes.io/arch operator: In values: - amd64 - key: karpenter.k8s.aws/instance-gpu-count operator: DoesNotExist - key: karpenter.k8s.aws/instance-accelerator-count operator: DoesNotExist ``` AMIs resolved from [`spec.amiSelectorTerms`]({{< ref "#specamiselectorterms" >}}): ```yaml spec: amiFamily: AL2 amiSelectorTerms: - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" status: amis: - id: ami-01234567890123456 name: custom-ami-amd64 requirements: - key: kubernetes.io/arch operator: In values: - amd64 - id: ami-01234567890123456 name: custom-ami-arm64 requirements: - key: kubernetes.io/arch operator: In values: - arm64 ``` ## status.instanceProfile [`status.instanceProfile`]({{< ref "#statusinstanceprofile" >}}) contains the resolved instance profile generated by Karpenter from the [`spec.role`]({{< ref "#specrole" >}}) ```yaml spec: role: "KarpenterNodeRole-${CLUSTER_NAME}" status: instanceProfile: "${CLUSTER_NAME}-0123456778901234567789" ```