Just a gentle reminder as we go into 2024: On February 1, 2024, AWS will begin charging customers for the public IPv4 addresses that we use. Now is a really good time to get familiar with IPv6 networking. For any workloads that still require IPv4 addresses for some reason, it is imperative to get familiar with the options to securely access EC2 instances running in private subnets. In this blog post, I demonstrate how to automate the configuration of one of these options: the EC2 Instance Connect Endpoint. Read on to learn more...

Introduction

In preparation for another blog post, I needed an EC2 instance to load sample data into Aurora Postgres. Normally, I would deploy an EC2 instance with a public IPv4 address and assign the AmazonSSMManagedInstanceCore IAM policy so that I could use Session Manager to remotely access it without bothering with keypairs or keeping up with my ISP's IP address changes for my security groups. However, with the upcoming public IPv4 address pricing change (effective February 1, 2024!), I decided to start placing my test instances in private subnets only and to evaluate how to administer them securely with minimal complexity and, preferably, minimal cost. I investigated using AWS Systems Manager Session Manager, but the required steps for my simple use case brought me back to the drawing board. I then remembered the Endpoint Instance Connect Endpoint announcement from June 2023. I re-read that blog post, and it seemed simple enough. Plus, there is no additional cost to use the service unless you use it to access instances that are in a different subnet than the subnet you assign to the endpoint (i.e. inter-AZ data transfer charges).

Automating the Deployment

I typically use the Pulumi AWS Classic Provider for my automation, but I found the support for the EC2 Instance Connect Endpoint in the AWS Native Provider. Per Pulumi's documentation, the Native provider is still in public preview. So, in my sample code, I use a combination of both the Classic and the Native providers.

Please note that the first time you use the AWS Native provider, you may be prompted to specify a pulumi config setting for your region. You can do so with the following command:

pulumi config set aws-native:region us-east-1

Also, if you will be using this service, be aware of the AMIs that include EC2 Instance Connect already vs. which AMIs will require you to deploy it first before it can be used. You can find that guidance in the AWS Documentation.

Here is my import statement that features the ec2 packages from both providers:

import (
	"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ec2"
	ec2native "github.com/pulumi/pulumi-aws-native/sdk/go/aws/ec2"
	"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/iam"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

Note that I provide an alias (ec2native) for the native provider to clarify which package I'm using for each resource and to mitigate compile errors.

One important prerequisite is to ensure your EC2 Instance Connect Endpoint and your EC2 Instances all have security groups assigned with the following minimal settings:

1. The EC2 Instance Connect Endpoint must have an explicitly-defined outbound rule allowing either all traffic or, optimally, TCP 22 and/or 3389 to the security group(s) or IP addresses of your EC2 instances.

2. The EC2 instances must allow TCP 22 and/or 3389 from the EC2 Instance Connect Endpoint security group.

		// Create a security group for compute resources to access the database.
		computeSg, err := ec2.NewSecurityGroup(ctx, "computeAccessDBSecurityGroup", &ec2.SecurityGroupArgs{
			VpcId: vpc.ID(),
		})
		if err !=  nil {
			return err
		}
		// Create a security group for compute resources to access the database.
		eicSg, err := ec2.NewSecurityGroup(ctx, "ec2InstanceConnectSecurityGroup", &ec2.SecurityGroupArgs{
			VpcId: vpc.ID(),
		})
		if err != nil {
			return err
		}
		// Allow access from EC2 Instance Connect Endpoint on TCP 22.
		_, err = ec2.NewSecurityGroupRule(ctx, "ec2InstanceConnectEndpointToEc2SecurityGroupRule", &ec2.SecurityGroupRuleArgs{
			Type:            pulumi.String("ingress"),
			FromPort:        pulumi.Int(22),
			ToPort:          pulumi.Int(22),
			Protocol:        pulumi.String("tcp"),
			SecurityGroupId: computeSg.ID(),
			SourceSecurityGroupId: eicSg.ID(),
		})
		if err != nil {
           		return err
		}
		// IMPORTANT: Allow outbound traffic from EC2 Instance Connect Endpoint on TCP 22.
		_, err = ec2.NewSecurityGroupRule(ctx, "ec2InstanceConnectEndpointEgressToEc2SecurityGroupRule", &ec2.SecurityGroupRuleArgs{
			Type:            pulumi.String("egress"),
			FromPort:        pulumi.Int(22),
			ToPort:          pulumi.Int(22),
			Protocol:        pulumi.String("tcp"),
			SecurityGroupId: eicSg.ID(),
			SourceSecurityGroupId: computeSg.ID(),
		})
		if err != nil {
			return err
		}

If you forget to do step #1, you will get an error message that may be a little hard to decipher:

Websocket Closure Reason: Unable to connect to target
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

If you forget to do step #2, the EC2 Instance Connect Endpoint will not be able to SSH/RDP to your instance.

The code to deploy the EC2 Instance Connect Endpoint is actually very simple.

		_, err = ec2native.NewInstanceConnectEndpoint(ctx, "instanceConnectEndpoint", &ec2native.InstanceConnectEndpointArgs{
			SubnetId:          subnet1.ID(), // Reference to the private subnet
			SecurityGroupIds:  pulumi.StringArray{eicSg.ID()}, // Reference to the security group
			PreserveClientIp:  pulumi.Bool(false),
		})
		if err != nil {
			return err
		}

I call the ec2native.NewInstanceConnectEndpoint method. The arguments I care about the most are SubnetId, SecurityGroupIds, and PreserveClientIp. The first two parameters consist of the subnet that the EC2 Instance Connect Endpoint will be placed in and the security group it will use. The third parameter determines whether your local ssh/rdp's IP address will be preserved. If you set this option to true, the instance security group(s) must also allow inbound traffic from local IP address. Be aware there are limitations to the client IP preservation feature that you can find in the [AWS Documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect-using-eice.html#ec2-instance-connect-endpoint-limitations.

Once you pulumi up, and the stack deployment completes, you can access the instance with the following command (be sure to substitute your EC2 instance's actual ID for my placeholder ID):

aws ec2-instance-connect ssh --instance-id i-0123456789abcdef

If your AMI does not use the ec2-user user, you can specify it with the --os-user option.

aws ec2-instance-connect ssh --instance-id i-0123456789abcdef --os-user ubuntu

Wrapping Things Up...

In this blog post, I shared how you can use the EC2 Instance Connect Endpoint to securely connect to EC2 instances in your private subnets. Be aware of the prerequisites to use this service AND the potential data transfer charges if your EC2 instances are in different availability zones than your EC2 Instance Connect Endpoint. Also, note that the EC2 Instance Connect Endpoint can take up to 5-6 minutes to deploy. If you want to see the code for my entire test environment automation including the VPC, the EC2 instance, and EC2 Instance Connect Endpoint, you can find it on GitHub.

If you found this article useful, let me know on LinkedIn!