Unable to SSH to EC2 instance after successfully running Instance Connect send-ssh-public-key command

Question

**Quick TLDR issue summary**

I am able to successfully run `aws ec2-instance-connect send-ssh-public-key ...`, however when I try to connect via SSH, I get `Permission denied (publickey,gssapi-keyex,gssapi-with-mic).`.

I am also unable to connect via the Connect option in the AWS Console.

**Details**

I have an EC2 with the following details:

* AMI: `al2023-ami-minimal-2023.4.20240416.0-kernel-6.1-arm64`
* Network: Public subnet with public IP address and hostname
* For the purpose of testing, I've allowed ALL inbound TCP on all ports and all IPv4 addresses
* Region: eu-west-2

I am able to successfully run `aws ec2-instance-connect send-ssh-public-key` as follows:

```
aws ec2-instance-connect send-ssh-public-key 
  --region eu-west-2 
  --instance-id INSTANCE_ID 
  --instance-os-user ec2-user 
  --ssh-public-key file://.ssh/MY_INSTANCE_CONNECT_KEY.pub
```

The output I get is:

```
{
    "RequestId": "REQUEST_ID",
    "Success": true
}
```

Immediately after (definitely before 60s), I run:

```
ssh -i ~/.ssh/MY_INSTANCE_CONNECT_KEY ec2-user@HOSTNAME
```

And I get:

> ec2-user@HOSTNAME: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

I have also tried using Instance Connect via the AWS Console in the browser, and I get this:

> Failed to connect to your instance
> Error establishing SSH connection to your instance. Try again later.

![Enter image description here](/media/postImages/original/IM_ka-kOFwT0WaSx3yjVVrzA)

**Debugging done so far**

1. When I create the EC2 instance, I assign it an SSH key using a key pair I've set in EC2, I am able to connect using this key with no issues, so I can confirm that SSH is definitely working on the server.
 2. When I run `send-ssh-public-key` on my local machine and then connect to the server using the other SSH key (set via key pairs), I cannot see the key I set via Instance Connect when running `cat /home/ec2-user/.ssh/authorized_keys`.
 3. The permissions of `authorized_keys` is 600
 4. I've tried using both `ed25519` and `rsa` format keys
 5. I've tried adding `-o "IdentitiesOnly=yes"` to the SSH command as suggested by Riku, and unfortunately this didn't change anything

I'm at a bit of a loss as to what the issue could be here.

I would be grateful if anyone could suggest any other debugging steps I can follow to identify what the cause might be?

Thanks in advance

Accepted Answer

The issue for me was the AMI I was using: `al2023-ami-minimal-2023.4.20240416.0-kernel-6.1-arm64`

When I switched the the following image, the issue resolved: `amazon/al2023-ami-2023.4.20240416.0-kernel-6.1-arm64`

I suspect that the image with `miminal` in the name does not work properly with AWS Instance Connect.

Answer

Hello.

If you look at this document, the command option "-o "IdentitiesOnly=yes"" is set.  
Also, the expiration date is 60 seconds after executing "send-ssh-public-key".  
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-methods.html#ec2-instance-connect-connecting-aws-cli  
> Use the ssh command to connect to the instance using the private key before the public key is removed from the instance metadata (you have 60 seconds before it is removed). Specify the private key that corresponds to the public key, the default username for the AMI that you used to launch your instance, and the instance's public DNS name (if connecting over a private network, specify the private DNS name or IP address). Add the IdentitiesOnly=yes option to ensure that only the files in the ssh config and the specified key are used for the connection.

```
ssh -o "IdentitiesOnly=yes" -i my_key ec2-user@ec2-198-51-100-1.compute-1.amazonaws.com
```

> When I run send-ssh-public-key on my local machine and then connect to the server using the other SSH key (set via key pairs), I cannot see the key I set via Instance Connect when running cat /home/ec2-user/.ssh/authorized_keys.

EC2 Instance Connect generates a temporary key for SSH connections and uses it to connect to your EC2 instance.  
So, I thought that it might not be possible to check with /home/ec2-user/.ssh/authorized_keys.  
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect-linux-inst-eic.html  
> When you connect to an instance using EC2 Instance Connect, the Instance Connect API pushes an SSH public key to the instance metadata where it remains for 60 seconds. An IAM policy attached to your user authorizes your user to push the public key to the instance metadata. The SSH daemon uses AuthorizedKeysCommand and AuthorizedKeysCommandUser, which are configured when Instance Connect is installed, to look up the public key from the instance metadata for authentication, and connects you to the instance.

By the way, have you cleared the prerequisites listed in the document below?  
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-prerequisites.html

Unable to SSH to EC2 instance after successfully running Instance Connect send-ssh-public-key command

Relevant content