Easily get started with Amazon EC2 Inferentia2 or Trainium1 instances with Hugging Face's Neuron DLAMI

If you are getting started with AWS Inferentia or AWS Trainium, an easy way to deploy an EC2 instance is using an existing Amazon Machine Image (AMI) that has all the libraries and drivers pre-installed.

This article walks through how to deploy an EC2 instance using the Hugging Face Neuron Deep Learning AMI.

1. First, decide what region(s) to deploy in. See this article for a list of regions that support Inferentia or Trainium instances. If you don't have any existing infrastructure or other requirements, consider us-east-1 or us-west-2 for lower prices.

2. Next, make sure that you have increased your service quota for Inferentia or Trainium for that region (see this article for details). (you can request quota for multiple regions without incurring any extra charges)

3. Now you should be ready to launch an EC2 instance. There are screen shots below, but your console may look slightly different. For more details on deploying an EC2 instance in general here.

4. The most important step is in the Application and OS Images (Amazon Machine Image) section. Enter "Neuron" in the catalog search bar and hit Enter.
   

5. You will see the images created by Amazon, but to find the Hugging Face version, click on "AWS Marketplace AMIs".

6. You should click "Select" and "Subscribe now". It may look like there is a cost for the image, but the cost shown is only the cost for the instance. There is no charge for the use of the AMI.

7. Select your instance type. If you are testing on a single core (for example, tracing or programing the Neuron Kernel Interface), consider an inf2.xlarge (least expensive). If you are running out of system RAM, consider a trn1.2xlarge or inf2.8xlarge. They all come with a single Neuron device with two cores but vary in the cost, amount of CPU cores, and CPU RAM.

8. Create a Key pair if you don't have one (and save the private key!)

9. Create a security group to allow incoming SSH traffic (if you don't have one) (if you will be accessing your instance through AWS System Manager or a cloud desktop, you may not need this)

10. Change your storage if you are loading very large or very small models.

11. Launch the instance! Then you can click on the instance details to find the external IP address. If you need help connecting with SSH, see the instructions here.

12. When you connect, you should see an announcement that details information about the environment and other Python virtual environments that are available (pytorch 2.1 is the default)