Questions tagged with AWS Inferentia

Hello, I tried using [Inf1 EC2 instance](https://aws.amazon.com/ec2/instance-types/inf1/) for deploying my ML model. I need to monitor the GPU usage of the ML model. I could find the CPU usage in the aws console, but not gpu usage. Already tried: 1. https://docs.aws.amazon.com/dlami/latest/devguide/tutorial-gpu-monitoring-gpumon.html This didn't work. It threw this error ``` (python3) ubuntu@ip-xxx-mm-yy-zzz:~/tools/GPUCloudWatchMonitor$ python3 gpumon.py Traceback (most recent call last): File "gpumon.py", line 146, in <module> nvmlInit() File "/home/ubuntu/anaconda3/envs/python3/lib/python3.8/site-packages/pynvml/nvml.py", line 1450, in nvmlInit nvmlInitWithFlags(0) File "/home/ubuntu/anaconda3/envs/python3/lib/python3.8/site-packages/pynvml/nvml.py", line 1440, in nvmlInitWithFlags _nvmlCheckReturn(ret) File "/home/ubuntu/anaconda3/envs/python3/lib/python3.8/site-packages/pynvml/nvml.py", line 765, in _nvmlCheckReturn raise NVMLError(ret) pynvml.nvml.NVMLError_DriverNotLoaded: Driver Not Loaded ``` Also, `nvidia-smi` didn't work ``` (python3) ubuntu@ip-xxx-mm-yy-zzz:~/tools/GPUCloudWatchMonitor$ nvidia-smi NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running. ``` Kindly provide some help to monitor GPU usage.

I am trying to deploy Pytorch model on ml.inf1.xlarge instance. Image: 301217895009.dkr.ecr.us-west-2.amazonaws.com/sagemaker-neo-pytorch:1.5.1-inf-py3 My python code using some oepncv functions, and when I am trying to run the infernce I got the following error: ModuleNotFoundError: No module named 'cv2' I tried to add opencv-python-headless to requirements.txt, but then I got another error ImportError: libgthread-2.0.so.0: cannot open shared object file How I can use opencv with the ml.inf1 instances?

We have been trying to deploy our multiple models to a multi-model endpoint that uses inference machines (inf.xlarge) without luck. ClientError: An error occurred (ValidationException) when calling the CreateEndpointConfig operation: MultiModel mode is not supported for instance type ml.inf1.xlarge. This isn't good, is that really the case, or have we messed up somewhere during the process? Thanks