Sagemaker endpoint deployment not working

Question

Hey everyone,
I am facing some issue in deploying my endpoint on Sagemaker (later to be invoked by a downstream service - ECS).
Here's my code in deploying it:

```
import os, tarfile, uuid, boto3, sagemaker, json

from sagemaker.huggingface import HuggingFaceModel, get_huggingface_llm_image_uri

try:
	role = sagemaker.get_execution_role()
except ValueError:
	iam = boto3.client('iam')
	role = iam.get_role(RoleName='VisionAISageMakerRole ')['Role']['Arn']

hub = {
	'HF_MODEL_ID':'google/paligemma-3b-ft-cococap-448',
	'SM_NUM_GPUS': json.dumps(1),
	'HF_API_TOKEN': os.environ["HF_TOKEN"],
    "HF_TASK": "image-to-text",
}

REGION = "ca-central-1"
sess = sagemaker.Session(boto3.Session(region_name=REGION))

huggingface_model = HuggingFaceModel(
	transformers_version="4.49.0",
    pytorch_version="2.6.0",
    py_version="py312",
	env=hub,
	role=role,
    sagemaker_session=sess,
)

sagemaker_client = boto3.client("sagemaker")
predictor = huggingface_model.deploy(
        initial_instance_count=1,
        instance_type="ml.g4dn.xlarge",
        container_startup_health_check_timeout=300,
    )
```

WHen I run this segment of code on a sagemaker notebook instance (aka try to deploy the endpoint) this is the error i get:

```
W-9000-google__paligemma-3b-ft-c-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - No inference script implementation was found at `inference`. Default implementation of all functions will be used.
```

Here's my directory structure:
./test.ipynb
./inference.py

Both inference the jupyter notebook are in the root.

re:Post Agent · Answer

When deploying a SageMaker endpoint, the error message "No inference script implementation was found at `inference`. Default implementation of all functions will be used" indicates that SageMaker cannot locate your inference script properly.

There are a few issues to address:

1. In your HuggingFaceModel deployment, you haven't specified the path to your inference script. When using the HuggingFaceModel, you need to explicitly provide the source_dir and entry_point parameters to point to your inference implementation.

2. Your current directory structure has the inference.py file in the root directory, but you haven't told SageMaker where to find it.

To fix this issue, modify your HuggingFaceModel initialization to include the source_dir and entry_point parameters:

```python
huggingface_model = HuggingFaceModel(
transformers_version="4.49.0",
pytorch_version="2.6.0",
py_version="py312",
env=hub,
role=role,
source_dir=".", # Point to the directory containing your inference script
entry_point="inference.py", # Specify the inference script filename
sagemaker_session=sess,
)
```

Additionally, ensure that your IAM role has the necessary permissions for creating and managing SageMaker resources. The role needs permissions for:
- sagemaker:CreateModel
- sagemaker:CreateEndpointConfig
- sagemaker:CreateEndpoint
- Permissions to describe and list models, endpoints, and endpoint configurations

If you're still experiencing issues after making these changes, check the CloudWatch logs for the endpoint to identify any other potential problems with your model deployment.  
**Sources**  
[SageMaker Python SDK Troubleshooting Guide - Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-python-sdk-troubleshooting.html)  
[Troubleshoot Amazon SageMaker AI model deployments - Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-model-troubleshoot.html)

Sagemaker endpoint deployment not working

Add your answer

Relevant content