Setup custom inference serverless endpoint on AWS Sagemaker

I'm using a HuggingFace model locally to return a vector instead of the normal pipeline that returns a zero-image classification response and I'm trying to get this to work on Sagemaker Serverless.

from PIL import Image
from transformers import CLIPProcessor, CLIPModel
import numpy as np

....
model = CLIPModel.from_pretrained("patrickjohncyh/fashion-clip")
processor = CLIPProcessor.from_pretrained("patrickjohncyh/fashion-clip")

image = Image.open(image_path)
inputs = processor(images=image, return_tensors="pt", padding=True)
image_vector = model.get_image_features(**inputs).squeeze().detach().numpy()
return image_vector

I've read a lot about how to get things working with a normal HuggingFace model but I'm really struggling to find out how to do this using CLIPProcessor and CLIPModel with a serverless endpoint on Amazon Sagemaker. I hope someone here can help me! Thanks!