Sagemaker Async endpoint never returns result to app hosted on ECS

I'm currently running a web application on ECS with sagemaker for model predictions. I have a model and endpoint set up for async inference and when I run the application locally inference is successful and I can see on cloudwatch that the backlog increases and is then cleared out. However, when I try the same on the ECS hosted app the backlog size increases but all the invocations fail. Then I check the logs in cloudwatch and I get this for each invocation:

Inference request succeeded. ModelLatency: 49244118 us, RequestDownloadLatency: 175208 us, ResponseUploadLatency: 89918 us, TimeInBacklog: 19685 ms, TotalProcessingTime: 69210 ms

Could it be a networking issue? Any insights would be helpful, thanks. Also, I don't get a notification from the failure SNS topic.