Streaming response from api gateway

Question

Hello, we have an app running in ecs which we access through a rest api gateway. We have just built an endpoint that will stream the response of an llm which works locally however when I try and call this api when it is running in aws it does not stream. It returns is all at once. It does not return the header Transfer-Encoding: chunked. Is there a way to make the rest api gateway return this as a stream? or if I switch it to a http api gateway? I'd rather not have to build a websocket api gateway.

local response
```
content-type: text/event-stream; charset=utf-8
connection: keep-alive
Transfer-Encoding: chunked
```
aws response headers
```
Content-Type: text/event-stream; charset=utf-8
Content-Length: 17000
Connection: keep-alive
```

I have also set up a http api gateway with a HTTP_PROXY integration to our ALB but I got the same response as the rest api gateway

re:Post Agent · Answer

Unfortunately, REST API Gateway does not natively support streaming responses or chunked transfer encoding. The behavior you're experiencing, where the response is returned all at once, is the default for REST API Gateway.

However, you have a few options to achieve streaming-like behavior:

~~1. HTTP API Gateway: Switching to HTTP API Gateway could be a solution. HTTP API Gateway supports response streaming for certain integration types, particularly when integrated with AWS Lambda functions that are configured for response streaming. If your ECS task can be adapted to work with Lambda, this could be a viable option.~~ This is not correct. HTTP API does not support response streaming.

2. Proxy integration: If you're not already using it, you could set up a proxy integration in your REST API Gateway. This would allow the response to pass through from your ECS application without modification by API Gateway. However, this still might not solve the streaming issue entirely, as API Gateway may buffer the response.

3. WebSocket API: While you mentioned preferring not to use WebSocket API Gateway, it is worth noting that this would be the most straightforward way to implement true streaming behavior.

4. Custom solution: You could implement a custom streaming solution at the application level. This might involve breaking your response into smaller chunks and having the client reassemble them, or implementing long-polling techniques.

5. Direct access: If possible, you could consider exposing your ECS service directly (with appropriate security measures) and bypass API Gateway altogether for this specific streaming endpoint.

It's important to note that even if you manage to get chunked transfer encoding working, API Gateway has certain timeout limits that might interfere with long-running streams.

Given your requirements and current setup, switching to HTTP API Gateway and adapting your backend to work with Lambda's response streaming capabilities might be the most suitable option if you want to maintain an API Gateway architecture while enabling streaming responses.

Remember to thoroughly test any changes, as switching from REST to HTTP API Gateway may have other implications for your overall system architecture.
      
**Sources**  
[Gateway responses for REST APIs in API Gateway - Amazon API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-gatewayResponse-definition.html)  
[Methods for REST APIs in API Gateway - Amazon API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-method-settings.html)  
[Receive an API response with a compressed payload in API Gateway - Amazon API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-receive-response-with-compressed-payload.html)  
[Writing response streaming-enabled Lambda functions - AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/config-rs-write-functions.html)

Didier_Durand · Answer

Hi,

You will get all the details about streaming LLM answers through API gateway with WebSocket in this very complete blog post: https://amlanscloud.com/llmstreampost/

Please, follow its guidance to get your use case to work.

Associated repo with code is here: https://github.com/amlana21/llm-stream-publish

If you want to implement your resources via CloudFormation (I always do), this one will help: https://serverlessland.com/patterns/apigw-websocket-api-bedrock-streaming

Best,

Didier

Streaming response from api gateway

Relevant content