1 Answer
- Newest
- Most votes
- Most comments
0
Cohere models support streaming, but you need to pass the "stream" : true
parameter in the JSON body to enable streaming, in addition to using the invoke_model_with_response_stream
API.
You can try the following code:
body = json.dumps({
"prompt": prompt,
"max_tokens": max_token_count,
"temperature": temperature,
"p": top_p,
"stop_sequences": stop_sequences,
"stream": True,
})
response = brt.invoke_model_with_response_stream(
modelId='cohere.command-text-v14',
body=body,
accept=accept,
contentType=contentType
)
stream = response.get('body')
if stream:
for event in stream:
chunk = event.get('chunk')
if chunk:
print(json.loads(chunk.get('bytes').decode()))
Relevant content
- asked 2 months ago
- asked 3 months ago
- asked 19 days ago
- asked 5 months ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 months ago
- AWS OFFICIALUpdated 10 months ago
- AWS OFFICIALUpdated 2 months ago