How to write data from Glue ETL Streaming Job to Kinesis Data Stream?

0

Hello! According to the documentation, it should be possible to write data to Kinesis from Glue Streaming ETL Job using glueContext.write_dynamic_frame_from_options. On the other hand, this feature is very poorly documented. Here you won't find any mention about Kinesis. Moreover, I can't find any example of writing to Kinesis using Glue.

After multiple tries, I managed to send some data to Kinesis using this piece of code:

        kinesissink = glueContext.write_dynamic_frame.from_options(frame=apply_mapping, connection_type="kinesis",
                                                              connection_options={"streamARN": "arn:aws:kinesis:eu-central-1:ACCOUNT_ID:stream/outputstream"},
                                                              transformation_ctx="kinesissink")

The problem is that it works if and only if this stream is in eu-east-1! I tried to pass various parameters to connection_options, including: streamName, endpointUrl, region - nothing helped. In the logs of the job I can see:

2024-03-06T23:35:08.603+01:00	24/03/06 22:35:08 INFO LogInputStreamReader: [2024-03-06 22:35:08.597475] [0x000000ba][0x00007f21499727c0] [info] [kinesis_producer.cc:87] Using Region: us-east-1
2024-03-06T23:35:08.604+01:00	24/03/06 22:35:08 INFO LogInputStreamReader: [2024-03-06 22:35:08.597485] [0x000000ba][0x00007f21499727c0] [info] [kinesis_producer.cc:48] Using default Kinesis endpoint

It's Using Region: us-east-1 no matter what. I defined my job in eu-central-1.

I'd appreciate any help.

2 Answers
0

I think is a defect and the code is getting the property from the original config and not the case insensitive one.
Please try with "endpointurl" and confirm if it changes the region and I will open a ticket to correct the defect.

profile pictureAWS
EXPERT
answered 2 months ago
0

Hello! I tried this and with no effect - it didn't work. Still region is us-east-1...

answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions