How to connect Kinesis Data Streams with EMR in EMR Studio?

I need to load data from Kinesis Data Streams to EMR via EMR Studio. I Follow this sample but doesn't work: https://github.com/awslabs/spark-sql-kinesis-connector

Topics

Tags

Language

English

asked 2 months ago835 views

1 Answer

Are these answers helpful? Upvote the correct answer to help the community benefit from your knowledge.

Hello! Could you share the error you are getting?

answered 2 months ago

Erika Nagamine
2 months ago
Follow the error: An error was encountered: [STREAM_FAILED] Query [id = 8e197d37-61c3-48de-b388-b5b7e4b0078d, runId = 7a4bf9bb-97af-495e-8f61-edd54ef7d8dc] terminated with exception: Unable to fetch committed metadata from previous batch id 0. Some data may have been missed Traceback (most recent call last): File "/mnt1/yarn/usercache/livy/appcache/application_1709526661976_0019/container_1709526661976_0019_01_000001/pyspark.zip/pyspark/sql/streaming/query.py", line 221, in awaitTermination return self._jsq.awaitTermination() File "/mnt1/yarn/usercache/livy/appcache/application_1709526661976_0019/container_1709526661976_0019_01_000001/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in call return_value = get_return_value( File "/mnt1/yarn/usercache/livy/appcache/application_1709526661976_0019/container_1709526661976_0019_01_000001/pyspark.zip/pyspark/errors/exceptions/captured.py", line 185, in deco raise converted from None pyspark.errors.exceptions.captured.StreamingQueryException: [STREAM_FAILED] Query [id = 8e197d37-61c3-48de-b388-b5b7e4b0078d, runId = 7a4bf9bb-97af-495e-8f61-edd54ef7d8dc] terminated with exception: Unable to fetch committed metadata from previous batch id 0. Some data may have been missed

also I follow this example: https://github.com/aws-samples/spark-kinesis-connector-emr-sample

Relevant content