By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Pyflink on Managed Service for Apache Flink

0

Hello,

I am trying to get an Apache Flink job (Datastream API) running to ingest JSON strings from S3 (JSON lines files delivered every 5 minutes), which have been written to that source S3 by a Kinesis Firehose, group incoming records and store the result in a new S3 bucket in Parquet format.

So far I have been able to ingest the records, but I cannot get the windowing to work. A count window setup that I got working does not reliably write the correct number records and the number differs even with the same source data. I could not figure out the time windowing yet from the docs.

Managed Service for Apache Flink (formerly Kinesis Data Analytics) works with Flink 1.15, but there is a more recent version available, namely 1.17 which is much more developed and seems to be much more user friendly.

Is there any chance AWS will support version 1.17 in the near future? I don't see myself getting this to work with 1.15.

Thanks in advance, Pieter

Topics
StorageAnalyticsInternet of Things (IoT)
Tags
Amazon Simple Storage ServiceAmazon Kinesis Data AnalyticsAmazon Kinesis
Language
English
Pieter
asked 7 months ago1000 views
1 Answer
0

Greetings,

The backend team for Amazon Managed Service for Apache Flink (formerly known as Kinesis Data Analytics) is actively addressing this matter, and we anticipate that Managed Service for Apache Flink will support Flink 1.17 in the near future. Due to the managed nature of the service, features are required to be tested and pre-configured to align with the existing ecosystem because of which I unfortunately would not be able to provide an ETA for the same.

We recommend keeping an eye on our AWS forums for updates on newly released features and announcements:

What's New - https://aws.amazon.com/new/ Blogs - https://aws.amazon.com/blogs/aws/

Additionally, if you need assistance with the issue you are currently experiencing, I kindly request that you open a support case [1] with us, as we require details that are non-public information.

References: [1] https://docs.aws.amazon.com/awssupport/latest/user/case-management.html#creating-a-support-case

Thank you.

AWS
SUPPORT ENGINEER
AWS-User-2952566
answered 7 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content