Pyflink on Managed Service for Apache Flink

Question

Hello,

I am trying to get an Apache Flink job (Datastream API) running to ingest JSON strings from S3 (JSON lines files delivered every 5 minutes), which have been written to that source S3 by a Kinesis Firehose, group incoming records and store the result in a new S3 bucket in Parquet format.

So far I have been able to ingest the records, but I cannot get the windowing to work. A count window setup that I got working does not reliably write the correct number records and the number differs even with the same source data. I could not figure out the time windowing yet from the docs.

Managed Service for Apache Flink (formerly Kinesis Data Analytics) works with Flink 1.15, but there is a more recent version available, namely 1.17 which is much more developed and seems to be much more user friendly.

Is there any chance AWS will support version 1.17 in the near future? I don't see myself getting this to work with 1.15.

Thanks in advance,
Pieter

Answer

Greetings,

The backend team for Amazon Managed Service for Apache Flink (formerly known as Kinesis Data Analytics) is actively addressing this matter, and we anticipate that Managed Service for Apache Flink will support Flink 1.17 in the near future. Due to the managed nature of the service, features are required to be tested and pre-configured to align with the existing ecosystem because of which I unfortunately would not be able to provide an ETA for the same.

We recommend keeping an eye on our AWS forums for updates on newly released features and announcements:

What's New - https://aws.amazon.com/new/
Blogs      - https://aws.amazon.com/blogs/aws/

Additionally, if you need assistance with the issue you are currently experiencing, I kindly request that you open a support case [1] with us, as we require details that are non-public information.

References:
[1] https://docs.aws.amazon.com/awssupport/latest/user/case-management.html#creating-a-support-case

Thank you.

Pyflink on Managed Service for Apache Flink

관련 콘텐츠