AWS Glue on docker - java.lang.UnsatisfiedLinkError: no AWSGlueSimdNative in java.library.path

0

Hi, I'm running my AWS Glue pyspark job on docker locally based on this description - https://aws.amazon.com/blogs/big-data/develop-and-test-aws-glue-version-3-0-jobs-locally-using-a-docker-container. I'm getting such exception when I try to display data from glue dynamic frame: gdf.show(5)

23/02/06 11:20:09 WARN SimdCsvParser$: Could not find AWSGlueSimdNative lib from /home/glue_user/native_libs/amd64/hadoop-lzo-native:/home/glue_user/native_libs/amd64/hadoop-native/:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib java.lang.UnsatisfiedLinkError: no AWSGlueSimdNative in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860) at java.lang.Runtime.loadLibrary0(Runtime.java:843) at java.lang.System.loadLibrary(System.java:1136) at com.amazonaws.services.glue.readers.csv.SimdCsvParser$.loadNativeLib(SimdCsvParser.scala:366) at com.amazonaws.services.glue.readers.csv.SimdCsvParser$.isAllowed(SimdCsvParser.scala:361) at com.amazonaws.services.glue.readers.CSVReader.init(CSVReader.scala:258) at com.amazonaws.services.glue.hadoop.TapeHadoopColumnarReader.initialize(TapeHadoopRecordReader.scala:272)

Do you know if some native library is missing for image amazon/aws-glue-libs:glue_libs_4.0.0_image_01 ? Regards,

asked a year ago208 views
1 Answer
1

Hello @rePost-User-8465222,

The WARN message outlined in your query is occurring because the native binaries for vectorized SIMD CSV reader are not available in Local development environment (including docker images). As of now, optimizePerformance flag can be set only in AWS Glue ETL Job system.

We are in the process of updating our documentation to outline this limitation. To suppress the WARN message, you can remove the optimizePerformance flag or set it to False.

AWS
SUPPORT ENGINEER
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions