EMR: Issue in Hudi using clustering and compaction inline

0

I am facing issues using inline clustering and compaction in EMR, with the following error.. EMR : 6.13.0 Hudi: 0.13.1 com.esotericsoftware.kryo.KryoException: Unable to find class: org.apache.hudi.common.util.collection.ImmutablePair at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:160) ~[kryo-shaded-4.0.2.jar:?] at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133) ~[kryo-shaded-4.0.2.jar:?] at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:693) ~[kryo-shaded-4.0.2.jar:?] at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:388) ~[kryo-shaded-4.0.2.jar:?] at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:302) ~[kryo-shaded-4.0.2.jar:?] at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813) ~[kryo-shaded-4.0.2.jar:?] at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:312) ~[spark-core_2.12-3.4.1-amzn-0.jar:3.4.1-amzn-0] at org.apache.spark.serializer.SerializerHelper$.deserializeFromChunkedBuffer(SerializerHelper.scala:52) ~[spark-core_2.12-3.4.1-amzn-0.jar:3.4.1-amzn-0] at org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:108) ~[spark-core_2.12-3.4.1-amzn-0.jar:3.4.1-amzn-0] at org.apache.spark.scheduler.TaskResultGetter$$anon$3.$anonfun$run$1(TaskResultGetter.scala:75) ~[spark-core_2.12-3.4.1-amzn-0.jar:3.4.1-amzn-0] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?] at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:2122) ~[spark-core_2.12-3.4.1-amzn-0.jar:3.4.1-amzn-0] at org.apache.spark.scheduler.TaskResultGetter$$anon$3.run(TaskResultGetter.scala:63) ~[spark-core_2.12-3.4.1-amzn-0.jar:3.4.1-amzn-0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] Caused by: java.lang.ClassNotFoundException: org.apache.hudi.common.util.collection.ImmutablePair at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) ~[?:?] at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[?:?] at java.lang.ClassLoader.loadClass(ClassLoader.java:525) ~[?:?] at java.lang.Class.forName0(Native Method) ~[?:?] at java.lang.Class.forName(Class.java:467) ~[?:?] at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154) ~[kryo-shaded-4.0.2.jar:?] ... 15 more Is it something anyone faced before ? Added spark-bundle and hudi-utlities jar as --jars argument when submitting Spark job.

subash
asked 6 months ago96 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions