EMR: Issue in Hudi using clustering and compaction inline

0

I am facing issues using inline clustering and compaction in EMR, with the following error.. EMR : 6.13.0 Hudi: 0.13.1 com.esotericsoftware.kryo.KryoException: Unable to find class: org.apache.hudi.common.util.collection.ImmutablePair at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:160) ~[kryo-shaded-4.0.2.jar:?] at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133) ~[kryo-shaded-4.0.2.jar:?] at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:693) ~[kryo-shaded-4.0.2.jar:?] at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:388) ~[kryo-shaded-4.0.2.jar:?] at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:302) ~[kryo-shaded-4.0.2.jar:?] at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813) ~[kryo-shaded-4.0.2.jar:?] at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:312) ~[spark-core_2.12-3.4.1-amzn-0.jar:3.4.1-amzn-0] at org.apache.spark.serializer.SerializerHelper$.deserializeFromChunkedBuffer(SerializerHelper.scala:52) ~[spark-core_2.12-3.4.1-amzn-0.jar:3.4.1-amzn-0] at org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:108) ~[spark-core_2.12-3.4.1-amzn-0.jar:3.4.1-amzn-0] at org.apache.spark.scheduler.TaskResultGetter$$anon$3.$anonfun$run$1(TaskResultGetter.scala:75) ~[spark-core_2.12-3.4.1-amzn-0.jar:3.4.1-amzn-0] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?] at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:2122) ~[spark-core_2.12-3.4.1-amzn-0.jar:3.4.1-amzn-0] at org.apache.spark.scheduler.TaskResultGetter$$anon$3.run(TaskResultGetter.scala:63) ~[spark-core_2.12-3.4.1-amzn-0.jar:3.4.1-amzn-0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] Caused by: java.lang.ClassNotFoundException: org.apache.hudi.common.util.collection.ImmutablePair at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) ~[?:?] at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[?:?] at java.lang.ClassLoader.loadClass(ClassLoader.java:525) ~[?:?] at java.lang.Class.forName0(Native Method) ~[?:?] at java.lang.Class.forName(Class.java:467) ~[?:?] at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154) ~[kryo-shaded-4.0.2.jar:?] ... 15 more Is it something anyone faced before ? Added spark-bundle and hudi-utlities jar as --jars argument when submitting Spark job.

subash
gefragt vor 6 Monaten103 Aufrufe
Keine Antworten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen