How to set spark.driver.extraLibraryPath in EMR Serverless

0

Hello, i tried run JNI code which uses the file as a dependency libMylibName.so. Before(not in EMR Serverless) i can use it just specify --conf spark.driver.extraLibraryPath=/path_to_my_lib and --conf spark.executor.extraLibraryPath=/path_to_my_lib , but this options not supported in Serverless following here documentation i found similar answer for python, but it not suit for me post, because i use Scala

Error which i got if not specify spark.driver.extraLibraryPath and spark.executor.extraLibraryPath= Exception in thread "main" java.lang.UnsatisfiedLinkError: no MylibName in java.library.path

Question: how i can specify this dependency for my spark application?

FYI: --conf spark.driver.extraJavaOptions="-Djava.library.path=s3://path_to_libs" also not work correct it changed java.library.path Variable but i continue got UnsatisfiedLinkError Enter image description here

posta un anno fa1027 visualizzazioni
1 Risposta
1

I found the solution on my own and publish it here:

--conf spark.archives=s3://path_to_file/my_libs_so.tar.gz#my_libs --conf spark.executor.extraJavaOptions="-Djava.library.path=/home/hadoop/my_libs" --conf spark.driver.extraJavaOptions="-Djava.library.path=/home/hadoop/my_libs"

Archive my_libs_so.tar.gz should contains all your the necessary *.so files

con risposta un anno fa
AWS
ESPERTO
verificato un anno fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande