2 réponses
- Le plus récent
- Le plus de votes
- La plupart des commentaires
1
Hello,
If you are using a Jupyter notebook with your Glue docker image, then you can use Spark magic command %%configure to set the Spark's driver and executor memory/vcores depending on your system/computer.
You can get a list of available spark magic commands by simply running %help in a cell
The command looks something like below
%%configure
{
"driverMemory":"2000M",
"executorMemory":"2000M"
}
You can get list of configurable Spark parameters from here
0
I figured out that I was not using the right Docker Image. Since I was on a Mac M1 , I had to use the Glue3.0 ARM image instead I was using the AMD image causing this problem . Closing this question.
répondu il y a un an
Contenus pertinents
- demandé il y a 6 mois
- demandé il y a un an
- demandé il y a 7 mois
- demandé il y a 3 mois
- AWS OFFICIELA mis à jour il y a 6 mois
- AWS OFFICIELA mis à jour il y a un an
- AWS OFFICIELA mis à jour il y a 2 ans
This did not work for me