- Más nuevo
- Más votos
- Más comentarios
Hello,
I would say the most cost efficient,simple and fastest way to start Glue ETL development are:
- Use Glue Docker image
- Use Interactive session
References:
[2] https://docs.aws.amazon.com/glue/latest/dg/interactive-sessions.html
Yes , I agree, I am more inclined to use the docker image , but I faces quite a lot of challenges and still facing , in setting it up. For starter , the Jupyter notebook does not have nbextensions preinstalled ,and I could not get it installed either , asked a separate question on that . Then the SPARK is very slow .
Havent tried the interative sessions , will start with it soon .
Hi,
It always depend on the actual priorities you have to define the trade off between cost efficiency and developer experience.
The most cost efficient I would agree is using the docker container . If you prefer to use a traditional IDE environment instead of a notebook you should be able to use it for that as well. Performance will depend on the local machine.
If you give more weight to a flexible developer experience using notebooks and performance when running the jobs or cells then Interactive sessions would be a better choice. You can install your notebook with all extensions and tune the number of DPUs you want to use changing that between sessions.
If the developer is fine with a managed notebook, Glue Studio Notebooks would have the same cost of Interactive sessions (it just depends on the DPU you select for the session and duration of the session), same configuration based on session . Glue Studio Notebooks are only available in some regions currently.
hope this helps
Is there a way to increase DPU or Spark config on the Docker ?
Contenido relevante
OFICIAL DE AWSActualizada hace 2 años- OFICIAL DE AWSActualizada hace 3 meses
I started using VSCode , which was much simpler to setup and work on . Start the Docker image for Glue Open VS Code Attach the container It will automatically install Python and other libraries . Good to go .