1 Respuesta
- Más nuevo
- Más votos
- Más comentarios
1
I believe I got this figured out. I was missing a step, where I had to add additionalOptions to my sink. As explained here: https://medium.com/analytics-vidhya/add-new-partitions-in-aws-glue-data-catalog-from-aws-glue-job-79b0442b17af
# parameter "enableUpdateCatalog" tells the aws glue job to update #the glue data catalog as the new partitions are created
additionalOptions = {"enableUpdateCatalog": True}
# define the partition keys
additionalOptions["partitionKeys"] = ["year", "month","day"]
After doing this, I saw that the partitions (nested directories) were being created. Now I am having some other isuses with querying, but I will post that as a separate question on here.
respondido hace 2 años
Contenido relevante
- OFICIAL DE AWSActualizada hace un año
- OFICIAL DE AWSActualizada hace 2 años
- OFICIAL DE AWSActualizada hace 3 años