2 réponses
- Le plus récent
- Le plus de votes
- La plupart des commentaires
0
Here is the documentation on how to view your AWS Glue Job Logs: https://docs.aws.amazon.com/glue/latest/dg/monitor-continuous-logging-view.html
répondu il y a un an
0
The problem ended up being permissions problems This video helped https://www.youtube.com/watch?v=UUoQAe_NzaA&list=PL7bE4nSzLSWfYAc3q1vEYFi145Mt_DLcF&index=2
Changing my S3 source to one that started with "aws-glue-" solved it. When the AWSGlueService role is added it is defaulted to buckets that start with aws-glue
Why it did not trigger an actual error and said the job succeeded, I don't know
répondu il y a un an
Contenus pertinents
- demandé il y a un an
- demandé il y a 9 mois
- demandé il y a un an
- demandé il y a 2 mois
- AWS OFFICIELA mis à jour il y a 2 ans
Have you verified if the S3 endpoint to the file is correct?
Have you done a ApplyMapping_node2.show() to make sure it has data loaded and the mapping was correctly applied?
vtjean Yes the endpoint is correct, I verified
Gonzalo Herreros ApplyMapping_node2.show() does not display anything, does that mean it is not mapping? I put the statement after the 23/05/22 16:01:18 WARN main: End - .show() below
23/05/22 16:01:23 INFO DAGScheduler: Job 0 finished: save at JDBCUtils.scala:897, took 0.020324 s 23/05/22 16:01:19 INFO GlueContext: The DataSink in action for the given format/connectionType (sqlserver) is com.amazonaws.services.glue.sinks.SQLServerDataSink 23/05/22 16:01:19 INFO GlueContext: Glue secret manager integration: secretId is not provided. 23/05/22 16:01:19 INFO GlueContext: Using location: h3-glue-db.dbo.h3_load 23/05/22 16:01:19 INFO GlueContext: getCatalogSink: catalogId: null, nameSpace: h3-sql-database, tableName: h3_glue_db_dbo_h3_load, isRegisteredWithLF: false 23/05/22 16:01:18 WARN main: End - .show() 23/05/22 16:01:17 WARN main: Start - .show() 23/05/22 16:01:16 INFO GlueContext: The DataSource in action : com.amazonaws.services.glue.HadoopDataSource 23/05/22 16:01:15 INFO GlueContext: Glue secret manager integration: secretId is not provided. 23/05/22 16:01:14 INFO GlueContext: GlueMetrics configured and enabled 23/05/22 16:01:12 INFO Utils: Successfully started service 'sparkDriver' on port 44775.