2 Answers
- Newest
- Most votes
- Most comments
0
Here is the documentation on how to view your AWS Glue Job Logs: https://docs.aws.amazon.com/glue/latest/dg/monitor-continuous-logging-view.html
answered a year ago
0
The problem ended up being permissions problems This video helped https://www.youtube.com/watch?v=UUoQAe_NzaA&list=PL7bE4nSzLSWfYAc3q1vEYFi145Mt_DLcF&index=2
Changing my S3 source to one that started with "aws-glue-" solved it. When the AWSGlueService role is added it is defaulted to buckets that start with aws-glue
Why it did not trigger an actual error and said the job succeeded, I don't know
answered a year ago
Relevant content
- Accepted Answerasked a year ago
- asked 2 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a year ago
Have you verified if the S3 endpoint to the file is correct?
Have you done a ApplyMapping_node2.show() to make sure it has data loaded and the mapping was correctly applied?
vtjean Yes the endpoint is correct, I verified
Gonzalo Herreros ApplyMapping_node2.show() does not display anything, does that mean it is not mapping? I put the statement after the 23/05/22 16:01:18 WARN main: End - .show() below
23/05/22 16:01:23 INFO DAGScheduler: Job 0 finished: save at JDBCUtils.scala:897, took 0.020324 s 23/05/22 16:01:19 INFO GlueContext: The DataSink in action for the given format/connectionType (sqlserver) is com.amazonaws.services.glue.sinks.SQLServerDataSink 23/05/22 16:01:19 INFO GlueContext: Glue secret manager integration: secretId is not provided. 23/05/22 16:01:19 INFO GlueContext: Using location: h3-glue-db.dbo.h3_load 23/05/22 16:01:19 INFO GlueContext: getCatalogSink: catalogId: null, nameSpace: h3-sql-database, tableName: h3_glue_db_dbo_h3_load, isRegisteredWithLF: false 23/05/22 16:01:18 WARN main: End - .show() 23/05/22 16:01:17 WARN main: Start - .show() 23/05/22 16:01:16 INFO GlueContext: The DataSource in action : com.amazonaws.services.glue.HadoopDataSource 23/05/22 16:01:15 INFO GlueContext: Glue secret manager integration: secretId is not provided. 23/05/22 16:01:14 INFO GlueContext: GlueMetrics configured and enabled 23/05/22 16:01:12 INFO Utils: Successfully started service 'sparkDriver' on port 44775.