Questions tagged with AWS Glue
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
I'm working on a project that makes use of Glue Record Matching transforms which, by my best research though AWS docs, is only supported in Glue 2.0 jobs (and additionally, the maximum Glue version I...
0
answers
0
votes
56
views
asked 4 months agolg...
I am trying to write a pyspark dataframe to S3 and the AWS data catalog using the Iceberg format and the pyspark.sql.DataFrameWriterV2 with the createOrReplace function. When I write the same...
1
answers
0
votes
639
views
asked 4 months agolg...
Hi. I am trying to run an AWS Glue job where I transfer data from S3 to Amazon Redshift. However, I am receiving the following error:
```
Error Category: UNCLASSIFIED_ERROR; An error occurred while...
2
answers
0
votes
1124
views
asked 4 months agolg...
I have a data pipeline built in Redshift Serverless, with some final tables being the result. We are also running a web app that I have set up an Aurora Serverless Postgres DB, to run from. The idea...
0
answers
0
votes
121
views
asked 4 months agolg...
Can someone please help with this error? I have a csv file in an S3 bucket, created a crawler to update a table in glue, and the crawler runs but when I try to view the data in athena I get this...
1
answers
0
votes
573
views
asked 4 months agolg...
Hi this question is regarding corrupt or malformed records in Glue ETL.
Spark DataFrames obviously have an option for indicated column for _corrupt_record when this happens and the entire record is...
1
answers
0
votes
210
views
asked 4 months agolg...
Hello, I would like to know if there is a way to query Iceberg tables (backed with S3 parquet files) cataloged within the AWS Glue Catalog using AWS Databrew. (maybe through Athena?).
Also, is it...
2
answers
0
votes
576
views
asked 4 months agolg...
Hi
Trying to craw connect logs create bad metadata with fields like this inside the table:
struct<connect\:Subtype:struct<ValueString:string>>
obvious running this struct inside athena result in a...
0
answers
0
votes
428
views
asked 4 months agolg...
Hi,
Have followed the below documentation to set up the Spark History server to see Spark UI Logs. Am able to run the container but not able to access the URL http://localhost:18080 .
docker run...
1
answers
0
votes
229
views
asked 5 months agolg...
We connected Timestream to Athena using the [Athena Timestream connector](https://docs.aws.amazon.com/athena/latest/ug/connectors-timestream.html). When running a federated query through Athena to...
2
answers
0
votes
778
views
asked 5 months agolg...
i got a mongodb atlas cluster outside aws. I want to use aws glue with my mongo db databases so i created a connection but im getting "InvalidInputException: Unable to resolve any valid connection". ...
1
answers
1
votes
268
views
asked 5 months agolg...
```
df = spark.read.parquet("s3://folder/")
df = df.withColumn('filename', input_file_name())
AmazonS3_node1697616892615 = DynamicFrame.fromDF(df, glueContext, "s3sparkread")
```
if this is the code...
1
answers
0
votes
365
views
asked 5 months agolg...