Questions tagged with Amazon EMR
Amazon EMR is a cloud big data platform for running large-scale distributed data processing jobs, interactive SQL queries, and machine learning (ML) applications using open-source analytics frameworks such as Apache Spark, Apache Hive, and Presto.
Content language: English
Filter questions
Select tags to filter
Sort by
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
347 results
As cloud and DevOps engineers, you'll eventually encounter big data workloads that need scalable processing. This is where Amazon EMR (Elastic MapReduce) comes in. If you're already using Infrastructu...
1
answers
0
votes
104
views
asked 2 months ago
Hi,
I'm working on a Spark Scala App that reads Delta Lake (with CDF) tables directly from S3 (not catalog registred) and writes a parquet table. The productive process is being executed on EMR 6.11 ...
3
answers
0
votes
137
views
asked 2 months ago
**Service and environment**
* Product: Amazon EMR
* Release label: emr-7.13.0
* Applications: Spark, Livy (and others as applicable)
* Region: [e.g. us-east-1]
* Workload: PySpark / Livy submitting Py...
2
answers
2
votes
404
views
asked 6 months ago
Hello,
I am running an Apache Spark job on Amazon EMR that needs to connect to an Amazon MSK cluster configured with IAM authentication. The EMR cluster has an IAM role with full MSK permissions, and...
1
answers
0
votes
431
views
asked a year ago
Hi Team,
I'm trying to set up the Amazon Q on the EMR studio notebook workspace and followed this guide: https://docs.aws.amazon.com/amazonq/latest/qdeveloper-ug/emr-setup.html?trk=769a1a2b-8c19-4976...
1
answers
0
votes
197
views
asked a year ago
Getting this issue in Amazon EMR during a pyspark job execution.
```
df = spark.read.parquet("s3a://test/raw-billing-cor-data/cur2/123456789/cid-cur2/data/BILLING_PERIOD=2025-08/")
py4j.protocol.Py4...
1
answers
0
votes
299
views
asked a year ago
Hi.
I am trying to configure ZGC in HBase following the recommendations, but the JAVA_HOME and HBASE_REGIONSERVER_GC_OPTS variables are not modified in the /etc/hbase/conf/hbase-env.sh file. Has anyo...
1
answers
0
votes
207
views
asked a year ago
I am deploying an EMR HBase cluster with EMR WAL enabled using Terraform.
The cluster is created successfully and the WALs are visible using the emrwal CLI.
When I change some configuration of my clus...
1
answers
0
votes
210
views
asked a year ago
Hi everyone,
I currently have an EMR cluster (emr-6.9.0) running a real-time ingestion process. To save disk space, I’ve been using the **Cloud Shuffle Storage Plugin** for Apache Spark.
Now, I need ...
1
answers
0
votes
447
views
asked a year ago
Hi ,
Recently i started facing issues with EMR (EC2 is out of capacity), mentioning that "EC2 is out of capacity for m6a.12xlarge in availability zone us-east-1c"
I tried different machines in same ...
1
answers
0
votes
395
views
asked a year ago
Hi Team,
I'm Getting the below error while writing the table in postgres using glue spark script
can you please help me on this issue.
"Table or view "asset_aircraft_rpt" already exists. SaveMode: ...
1
answers
0
votes
213
views
asked a year ago
Hello,
I'm facing a pretty annoying error. Whenever I try to execute a UDF function on a EMR Notebook I get the following error:
```
py4j.protocol.Py4JJavaError: An error occurred while calling o157...
2
answers
0
votes
272
views
asked a year ago