Resolving EMR Spark Timeout with Lake Formation Runtime Roles When S3 Location Uses a Service-Linked Role

Amazon EMR supports runtime roles for EMR steps, which enable fine-grained access control (FGAC) through AWS Lake Formation. This allows you to assign different IAM roles to individual Spark steps, and Lake Formation governs which tables and columns each role can access.

A common setup issue can cause all Spark steps to fail with a 300-second timeout even though the EMR cluster appears healthy. This article explains the root cause and provides the fix.

The Problem After enabling Lake Formation runtime roles on your EMR cluster, all Spark steps fail with this error:

ERROR UserDriverServiceImpl: Did not receive system driver address within timeout of 300000ms java.util.concurrent.TimeoutException: null at org.apache.spark.fgac.server.UserDriverServiceImpl.getSystemDriverAddress(UserDriverServiceImpl.java:32) at org.apache.spark.fgac.client.SparkContextClientImpl.createChannel(SparkContextClientImpl.java:377) at org.apache.spark.sql.SecuredClientHiveSessionStateBuilder.init(SecuredClientHiveSessionStateBuilder.scala:29) The step retries three times (5 minutes each), then fails after approximately 16 minutes total. You may also see:

ERROR reading Table db.table: Error while instantiating org.apache.spark.sql.SecuredClientHiveSessionStateBuilder Caused by: java.lang.RuntimeException: Did not receive system driver address within timeout of 300000ms Key observations:

The EMR Record Server starts successfully (all 5 FGAC services running) Kerberos ticket renewals succeed Non-Lake Formation steps on the same cluster work fine CloudTrail shows zero lakeformation:GetDataAccess calls, meaning the request never reaches Lake Formation Why This Happens The root cause is how your S3 data location is registered in Lake Formation.

When you register an S3 location in the Lake Formation console, it defaults to using the AWSServiceRoleForLakeFormationDataAccess, which is a Service-Linked Role (SLR). This works for many Lake Formation use cases, but EMR on EC2 cannot obtain temporary credentials from Service-Linked Roles via the GetDataAccess API.

The failure is silent. EMR's Record Server starts normally, but the FGAC system driver cannot establish a connection to vend credentials. The Spark session initialization hangs until the 300-second timeout expires.

How to Verify Run this command to check how your S3 locations are registered:

aws lakeformation list-resources --region <your-region> Examine the RoleArn field for each resource. If it contains AWSServiceRoleForLakeFormationDataAccess, that's the issue:

{ "ResourceInfoList": [ { "ResourceArn": "arn:aws:s3:::my-data-bucket", "RoleArn": "arn:aws:iam::123456789012:role/aws-service-role/lakeformation.amazonaws.com/AWSServiceRoleForLakeFormationDataAccess" } ] } You can also verify in the Lake Formation console by navigating to Data lake locations and checking the IAM role column.

To confirm the pattern matches:

Check Expected if this is the issue Step stderr 3× "Did not receive system driver address within timeout of 300000ms" CloudTrail (Lake Formation events) Zero GetDataAccess calls during the failure window Runtime role setup in step controller logs User creation and role mapping succeed Non-Lake Formation steps on same cluster Succeed normally Solution Step 1: Deregister the S3 location In the console, go to Lake Formation → Data lake locations → select the location → Deregister

Or via CLI:

aws lakeformation deregister-resource
--resource-arn arn:aws:s3:::my-data-bucket See: Deregistering a data lake location

Step 2: Create a user-defined IAM role for Lake Formation Create a new IAM role that Lake Formation can assume to access your S3 data. This replaces the default SLR.

See: Requirements for the role used to register a location

Step 3: Re-register the S3 location with the user-defined role aws lakeformation register-resource
--resource-arn arn:aws:s3:::my-data-bucket
--role-arn arn:aws:iam::123456789012:role/MyLakeFormationDataAccessRole See: Registering a data lake location

Step 4: Re-run your Spark step No cluster restart is needed. The next step submission will use the new registration.

Additional Consideration: Bootstrap Action Timing If you resolve the SLR issue but the timeout returns after adding bootstrap actions, you may be hitting a separate race condition.

Long-running bootstrap scripts (for example, compiling Python from source with make install) can delay HDFS DataNode startup. The EMR SecretAgent attempts to upload the UserKeystore_spark_system.jks file to HDFS, but exhausts its 30 retries before DataNodes become available. Without this keystore, the FGAC system driver cannot establish secure connections.

Check the SecretAgent logs for:

File /user/sparksystem/sslKeys/UserKeystore_spark_system.jks could only be written to 0 of the 1 minReplication nodes. There are 0 datanodes running and 0 nodes are excluded in this operation. The fix is to replace source-compilation bootstrap scripts with pre-compiled binaries downloaded from S3. The aws-samples/aws-emr-utilities repository provides a ready-to-use approach: pre-compile Python once, archive it to S3, and use a bootstrap action that simply downloads and extracts it (~10 seconds vs. minutes for make install).

References AWS Lake Formation Best Practices Introducing runtime roles for Amazon EMR steps Enable Lake Formation with Amazon EMR Lake Formation fine-grained access control Requirements for the role used to register a location