Error in connecting between Data Firehose and Glue ETL. Data format conversion error

Hi Prasanth,

Please go through the below steps and documentation link it will help to resolve your issue.

Step 1: Create or Identify an IAM Role

Create a new IAM role (or identify an existing one) that the Firehose service will assume to interact with AWS Glue.

Step 2: Attach Policies to the IAM Role

Attach the necessary policies to the IAM role to allow Kinesis Firehose to interact with AWS Glue and S3.

Example Policy for Firehose to Assume the Role:

Attach this policy to the IAM role to allow Kinesis Firehose to assume it:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "firehose.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Example Policy for Required Permissions:

Attach this policy to the IAM role to grant the necessary permissions for S3 and Glue:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:PutObjectAcl",
        "glue:GetTable",
        "glue:GetTableVersion",
        "glue:GetTableVersions"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name/*",
        "arn:aws:glue:region:account-id:catalog",
        "arn:aws:glue:region:account-id:database/your-database-name",
        "arn:aws:glue:region:account-id:table/your-table-name"
      ]
    }
  ]
}

Step 3: Configure Kinesis Firehose to Use the IAM Role

Configure your Kinesis Data Firehose delivery stream to use the IAM role for data format conversion:

• In the AWS Management Console, go to the Kinesis Data Firehose delivery stream.

• Under "Data transformation" settings, specify the IAM role you configured above.

Step 4: Test the Configuration Test the delivery stream to ensure that data is successfully converted to Parquet format and stored in the S3 bucket.

https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html

https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html