AWS GLUE to Open Search Index custom Index Id

Question

I have a glue job which pushes the data from glue into open search. 
The index Id column is automatically created while inserting the data into open search.
I would like to pass the index id  _id field into open search while pushing the data using glue. 
what parameter should I be using or how can I pass a custom specific index_id to an open search index?

Accepted Answer

Basically there two ways, I did this using dynamic mapping with dynamic documents

#### Define a Hidden Column
In your Glue Catalog table, add a hidden column `named _id`. This column will store the custom index ID you want to use for each record.

#### Access the Value in Glue Script
Within your Glue job script (Python or Scala), access the value of the `hidden _id` column for each record being processed.

#### Enable Dynamic Documents
When configuring the OpenSearch data sink in your Glue job, set the "Dynamic Documents" option to "true" within the connection properties. This allows Glue to dynamically add the _id field with your custom value during data insertion.

The other way is pre defined index mapping, let me know if you want to know about it.

Here are the references that might help you
1. https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-dynamic-frame-writer.html
2. https://docs.aws.amazon.com/glue/latest/dg/connecting-to-data-opensearch.html
3. https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

AWS GLUE to Open Search Index custom Index Id

Define a Hidden Column

Access the Value in Glue Script

Enable Dynamic Documents

相關內容