Custom classifier for AWS Glue crawler

1

I have a set of files in my S3 bucket which have a delimiter ASCII 31 (unit separator). I am using a crawler to read these files and create the tables in AWS Glue catalog. I tried using the custom delimiter in the classifiers but with no luck since this is a non-printable character. What is the best way to incorporate this delimiter within a crawler?

2 Answers
1

To incorporate the ASCII 31 delimiter within a Glue Crawler, follow the steps below:

  1. Create a Custom Classifier - Because ASCII 31 is non-printable, you'll need to use it's escape sequence. Under the classifier's "Delimiter" field, enter "\u001F" representing the unit separator.

  2. Update your Crawler Configuration - In order to use the custom classifier created above, configure the Glue crawler's "CSV Classifier" settings by selecting the ASCII 31 custom classifier.

  3. Modify Glue Job (Depending on Job Code) - If your job code involves delimiter handling logic, make sure it is updated to account for the updated "\u001F" delimiter.

Below are links to the official AWS documentation on writing custom classifiers and adding them to a Glue crawler: https://docs.aws.amazon.com/glue/latest/dg/custom-classifier.html https://docs.aws.amazon.com/glue/latest/dg/add-classifier.html

If you have any further questions or encounter further issues, feel free to reach out with more information!

answered 7 months ago
0

I have similar issue with crawler, so used spark code as below. It may help you

delimiter_char31 = chr(31)
df = spark.read.option("header","false").option('delimiter',delimiter_char31 ).csv("s3://abc/test.txt")
answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions