An error has been thrown from the AWS Athena client. SYNTAX_ERROR: line 2:43: '2023-02-07T23:59:59.613000+00:00' is not a valid timestamp literal

0

I can't find a proper way of setting the correct data type for a timestamp attribute on my Athena table parquet in order to query for time intervals. im creating the table via a crawler on parquet files resultant from a glue interactive job (PySpark)

**below my Athena table and the timestamp formats im trying **

  1. ts (datatype timestamp) : 2023-02-07 23:59:59.460000
  2. ts_iso_stamp (datatype timestamp) : 2023-02-07 23:59:59.460000
  3. ts_iso_str (datatype timestamp) : "2023-02-07T23:59:59.460000+00:00"

im applying this mapping in my glue job to create the various formats

ApplyMapping.apply(frame = dyf, mappings = [    ("date", "date", "date", "date")
                                                              ,("ts", "string", "ts", "timestamp")
                                                              ,("ts_iso", "string", "ts_iso_stamp", "timestamp")
                                                              ,("ts_iso", "string", "ts_iso_str", "string")  ]

ts_iso looks like this when you run .show() on the Dynamic Dataframe. a valid iso format as far as I understand.

"ts_iso": "2023-02-07T23:59:59.460000+00:00"

Enter image description here

query examples failing same error

select * from searchdb.queries_lu where appid = 'aHs7sPcX66ytv2FqmRnv'
                            AND ts >= TIMESTAMP '2023-02-07 23:59:59.405000'
                            AND ts <= TIMESTAMP '2023-02-08 00:00:00.637000';


select * from searchdb.queries_lu where appid = 'aHs7sPcX66ytv2FqmRnv'
                            AND ts_iso_stamp >= TIMESTAMP '2023-02-07 23:59:59.405000'
                            AND ts_iso_stamp <= TIMESTAMP '2023-02-08 00:00:00.637000';


select * from searchdb.queries_lu where appid = 'aHs7sPcX66ytv2FqmRnv'
                            AND ts_iso_str >= TIMESTAMP '2023-02-07T23:59:59.405000+00:00'
                            AND ts_iso_str <= TIMESTAMP '2023-02-08T00:00:00.637000+00:00';

Athena DDL

create table searchdb.queries_ac ( ts timestamp(23, 3) , ts_iso_stamp timestamp(23,3), ts_iso_str string(255), endpoint string(255), appid string(255), requestid string(255), "in" string(255), q string(255), qq string(255), results_id string(255), results_count string(255), requestline string(255), "date" string(255) );

已提问 1 年前462 查看次数
1 回答
1
已接受的回答

In general is better to avoid storing strings.
In your example there is you use millis instead of microsecond precision it should work (also don't use the T between date and time), e.g.:

ts >= TIMESTAMP '2023-02-07 23:59:59.405'
profile pictureAWS
专家
已回答 1 年前
  • Thanks Gonzalo , that was the missing part to dominate the Timestamp challenge . Regards

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则

相关内容