Glue Relationalize recursion / depth

0

Does relationalize only go one layer in depth? I have json objects that have 2 to 3 layers of struct arrays and I get this error on anything past the first array layer : An error occurred while calling o136.pyWriteDynamicFrame. Bad dbtable value expecting either schema.table or table

what am I missing here? code example (works as expected with objects that have 1 layer of array depth)

PostgreSQLtable_node3 = glueContext.create_dynamic_frame.from_catalog(database="tekmetric",table_name="repair_orders")
resolvechoice2 = ResolveChoice.apply(frame=PostgreSQLtable_node3, choice="make_struct", transformation_ctx="resolvechoice2")
dropnullfields3 = DropNullFields.apply(frame=resolvechoice2, transformation_ctx="datasource0").drop_fields("jobs.parts.dotNumbers")
dfc = Relationalize.apply(frame=dropnullfields3, staging_path="s3://tekmetric-temp/repair-orders/", name="repair_orders", transformation_ctx="relationalize4")

for df_name in dfc.keys():
    m_df = dfc.select(df_name)
    if m_df.count():
        glueContext.write_dynamic_frame.from_jdbc_conf(frame = m_df, catalog_connection = "postgres", connection_options = {"dbtable": df_name, "database": "tekmetric"})
已提问 2 年前731 查看次数
1 回答
1

Fixed this myself - it wasn't an issue with Relationalize per se ... writing to Postgres, the write_dynamic_frame did not enjoy periods in the table names. Fix :

for df_name in dfc.keys():
    m_df = dfc.select(df_name)
    final = df_name.replace('.val.','_')
    if m_df.count():
        glueContext.write_dynamic_frame.from_jdbc_conf(frame = m_df, catalog_connection = "postgres", connection_options = {"dbtable": final, "database": "tekmetric"})
已回答 2 年前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则