Glue Relationalize recursion / depth

0

Does relationalize only go one layer in depth? I have json objects that have 2 to 3 layers of struct arrays and I get this error on anything past the first array layer : An error occurred while calling o136.pyWriteDynamicFrame. Bad dbtable value expecting either schema.table or table

what am I missing here? code example (works as expected with objects that have 1 layer of array depth)

PostgreSQLtable_node3 = glueContext.create_dynamic_frame.from_catalog(database="tekmetric",table_name="repair_orders")
resolvechoice2 = ResolveChoice.apply(frame=PostgreSQLtable_node3, choice="make_struct", transformation_ctx="resolvechoice2")
dropnullfields3 = DropNullFields.apply(frame=resolvechoice2, transformation_ctx="datasource0").drop_fields("jobs.parts.dotNumbers")
dfc = Relationalize.apply(frame=dropnullfields3, staging_path="s3://tekmetric-temp/repair-orders/", name="repair_orders", transformation_ctx="relationalize4")

for df_name in dfc.keys():
    m_df = dfc.select(df_name)
    if m_df.count():
        glueContext.write_dynamic_frame.from_jdbc_conf(frame = m_df, catalog_connection = "postgres", connection_options = {"dbtable": df_name, "database": "tekmetric"})
gefragt vor 2 Jahren731 Aufrufe
1 Antwort
1

Fixed this myself - it wasn't an issue with Relationalize per se ... writing to Postgres, the write_dynamic_frame did not enjoy periods in the table names. Fix :

for df_name in dfc.keys():
    m_df = dfc.select(df_name)
    final = df_name.replace('.val.','_')
    if m_df.count():
        glueContext.write_dynamic_frame.from_jdbc_conf(frame = m_df, catalog_connection = "postgres", connection_options = {"dbtable": final, "database": "tekmetric"})
beantwortet vor 2 Jahren

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen