使用 AWS re:Post 即表示您同意 AWS re:Post 使用條款
AWS re:Postre:Post

Glue secret manager integration: secretId is not provided

0

I have an existing AWS Glue script that has been successfully running in Glue 2.4 for some time. I went in today to upgrade it to Glue 3.0 and am unable to connect to my database. I am simply reading in data from an Oracle database using glueContext.create_dynamic_frame.from_catalog and then writing it to a postgresql database.

This is the line that it is failing on

datasource0 = glueContext.create_dynamic_frame.from_catalog(database = databaseName, table_name = tableName, transformation_ctx = f"dropfields{tableName}")

I have a connection defined and am able to successfully connect to the database. But I am getting the following error: Glue secret manager integration: secretId is not provided when I run the job.

Here is the full stack

24/01/31 20:57:39 ERROR GlueExceptionAnalysisListener: [Glue Exception Analysis] Last Executed Line number from script longview.py: 38
24/01/31 20:57:38 ERROR GlueExceptionAnalysisListener: [Glue Exception Analysis] {"Event":"GlueETLJobExceptionEvent","Timestamp":1706734658975,"Failure Reason":"Traceback (most recent call last):\n  File \"/tmp/longview.py\", line 38, in <module>\n    datasource0 = glueContext.create_dynamic_frame.from_catalog(database = databaseName, table_name = tableName, transformation_ctx = f\"dropfields{tableName}\")\n  File \"/opt/amazon/lib/python3.6/site-packages/awsglue/dynamicframe.py\", line 787, in from_catalog\n    return self._glue_context.create_dynamic_frame_from_catalog(db, table_name, redshift_tmp_dir, transformation_ctx, push_down_predicate, additional_options, catalog_id, **kwargs)\n  File \"/opt/amazon/lib/python3.6/site-packages/awsglue/context.py\", line 188, in create_dynamic_frame_from_catalog\n    return source.getFrame(**kwargs)\n  File \"/opt/amazon/lib/python3.6/site-packages/awsglue/data_source.py\", line 36, in getFrame\n    jframe = self._jsource.getDynamicFrame()\n  File \"/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py\", line 1305, in __call__\n    answer, self.gateway_client, self.target_id, self.name)\n  File \"/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py\", line 111, in deco\n    return f(*a, **kw)\n  File \"/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py\", line 328, in get_return_value\n    format(target_id, \".\", name), value)\npy4j.protocol.Py4JJavaError: An error occurred while calling o93.getDynamicFrame.\n: java.sql.SQLRecoverableException: Listener refused the connection with the following error:\nORA-12514, TNS:listener does not currently know of service requested in connect descriptor\n  (CONNECTION_ID=bB1y1v6uSpCWCObiZ7c4Yg==)\n\tat oracle.jdbc.driver.T4CConnection.handleLogonNetException(T4CConnection.java:870)\n\tat oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:675)\n\tat oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:1032)\n\tat oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:90)\n\tat oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:681)\n\tat oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:602)\n\tat com.amazonaws.services.glue.util.JDBCWrapper$.$anonfun$connectionProperties$5(JDBCUtils.scala:992)\n\tat com.amazonaws.services.glue.util.JDBCWrapper$.$anonfun$connectWithSSLAttempt$2(JDBCUtils.scala:944)\n\tat scala.Option.getOrElse(Option.scala:121)\n\tat com.amazonaws.services.glue.util.JDBCWrapper$.$anonfun$connectWithSSLAttempt$1(JDBCUtils.scala:944)\n\tat scala.Option.getOrElse(Option.scala:121)\n\tat com.amazonaws.services.glue.util.JDBCWrapper$.connectWithSSLAttempt(JDBCUtils.scala:944)\n\tat com.amazonaws.services.glue.util.JDBCWrapper$.connectionProperties(JDBCUtils.scala:988)\n\tat com.amazonaws.services.glue.util.JDBCWrapper.connectionProperties$lzycompute(JDBCUtils.scala:752)\n\tat com.amazonaws.services.glue.util.JDBCWrapper.connectionProperties(JDBCUtils.scala:752)\n\tat com.amazonaws.services.glue.util.JDBCWrapper.tableDF(JDBCUtils.scala:878)\n\tat com.amazonaws.services.glue.util.NoCondition$.tableDF(JDBCUtils.scala:88)\n\tat com.amazonaws.services.glue.util.NoJDBCPartitioner$.tableDF(JDBCUtils.scala:174)\n\tat com.amazonaws.services.glue.JDBCDataSource.getDynamicFrame(DataSource.scala:1014)\n\tat com.amazonaws.services.glue.DataSource.getDynamicFrame(DataSource.scala:101)\n\tat com.amazonaws.services.glue.DataSource.getDynamicFrame$(DataSource.scala:101)\n\tat com.amazonaws.services.glue.AbstractSparkSQLDataSource.getDynamicFrame(DataSource.scala:725)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:282)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:238)\n\tat java.lang.Thread.run(Thread.java:750)\nCaused by: oracle.net.ns.NetException: Listener refused the connection with the following error:\nORA-12514, TNS:listener does not currently know of service requested in connect descriptor\n  (CONNECTION_ID=bB1y1v6uSpCWCObiZ7c4Yg==)\n\tat oracle.net.ns.NSProtocolNIO.createRefusePacketException(NSProtocolNIO.java:816)\n\tat oracle.net.ns.NSProtocolNIO.handleConnectPacketResponse(NSProtocolNIO.java:396)\n\tat oracle.net.ns.NSProtocolNIO.negotiateConnection(NSProtocolNIO.java:207)\n\tat oracle.net.ns.NSProtocol.connect(NSProtocol.java:350)\n\tat oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1967)\n\tat oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:640)\n\t... 31 more\n","Stack Trace":[{"Declaring Class":"get_return_value","Method Name":"format(target_id, \".\", name), value)","File Name":"/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py","Line Number":328},{"Declaring Class":"deco","Method Name":"return f(*a, **kw)","File Name":"/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py","Line Number":111},{"Declaring Class":"__call__","Method Name":"answer, self.gateway_client, self.target_id, self.name)","File Name":"/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py","Line Number":1305},{"Declaring Class":"getFrame","Method Name":"jframe = self._jsource.getDynamicFrame()","File Name":"/opt/amazon/lib/python3.6/site-packages/awsglue/data_source.py","Line Number":36},{"Declaring Class":"create_dynamic_frame_from_catalog","Method Name":"return source.getFrame(**kwargs)","File Name":"/opt/amazon/lib/python3.6/site-packages/awsglue/context.py","Line Number":188},{"Declaring Class":"from_catalog","Method Name":"return self._glue_context.create_dynamic_frame_from_catalog(db, table_name, redshift_tmp_dir, transformation_ctx, push_down_predicate, additional_options, catalog_id, **kwargs)","File Name":"/opt/amazon/lib/python3.6/site-packages/awsglue/dynamicframe.py","Line Number":787},{"Declaring Class":"<module>","Method Name":"datasource0 = glueContext.create_dynamic_frame.from_catalog(database = databaseName, table_name = tableName, transformation_ctx = f\"dropfields{tableName}\")","File Name":"/tmp/longview.py","Line Number":38}],"Last Executed Line number":38,"script":"longview.py"}
24/01/31 20:57:38 ERROR ProcessLauncher: Error from Python:Traceback (most recent call last):
  File "/tmp/longview.py", line 38, in <module>
    datasource0 = glueContext.create_dynamic_frame.from_catalog(database = databaseName, table_name = tableName, transformation_ctx = f"dropfields{tableName}")
  File "/opt/amazon/lib/python3.6/site-packages/awsglue/dynamicframe.py", line 787, in from_catalog
    return self._glue_context.create_dynamic_frame_from_catalog(db, table_name, redshift_tmp_dir, transformation_ctx, push_down_predicate, additional_options, catalog_id, **kwargs)
  File "/opt/amazon/lib/python3.6/site-packages/awsglue/context.py", line 188, in create_dynamic_frame_from_catalog
    return source.getFrame(**kwargs)
  File "/opt/amazon/lib/python3.6/site-packages/awsglue/data_source.py", line 36, in getFrame
    jframe = self._jsource.getDynamicFrame()
  File "/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, in deco
    return f(*a, **kw)
  File "/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 328, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o93.getDynamicFrame.
: java.sql.SQLRecoverableException: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor
  (CONNECTION_ID=bB1y1v6uSpCWCObiZ7c4Yg==)
	at oracle.jdbc.driver.T4CConnection.handleLogonNetException(T4CConnection.java:870)
	at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:675)
	at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:1032)
	at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:90)
	at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:681)
	at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:602)
	at com.amazonaws.services.glue.util.JDBCWrapper$.$anonfun$connectionProperties$5(JDBCUtils.scala:992)
	at com.amazonaws.services.glue.util.JDBCWrapper$.$anonfun$connectWithSSLAttempt$2(JDBCUtils.scala:944)
	at scala.Option.getOrElse(Option.scala:121)
	at com.amazonaws.services.glue.util.JDBCWrapper$.$anonfun$connectWithSSLAttempt$1(JDBCUtils.scala:944)
	at scala.Option.getOrElse(Option.scala:121)
	at com.amazonaws.services.glue.util.JDBCWrapper$.connectWithSSLAttempt(JDBCUtils.scala:944)
	at com.amazonaws.services.glue.util.JDBCWrapper$.connectionProperties(JDBCUtils.scala:988)
	at com.amazonaws.services.glue.util.JDBCWrapper.connectionProperties$lzycompute(JDBCUtils.scala:752)
	at com.amazonaws.services.glue.util.JDBCWrapper.connectionProperties(JDBCUtils.scala:752)
	at com.amazonaws.services.glue.util.JDBCWrapper.tableDF(JDBCUtils.scala:878)
	at com.amazonaws.services.glue.util.NoCondition$.tableDF(JDBCUtils.scala:88)
	at com.amazonaws.services.glue.util.NoJDBCPartitioner$.tableDF(JDBCUtils.scala:174)
	at com.amazonaws.services.glue.JDBCDataSource.getDynamicFrame(DataSource.scala:1014)
	at com.amazonaws.services.glue.DataSource.getDynamicFrame(DataSource.scala:101)
	at com.amazonaws.services.glue.DataSource.getDynamicFrame$(DataSource.scala:101)
	at com.amazonaws.services.glue.AbstractSparkSQLDataSource.getDynamicFrame(DataSource.scala:725)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:750)
Caused by: oracle.net.ns.NetException: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor
  (CONNECTION_ID=bB1y1v6uSpCWCObiZ7c4Yg==)
	at oracle.net.ns.NSProtocolNIO.createRefusePacketException(NSProtocolNIO.java:816)
	at oracle.net.ns.NSProtocolNIO.handleConnectPacketResponse(NSProtocolNIO.java:396)
	at oracle.net.ns.NSProtocolNIO.negotiateConnection(NSProtocolNIO.java:207)
	at oracle.net.ns.NSProtocol.connect(NSProtocol.java:350)
	at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1967)
	at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:640)
	... 31 more

24/01/31 20:57:37 INFO GlueContext: The DataSource in action : com.amazonaws.services.glue.JDBCDataSource
24/01/31 20:57:37 INFO GlueContext: Glue secret manager integration: secretId is not provided.
24/01/31 20:57:37 INFO GlueContext: nameSpace: longview-prod, tableName: dwpbrdmp_longview_budget_data, connectionName longview-prod, vendor: oracle
24/01/31 20:57:37 INFO GlueContext: getCatalogSource: transactionId: <not-specified> asOfTime: <not-specified> catalogPartitionIndexPredicate: <not-specified> 
24/01/31 20:57:37 INFO GlueContext: getCatalogSource: catalogId: null, nameSpace: longview-prod, tableName: dwpbrdmp_longview_budget_data, isRegisteredWithLF: false, isGoverned: false, isRowFilterEnabled: false, useAdvancedFiltering: false, isTableFromSchemaRegistry: false
24/01/31 20:57:35 INFO GlueContext: GlueMetrics configured and enabled
24/01/31 20:57:32 INFO Utils: Successfully started service 'sparkDriver' on port 34409.

I was hoping not to have to change to using AWS Secrets. I just want to figure out why this is not working all of a sudden. I have tried providing additional_options = {"useConnectionProperties": "true","connectionName": "name-of-connection"}, but that didn't work. I even went back to Glue 2.0, but got the same result.

Any help would be greatly appreciated.

主題
分析開發人員工具資料庫
標籤
分析開發人員工具資料庫AWS Glue提取轉換與載入資料
語言
English
CUCVM
已提問 3 個月前檢視次數 437 次
1 個回答
1

You don't have to use secret manager, that's just an info message.
The real issue there is that the service specified in the JDBC url doesn't seem to be recognized, if you didn't change anything else I wonder if the newer thin driver provided on Glue 3 maybe cannot handle it (never heard of such backwards compatibility breach). What about Glue 4?

profile pictureAWS
專家
Gonzalo Herreros
已回答 3 個月前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋，同時有助於提問者的專業成長。

回答問題指南

相關內容