Why do you notice ORA-7445 or ORA-1001 errors around the maintenance window on my Amazon RDS for Oracle DB instance?

Short description

Amazon RDS for Oracle can write an ORA-7445 (core dump) message to the alert log when an Oracle server process ends unexpectedly while it closes or releases a cursor. Separately, an ORA-1001 (invalid cursor) message can appear when a session references a cursor that is no longer valid. When the Database Insights (Performance Insights) collection agent that Amazon RDS runs internally as rdsoracleperfmon owns the failing session, the error is typically benign and doesn't affect your database.

To identify the session that Oracle recorded and confirm whether you need to act, complete the steps in the Resolution section.

Resolution

Note the error and confirm that Performance Insights is turned on

1. Open the Amazon RDS console, choose Databases, and then choose your DB instance.
2. Choose the Logs & events tab. Then, in the Logs section, open the alert log (alert_<DB>.log).
3. Note the exact timestamp of the ORA-7445 or ORA-1001 entry so that you can match the error to the trace file that Oracle recorded.
4. Choose the Configuration tab, and confirm that Database counter metrics is Enabled for the DB instance. Alternatively, go to the Monitoring tab and choose View in Database Insights. Then, depending on your retention period, confirm that data exists in the DB Load Analysis tab for the timestamp that you noted in step 3.
5. If Database counter metrics was turned off at the time of the error, then rdsoracleperfmon isn't the source. Investigate the session that the trace file names instead.

Locate the trace file that Oracle recorded

1. In the alert log entry that you noted, find the Errors in file line, and copy the trace file path. For an ORA-7445 error, also copy the path on the Incident details in line and the incident number. A typical ORA-7445 entry looks like the following example:

   Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0xE0] [PC:0x56763AA, plcurClose()+26] [flags: 0x0, count: 1]
   Errors in file /rdsdbdata/log/diag/rdbms/<db>/<DB>/trace/<DB>_ora_12634.trc  (incident=1021850):
   ORA-07445: exception encountered: core dump [plcurClose()+26] [SIGSEGV] [ADDR:0xE0] [PC:0x56763AA] [Address not mapped to object] []
   Incident details in: /rdsdbdata/log/diag/rdbms/<db>/<DB>/incident/incdir_1021850/<DB>_ora_12634_i1021850.trc
   Use ADRCI or Support Workbench to package the incident.
   

2. Open the trace file. On the Logs & events tab of the Amazon RDS console, choose the trace file, and then choose View. For more information, see Viewing and listing database log files in the Amazon RDS User Guide. Alternatively, run the AWS Command Line Interface (AWS CLI) download-db-log-file-portion command to download the file. This command returns only a portion of the log file by default. To download the entire trace file, include the --starting-token 0 parameter, as shown in the following example:

   aws rds download-db-log-file-portion \
     --db-instance-identifier my-db-instance \
     --log-file-name trace/<DB>_ora_12634.trc \
     --starting-token 0 \
     --output text > trace.txt
   

   To list the trace files that are available for the DB instance, run the describe-db-log-files command:

   aws rds describe-db-log-files --db-instance-identifier my-db-instance
   

3. For an ORA-7445 error, use ADRCI to create an incident package for the incident number that you copied in step 1. Then, download the package, and open the trace files that it contains. For more information, see Working with diagnostic data in the Amazon RDS User Guide.

Confirm that the monitoring agent is the source

1. Open the trace file that you located in the previous section.

2. Find the MODULE NAME field, and check the client program that owned the failing session. On RDS for Oracle, you typically observe the Performance Insights collection agent named as the module, as shown in the following example:

   MODULE NAME:(rdsoracleperfmon@ip-123-45-6-789 (TNS V1-V3))
   

3. If you opened an incident trace file from an incident package, then check the program value in the client details section instead:

   client details:
     O/S info: user: rdsmon, term: , ospid: 12608
     machine: ip-123-45-6-789 program: rdsoracleperfmon@ip-123-45-6-789 (TNS V1-V3)
   

4. Confirm that the program that the trace file references is the monitoring agent (rdsoracleperfmon). For an ORA-1001 error, also check whether the trace file includes an ORA-06512 error stack that references SYS.DBMS_XPLAN, which the agent calls when it collects execution plans as part of performance data gathering.

5. Check the call stack in an ORA-7445 [plcurClose] trace file. The failure happens while a cached cursor is closed or released, which is why the error typically has no effect on the data in the database.

Interpret the finding

1. If the trace file references the monitoring agent (rdsoracleperfmon), then the cause is the Performance Insights agent restarting while a cursor that it opened was still being cleaned up. As per Oracle Support documentation, this failure happens at session cleanup, so you can safely ignore it (explained in Oracle KB153505). The failing session was already closing, so no user transaction, query result, or object was affected. No action, configuration change, or workaround is required.
2. If your monitoring flags every ORA- message in the alert log, then add an exception for entries whose trace file references rdsoracleperfmon so that this benign event doesn't generate repeated alerts.
3. If the trace file references a different program, or if Database counter metrics was Disabled at the time of the error, then the monitoring agent isn't the cause. Investigate the session and SQL statement that Oracle records in the trace file.
4. If you need help interpreting the diagnostic data, then contact AWS Support and provide the incident package.

Related information

• Monitoring DB load with Performance Insights on Amazon RDS
• Monitoring Amazon RDS with Amazon CloudWatch Database Insights
• Maintaining a DB instance
• RDS for Oracle database log files
• Viewing and listing database log files
• Downloading a database log file