How do I troubleshoot common errors when I use the OEM_AGENT option with Amazon RDS for Oracle?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Amazon RDS supports the Oracle Enterprise Manager (OEM) Management Agent through the OEM_AGENT option.

First, run the prechecks on your DB instance, and then see the related error section. To troubleshoot issues when you use the OEM_AGENT option, review the OEM agent logs after you export these logs to Amazon CloudWatch. For more information, see Publishing Oracle logs to Amazon CloudWatch Logs.

Prechecks

Before you start the resolution steps, complete the following prechecks on your DB instance:

1. Use the add-option-to-option-group AWS CLI to apply the OEM_AGENT to the option and to the DB instance.

2. To check the option group status, run the following AWS CLI command:

   aws rds describe-db-instances --db-instance-identifier <db-instance-name> --query 'DBInstances[*].[Engine,DBInstanceStatus,OptionGroupMemberships]'

3. Confirm that the output option group status is in-sync.
   If the option group status is INVALID, then the OEM_AGENT isn't installed because of issues with the network configuration setup and other prerequisites. Review the Events tab of your Amazon RDS DB instance from the AWS Management Console to verify the installation status of the OEM_AGENT option. You can also use the AWS CLI command describe-events. If the events show an issue with the installation of the OEM_AGENT option or network configuration, then make sure that the OEM_AGENT prerequisites are met.

4. Check the DB instance network configuration. The DB instance security group must allow OMS_HOST to listen to the OEM_AGENT port (default is 3872) and the RDS port (default is 1521).

5. Run a telnet test from the Oracle Management Service (OMS) server to your DB instance on the OEM_AGENT port and database port to check connectivity.

6. Check your network configurations. For example, check network access control lists (ACLs) and route tables. Do this to rule out the possibility of blockers or an explicit deny.

7. Make sure that the firewall between OMS and RDS allows traffic on both the DB listener port and OEM_AGENT port.

After you complete these steps, check that the OEM_AGENT upload works. For more information, see Making the Management Agent upload its OMS.

Troubleshoot errors

The DB instance doesn't show up in the Auto Discovery of targets on the OEM console

This issue occurs when the OMS server can't connect to the underlying host from SSH. This connection is a prerequisite at the OS level for Auto Discovery to work correctly. To resolve this error, you must manually add your Oracle DB instance as the target. For more information, see Limitations for Management Agent.

Unable to install the Oracle OEM_AGENT because the agent password is incorrect or expired

Make sure that the agent password is correct and that it isn't expired. Use the OEM server, to modify the agent registration password or create a new password.

Unable to install the Oracle OEM_AGENT because the DB instance can't reach the OMS host

You receive this error when the OEM_AGENT fails to install because the OMS host/port can't be reached from the RDS host. To troubleshoot this error, check whether the OMS host can be reached from your DB instance.

To validate the network connectivity between the OMS server and OEM_AGENT, test the connection from the RDS for Oracle instance to the OMS server. Do this so you can leverage access control lists (ACL) and UTL_TCP packages.

• Use the DBMS_NETWORK_ACL_ADMIN package that provides the interface to administer the network ACL. For more information, see Using DBMS_NETWORK_ACL_ADMIN on the Oracle website.
• Use the UTL_TCP.CONNECTION. a PL/SQL record type that's used to represent a TCP/IP connection. For more information, see Using UTL_TCP on the Oracle website.

Complete the following steps:

1. Open the Amazon Elastic Compute Cloud (Amazon EC2) console.

2. Launch an Amazon EC2 instance with the same network setup (SG/ACL) as your DB instance.

3. Run the following telnet command from the Amazon EC2 instance to the OMS host on port 4903:

   telnet OMS_HOST 4903

4. To validate the connectivity, run the following telnet test command in your OEM host from the OMS server to your DB instance:

   telnet RDS-instance-endpoint 1521 (RDS default port)

5. To check whether the RDS host can resolve the OMS hostname into an IP address, run the following command:

   SQL> SELECT UTL_INADDR.get_host_address('OMS_Host_Name') FROM dual;

6. Run a TCP Traceroute to check where the traffic is blocked.

You installed the OEM_AGENT option on your DB instance. Your security group might not be configured correctly

Even if your installation completes correctly, the RDS security group associated with your DB instance might be missing configurations. These configurations allow communication between the OMS host and DB instance.

To resolve this error, verify that the security group of the agent allows inbound traffic. And that your OMS host belongs to a security group that has access to the agent port. For more information, see Enterprise Manager Cloud Control Security Guide on the Oracle website.

Unable to install the Oracle OEM_AGENT because your OMS host version x.x.x.x isn't compatible with the agent version x.x.x.x.

You receive this error when there is a compatibility issue between your OEM_AGENT version and the OMS host version. OEM_AGENT integrates with the OMS when both components are compatible. To resolve this error, choose compatible versions for both OMS host and OEM agent. For information, see Requirements for Management Agent.

Your OMS host uses an untrusted third-party certificate

You receive this error when you successfully install the OEM_AGENT option, but your OMS host uses a third-party certificate that isn't trusted. To resolve this error, configure your OMS host with the required trust certifications from your third party.

There are missing required option settings in your OEM_AGENT option (Service: AmazonRDS; Status Code: 400; Error Code: InvalidParameterValue)

You receive this error when there are missing required settings in your OEM_AGENT. You must specify this setting. For more information on the required settings for OEM_AGENT, see Option settings for Management Agent.

Heartbeat Status: OMS responded illegally [ERROR - Failed to Update Target Type Metadata]

You receive this error when the OMS host is replaced after the OEM_AGENT option is attached to Amazon RDS.

To resolve this error, complete the following steps:

1. Clear the agent status, or restart the OEM_AGENT. For more information, see Administering the Management Agent.

2. Re-establish your connection with the OMS host.

3. Check for compatibility issues with the OMS version and OEM_AGENT version. To check if the table shows the OEM_AGENT version used in the option group, run the following command:

   select type_meta_ver from sysman.mgmt_target_type_versions where target_type = 'oracle_emd';

4. If the mgmt_target_type_versions output doesn't contain the OEM_AGENT version used in the option group, then install the OEM_AGENT version that is listed in the command output. This error shows that the required OMS side patches and plugins are missing. Be sure that OMS is set up correctly and all the required patches are applied.

If the agent is blocked, complete the following steps in the OEM console to resync the agent:

1. Sign in to the Cloud Control console.
2. Choose Setup, choose Manage Cloud Control, and then choose Agents.
3. Choose the agent that you want to resync.
4. From the Agent menu, choose Resynchronization....
5. Select Unblock agent on successful completion of agent resynchronization.
6. Choose Continue. The resync operation is submitted as a job.
7. Choose the job name link to check the resynchronization job's status.

After the job completes, verify the status of the agent that you resynchronized and all monitored targets.

Note: The DNS server must have continual functionality for the OEM monitoring to work effectively. The Agent emits heartbeats and pushes status updates to the OMS host. If the OMS host isn't reachable from the agent for an extended period of time, then OMS shows that the agent and database are down. Be sure that the DNS server functions.

To make the Oracle Management Agent upload the OMS associated with it, run the following command. When you run this command, its the same as when you run the emctl upload agent command. For more information. see EMCTL commands for Management Agent on the Oracle website.

SELECT rdsadmin.rdsadmin_oem_agent_tasks.upload_oem_agent() as TASK_ID from DUAL;

To restart the OEM agent after the agent state is cleared, run the following command:

SELECT rdsadmin.rdsadmin_oem_agent_tasks.restart_oem_agent() as TASK_ID from DUAL;

You can deploy only Oracle Management Agent 13c Release 4 (13.4.0.0.0) in the Oracle Enterprise Manager Cloud Control 13c Release 4 (13.4.0.0.0). Fresh deployment of earlier versions of Oracle Management Agent (13.2 and 13.3) aren't supported after OMS is upgraded to 13c Release 4. For more information, see Before you begin installing an Enterprise Manager System on the Oracle website.

Unable to install the OEM_AGENT option because the agent port conflicts with the OMS port. Update the option settings and try again

You receive this error because you have the wrong configuration for OEM_AGENT to work correctly. You might have specified the same port number for both the OMS port and the OEM_AGENT port. To resolve this error, change either the OMS port or the OEM_AGENT port number. Review the following Management Agent option settings:

• AGENT_PORT. This port on the DB instance listens for the OMS host. The default is 3872. Your OMS host must belong to a security group that has access to this port.
• OMS_PORT. This HTTPS port on the OMS Host listens for the Management Agent. To find the HTTPS upload port, connect to the OMS host and run the following command:

  emctl status oms -details

Unable to install the Oracle OEM_AGENT because your DB instance doesn't have enough storage. Confirm that option group is supported on your DB instance class and configuration. If so, verify all option group settings and retry

You receive this error when the storage for your DB instance doesn't have enough available storage as defined in the OEM_AGENT prerequisites. Increase the storage space, and then reinstall the OEM_AGENT option.

Filesystem / has X.XX% available space

You receive this error because of a use limitation in the OEM_AGENT option with an RDS for Oracle instance. Host metrics and the process list might not reflect the actual system state. Avoid the use of OEM to monitor the root file system or mount point file system. For more information, see Limitations for Management Agent.

The root file system of an Amazon RDS instance is maintained by the internal automation system. This automation system monitors the root file system at regular intervals to make sure that the file system has adequate space. If insufficient storage is detected, then the automation system adds adequate space to the root file system. Because the automation system manages the space in the root file system, you can ignore this error.

Related information

Oracle Management Agent for Enterprise Manager Cloud Control