How do I troubleshoot error messages in ElastiCache Redis clients?

Resolution

"Connection reset by peer" error

When the Redis server terminates the connection, you receive the following error message:

"java.io.IOException: Connection reset by peer
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer"

This error can originate from the server or client.

If you receive the error from the server, then take the following actions to resolve the issue:

• Check for underlying hardware or network issues.
• Run the CLIENT LIST command to check the client output buffer quota. For more information, see CLIENT LIST on the Redis website. Parameters in your cluster's parameter group define the client output buffer quotas in ElastiCache.
• Check whether the timeout value in your ElastiCache parameter group reached its quota.

If you receive the error from the client, then take the following actions to resolve the issue:

• Check whether the timeout settings in your application are too low.
• Check the official documentation or support pages for your Redis client library for known bugs that are related to connection resets.
• Verify that you have the latest version of your Redis client.
• If you turned on TLS for the cluster, then confirm that you configured encryption on the client.

"Connection refused" or "Unable to connect to Redis" error

When the server refuses the connection, you receive the following error message:

"JedisConnectionException: java.net.ConnectException: Connection refused
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused
ECONNREFUSED
org.springframework.data.redis.RedisConnectionFailureException: Unable to connect to Redis org.redisson.client.RedisConnectionException: Unable to connect to Redis server ###"

To resolve this issue, verify the following:

• The Redis endpoint is reachable. You can use tools such as, telnet or redis-cli.
• The security group rules allow traffic on port 6379.
• No firewalls are blocking the connection.
• There are no memory, CPU, network, or connection resource constraints.

Note: You can use tools such as tcpdump to analyze connection attempts.

"Connection timed out" error

When the connection times out, you receive the following error message:

"redis.exceptions.TimeoutError: Timeout connecting to server io.netty.channel.ConnectTimeoutException: connection timed out org.redisson.client.RedisTimeoutException: Unable to acquire connection!"

The Connection timed out error can be persistent or intermittent.

If the timeout error persists, then take the following actions to resolve the issue:

• Verify network connectivity and configuration to confirm connection to the ElastiCache Redis cluster.
• Confirm that you specified the correct Redis endpoints.

To troubleshoot persistent timeout errors, see Persistent connection issues.

If the timeout errors are intermittent, then take the following actions to resolve the issue:

• Use the ElastiCache console to check for node recovery or failover events.
• Check for memory, CPU, network, or connection resource constraints.
• Check if latency was high at the time of the issue.
• Check the timeout configuration in your client application.

"Read timed out" error

When the command doesn't complete within the timeout value that you configured, you receive the following error message:

"StackExchange.Redis.RedisTimeoutException: Timeout performing GET (5000ms) io.lettuce.core.RedisCommandTimeoutException: Command timed out org.springframework.data.redis.RedisConnectionFailureException: java.net.SocketTimeoutException: Read timed out redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out"

This error occurs when the Redis thread is processing complex or long-running commands.

To resolve the issue, take the following actions:

• Check for memory, CPU, network, or connection resource constraints.
• Verify that there's no resource throttling on the client side.
• Check if latency was high at the time of the issue.
• Check for long-running commands that block the Redis engine because Redis is a single threaded engine that processes one query at a time. For instructions, see SLOWLOG GET on the Redis website.
• Check for excessive commands per unit time. For instructions, see INFO on the Redis website.

"Operation timed out" error

When the connection remains idle longer than the timeout value that you configured, you receive the following error message:

"io.lettuce.core.RedisException: java.io.IOException: Operation timed out"

You might receive the Operation timed out error for the following reasons:

• The ElastiCache cluster has high load and can't process commands in time.
• Network traffic exceeds the bandwidth quota on the cluster or client side.

To resolve the issue, take the following actions:

• Check for memory, CPU, network, or connection resource constraints.
• Verify that there's no resource throttling on the client side.
• Check for long-running commands that block the Redis engine. For instructions, see SLOWLOG GET on the Redis website.
• Check the timeout configuration in your client application and cluster, and set appropriate timeout parameters to prevent idle disconnections.

"NOAUTH Authentication required" error

When you connect to a Redis server that requires authentication but doesn't provide valid credentials, you receive the NOAUTH Authentication required error message.

To resolve the issue, take the following actions:

• Provide the correct username and password to connect to the cluster.
• Confirm that the client library configuration and connection string has the correct syntax.

"ERR max number of clients reached" error

When a cluster node reaches its maximum connection quota, you receive the ERR max number of clients reached error message.

To resolve the issue, take the following actions:

• Check the CurrConnections metric and confirm that it's within the maxclients quota. For more information, see Maximum concurrent connected clients on the Redis website.
• Use connection pooling to reuse connections.
• Set appropriate timeout values in your client configuration.
• Configure clients to connect directly to read replicas for read operations to distribute the load across cluster nodes.

"LOADING Redis is loading the dataset in memory" error

When you connect to a node that's loading its dataset into memory during startup or replica synchronization, you receive the following error message:

"LOADING Redis is loading the dataset in memory (error) LOADING Redis is loading the dataset in memory"

It's a best practice to implement backoff. For information, see the Redis cluster client discovery and exponential backoff section in Best practices: Redis clients and Amazon ElastiCache for Redis.

"OOM command not allowed when used memory > 'maxmemory'" error

The maxmemory parameter sets the maximum amount of memory that a Redis node can use for data storage.

When memory usage exceeds the maxmemory quota, Redis applies the maxmemory-policy that you set in the parameter group. Based on the maxmemory-policy, Redis evicts keys or returns the OOM command not allowed error.

To resolve the issue, take the following actions:

• Configure the maxmemory-policy to evict keys based on your application requirements.
• Upgrade the cluster to a larger node type when memory usage consistently exceeds the quota.

"CROSSSLOT Keys in request don't hash to the same slot" error

When you run multi-key operations on a cluster that has cluster mode turned on and different hash slots, you receive the CROSSSLOT error.

To resolve this issue, see How do I resolve the CROSSSLOT error I receive when I use multi-key operations on an ElastiCache (Redis OSS) self-designed cluster?

"CLUSTERDOWN The cluster is down" error

When a node stops responding, the cluster becomes unavailable and you receive The cluster is down error message.

To resolve the issue, take the following actions:

• Use the ElastiCache console to check for node recovery or failover events.
• Check for memory, CPU, network, or connection resource constraints.
• Check whether the issue originates from slow-running commands For instructions, see SLOWLOG GET on the Redis website.

"READONLY You can't write against a read-only replica" error

The following are reasons why you receive an error when you try to write to a Redis node that you configured as a read replica:

• Your application connects directly to a read replica.
• DNS caching in the application routes traffic to the old primary node.

To resolve the issue, see How do I troubleshoot the READONLY error that I receive after a failover in my ElastiCache for Redis self-designed cluster?

"MOVED [slot] [IP address] [port]" error

The MOVED redirection message contains the hash slot, the IP address, and port of the node for the requested key:

"-MOVED 3999 127.0.0.1:6381"

The following are reasons why you might receive the MOVED error:

• The client tries to access a key in a hash slot that isn't on the connected node.
• You turned on cluster mode for the cluster, but the client isn't cluster aware.
• Your read replica doesn't have the readonly flag.
• Your client has outdated cluster information, such as a change in the number of shards or nodes.

To resolve the issues, take the following actions:

• If you turned on cluster mode, then confirm that the client is cluster aware.
• Run the readonly command to make sure that the replica node is reading requests. For information, see How do I resolve Redis client read requests that are redirected to the primary node of a shard instead of a replica node in ElastiCache for Redis?
• Configure the client to frequently query the cluster topology.
  Note: To view the details of the cluster's shards, run the CLUSTER SHARDS command. For cluster node details, run the CLUSTER NODES command. For more information see, CLUSTER SHARDS and CLUSTER NODES on the Redis website.