Intermittent RedisCommandTimeoutException with AWS Valkey Serverless despite low CPU/memory usage

Low Valkey CPU/memory means the server isn't the bottleneck — the issue is client-side. Two most likely causes:

1. Single multiplexed connection saturated

Lettuce uses one TCP connection by default. With multiple SQS consumers issuing synchronous RedisTemplate calls, commands queue on that single connection. Any network jitter or TLS renegotiation causes the queue to back up → 5s timeout.

Fix — enable connection pooling:

LettucePoolingClientConfiguration clientConfig =
    LettucePoolingClientConfiguration.builder()
        .poolConfig(new GenericObjectPoolConfig<>() {{
            setMaxTotal(16);
            setMinIdle(4);
        }})
        .clientOptions(clientOptions)
        .useSsl()
        .build();

2. Stale topology after slot migration

Valkey Serverless rebalances slots transparently. Your 60s topology refresh means Lettuce routes to stale nodes for up to a minute after a migration → MOVED redirections pile up → timeout.

Fix — reduce refresh interval:

.enablePeriodicRefresh(Duration.ofSeconds(15))  // was 60

Also reduce keepalive idle from 30s to 15s — Valkey Serverless may drop idle connections before your client detects it:

.idle(Duration.ofSeconds(15))
.interval(Duration.ofSeconds(5))

To confirm: check CloudWatch NewConnections (frequent reconnects = connection drops) and CurrConnections (hitting limits = pool exhaustion). Correlate timeout timestamps with any Valkey scaling events.

Based on your configuration and symptoms, here are several areas to investigate for intermittent timeout issues with Valkey Serverless:

1. Client-Side Timeout Configuration

Your current setup uses a 5-second command timeout, which may be appropriate for most operations but could be problematic in certain scenarios. Consider:

• Differentiate timeouts by operation type: Simple GET operations typically complete quickly, while SETEX and operations involving larger values may need more time. You might want to implement different timeout values for different command types rather than using a fixed 5-second timeout for all operations.

• Review your DynamicClusterTimeout implementation: Your custom timeout source with 5000ms command timeout and 2000ms connection timeout should be evaluated. The connection timeout (2000ms) is reasonable, but ensure your dynamic timeout logic isn't inadvertently causing issues during high-concurrency periods.

2. Connection Pool and Resource Management

With multiple Kubernetes pods (2-8 per microservice) and SQS-driven async consumers, you may be experiencing connection resource constraints:

• Implement connection pooling: Ensure you're using proper connection pooling to avoid exhausting available connections. Each pod creating multiple connections without proper pooling can quickly reach connection limits.

• Monitor connection counts: Check if you're approaching the maximum number of client connections. When the connection limit is reached, new connection attempts will fail or timeout. Use connection pooling and set appropriate timeout values to manage this.

• Configure clients for direct replica connections: If you're using read replicas, configure your clients to connect directly to them to distribute the connection load.

3. Network and Latency Considerations

Even with low CPU/memory on Valkey, network issues can cause timeouts:

• Check for network constraints: Verify there are no network bandwidth limitations, packet loss, or latency spikes between your Kubernetes cluster and Valkey Serverless. Network issues can manifest as intermittent timeouts even when the cache itself is healthy.

• Review security group rules: Ensure security groups allow traffic on the correct port and that there are no intermittent firewall or network policy issues in your Kubernetes environment.

4. Command Execution Patterns

Your timeout errors can occur when the Redis thread is processing complex or long-running commands:

• Check for slow-running commands: Even though your operations are simple key-value operations, review if there are any occasional slow commands being executed. You can enable and monitor the SLOWLOG to identify commands taking longer than expected.

• Evaluate command volume: High volumes of commands per unit time can cause queuing and delays. Monitor whether you're sending excessive commands during peak periods, which could lead to timeouts even with healthy resource utilization.

5. Serverless-Specific Considerations

With Valkey Serverless, there may be scaling or cold-start behaviors:

• Understand serverless scaling patterns: Serverless offerings may have different performance characteristics during scaling events. Monitor for any correlation between timeout occurrences and potential scaling activities.

• Implement retry logic with backoff: For intermittent failures, implement exponential backoff retry logic in your application to handle transient issues gracefully.

6. Monitoring and Diagnostics

To better diagnose the issue:

• Enable detailed logging: Configure slowlog-log-slower-than to a value appropriate for your workload (potentially lower than the default 10ms) to capture commands that are taking longer than expected.

• Monitor replication metrics: If using replication, check ReplicationBytes and ReplicationLag metrics to ensure replication health isn't contributing to delays.

• Track latency metrics: Monitor command-specific latency metrics, SuccessfulWriteRequestLatency, and SuccessfulReadRequestLatency to understand actual command execution times versus timeout thresholds.

Immediate Actions to Consider:

1. Review and potentially increase your command timeout for write operations (SETEX) while keeping GET timeouts lower
2. Verify connection pooling is properly configured across all your pods
3. Monitor actual connection counts against available limits
4. Enable SLOWLOG and review for any unexpectedly slow commands
5. Check for network latency or packet loss between Kubernetes and Valkey
6. Implement retry logic with exponential backoff for transient failures

The intermittent nature of your timeouts, combined with healthy CPU/memory metrics, suggests the issue is likely related to connection management, network latency, or occasional command queuing rather than resource exhaustion on the Valkey side.
Sources
Troubleshoot error messages in ElastiCache Redis clients | AWS re:Post
Configure a client-side timeout (Valkey and Redis OSS) - Amazon ElastiCache
Amazon ElastiCache Well-Architected Lens Performance Efficiency Pillar - Amazon ElastiCache