AWS Network Firewall Issue with openssl 3.5.1+ and matching rules correctly

This issue is not related to a specific Suricata version. Over the past few years, client libraries, applications, and programming languages have been adopting Post-Quantum Cryptography (PQC), including hybrid key exchange mechanisms such as X25519Kyber768Draft00 and X25519MLKEM768. OpenSSL confirmed support for PQC algorithms starting with version 3.5.

These algorithms change the TLS handshake behavior between client and server. Specifically, the client includes a large key share value in the TLS ClientHello message — the same message that firewalls and proxies inspect to extract the destination domain from the Server Name Indication (SNI) extension. In a typical Internet egress path, firewalls are often configured with application layer rules that allow HTTPS requests to approved domains while denying all others.

Because standard Internet MTU is limited to 1500 bytes, the addition of a large PQC key share causes the ClientHello to exceed this limit. The client must then segment the ClientHello across multiple TCP segments. A firewall in the path needs to reassemble these individual segments into the complete ClientHello before it can evaluate the SNI against configured stateful application layer rules.

If your firewall policy includes broadly scoped default drop rules (e.g., drop ip any any, drop tcp any any, or reject tls any any), these rules may inadvertently drop the individual TCP segments before the firewall has an opportunity to reassemble and inspect the complete TLS ClientHello. When this occurs, Suricata typically identifies the flow with proto: tcp and app_proto: tls, meaning both catch-all TCP rules and TLS-specific rules can match and cause the drop. Since Suricata is entirely rule-driven, the default drop rules need to account for this PQC behavior. Two options to consider:

1. Exclude the TLS application layer from a catch-all TCP rule — If using a broad drop tcp or reject tcp rule, add app-layer-protocol:!tls; to exempt TLS traffic from the catch-all, allowing the TLS handshake to complete reassembly before being evaluated by a dedicated TLS catch-all rule such as a drop tls or reject tls rule as described in option 2.

2. Use ssl_state:client_hello on default drop TLS rule — If using a drop tls or reject tls catch all rule, add the ssl_state:client_hello; keyword so the rule only applies after the ClientHello has been fully reassembled. This prevents the rule from matching on individual TCP segments that are part of the fragmented ClientHello.

The options above apply to customer-authored catch-all rules commonly seen. For AWS Network Firewall configured with strict evaluation order, the AWS public documentation includes additional approaches:

1. Use the Application Layer Drop Established stateful default action instead of Drop Established. Please refer to the AWS Network Firewall Developer Guide and the Network Firewall Best Practices Guide for additional details and rule considerations regarding this stateful default action.

2. Implement self-managed default drop rules instead of using the built-in stateful default actions. The self-managed rules in the documentation take a slightly different approach, leveraging flowbits and other keywords including ssl_state:client_hello to ensure TLS reassembly completes before drop rules are evaluated. Example egress drop rule configurations can be found in the AWS Network Firewall Best Practices Guide.

We recommend reviewing your current rule configuration to determine which approach best fits your environment. As a best practice, please test any rule changes in a non-production environment before applying them to production to ensure they behave as expected.