DTLS 1.3 client and server handshakes (RFC 9147), part 3 of 4 for #1468

[mondain]

DTLS 1.3 (RFC 9147), part 3 of 4: the client and server handshakes.

Stacked on #2440 (reliable handshake), which is stacked on #2439 (record layer). Review those first;
this PR's diff against #2440 is the handshake logic itself.

The series was planned as five parts and is now four: the client and the server are submitted together
here, because the two are mirror images and splitting them would have left a part that could not be tested
against anything. Part 4 is the post-handshake work.

With this PR a DTLSClientProtocol and DTLSServerProtocol pair whose peers list ProtocolVersion.DTLSv13
completes a full certificate-authenticated 1-RTT DTLS 1.3 handshake, including HelloRetryRequest, optional
client authentication, ACK-driven recovery from packet loss, and RFC 5705 / RFC 8446 7.5 exporters. The
immediate motivation is DTLS-SRTP (RFC 5764) over DTLS 1.3 for WebRTC, which is verified here end to end.

What a maintainer will want to check first

No shared-path changes. TlsClientProtocol, TlsServerProtocol, TlsProtocol and RecordStream are
untouched. The only file shared with TLS over TCP that changes at all is TlsUtils, which gains a
null != recordStream guard in establish13TrafficSecrets that TLS callers never reach.

DTLS 1.2 is unchanged on the wire. Every new branch in DTLSRecordLayer and DTLSReliableHandshake is
gated on the negotiated version; the legacy 13-byte record read paths only gain an additional else if; and
the transcript change below is conditional. The full existing DTLS 1.2 and TLS test suites pass unchanged.

There is exactly one behavioural change on the DTLS 1.2 path, and it is a deliberate convergence rather than
a side effect: notifySecureRenegotiation now fires from generateServerHello after
processClientExtensions, which is where TlsServerProtocol has always called it, instead of earlier from
processClientHello. No wire bytes move. The reason for the change is that gating that callback on the
versions the client offered, rather than on the version selected, made a DTLS 1.3 server reject a client
offering both versions without renegotiation_info or the SCSV — a client a BC TLS 1.3 server accepts.

Version ceilings. CLIENT_LATEST_SUPPORTED_DTLS and SERVER_LATEST_SUPPORTED_DTLS both move to
DTLSv13. To be precise about what that does and does not mean: AbstractTlsPeer.getSupportedVersions()
returns TLS versions, so every DTLS peer already names its own version list, and these two constants only
bound what isSupportedDTLSVersionClient/Server will permit. A peer that does not list DTLSv13 is
unaffected. DTLS 1.3 is not silently switched on for anyone.

The transcript

DTLS 1.3 hashes a 4-byte TLS-style handshake header — msg_type plus a uint24 length — not DTLS's 12-byte
header with message_seq, fragment_offset and fragment_length (RFC 9147 5.2). DTLSReliableHandshake
now defers encoding until the version is decided, because the ClientHello is hashed before a version exists,
and DTLSTranscriptHashTest asserts the resulting bytes against an independently computed SHA-256 rather
than against another BC peer.

This is worth calling out because it is the one class of defect that no BC-to-BC handshake can surface: both
peers would have agreed on the wrong transcript and every test would have passed while all interop failed.

DTLS-SRTP over DTLS 1.3

Verified, and the test earns the claim rather than asserting it: both peers derive byte-identical 60-byte
EXTRACTOR-dtls_srtp material; the ServerHello on the wire is checked to carry no use_srtp, proving the
profile was negotiated through the encrypted EncryptedExtensions; and the server deliberately selects the
last of two offered profiles so the client cannot be echoing its own preference. No main-source code was
needed for this — the existing SRTP plumbing works once the exporter does.

Known gaps

Stated plainly, because they are the reasons this is part 3 of 4 rather than the whole feature.

• No interop testing against another DTLS 1.3 implementation yet. This is the honest headline risk. The
  transcript header form is the one interop-critical fact covered by a byte-level test against independently
  computed bytes; the HelloRetryRequest message_hash substitution is the second. Everything else is
  currently proven BC-to-BC. Interop against BoringSSL, wolfSSL, OpenSSL master and NSS is being built out;
  community requests for OpenSSL 4.1/master and for NSS/Firefox on DTLS 1.3 support #1468 are noted and planned.
• A cookie-only HelloRetryRequest is not supported. A stateless third-party server's HRR carries a cookie
  and no key_share; this client answers missing_extension. Marked TODO[dtls13:psk_ke] in place. This is
  the most likely real interop failure, and it is a hard failure.
• No RFC 9147 5.1 stateless DoS countermeasure. DTLS 1.3 forbids HelloVerifyRequest, so the existing
  DTLSVerifier front end cannot front a 1.3 handshake. A 1.3-capable server reached through accept(DTLSRequest, ...) therefore
  refuses to negotiate 1.3 rather than failing obscurely later, with a diagnostic naming the cause. It is
  worth saying why it refuses instead of quietly negotiating 1.2: the DOWNGRD sentinel (RFC 8446 4.1.3) is
  derived from the server peer's own configured versions, not from anything this code can narrow per
  connection, so a local downgrade would leave the sentinel written and every 1.3-capable client aborting. A stateless HelloRetryRequest front end
  analogous to DTLSVerifier needs its own PR — it is a new public class, not a few lines — and an earlier
  draft of this PR that tried to add a half-measure was removed rather than shipped, because a cookie demand
  that is bypassable is worse than none.
• Out of scope here, deliberately: PSK and resumption, 0-RTT early data, connection IDs (RFC 9146), and
  post-handshake KeyUpdate, NewSessionTicket and client authentication. Post-handshake work is part 4. The
  corresponding skip13* paths and TODO[dtls13-psk] markers are in place.
• The second ClientHello's extensions are not compared against the first beyond the fields everything
  downstream depends on, mirroring the existing TODO on the TLS path.

Tests

837 tests in the tls module, no failures; checkstyle clean. tls/src/main/java stays within the legacy
Java 1.4/1.5 Ant build constraints.

Beyond the handshake variants, the tests that carry the most weight are the ones with negative controls:
MTU 512 against MTU 1500 to establish from the wire, without decrypting, that the authenticated client flight
genuinely fragments; a deterministic drop of that flight, with the retransmission proven by the observed
epoch sequence; null versus zero-length exporter context yielding the same material under 1.3 and different
material under 1.2, which distinguishes RFC 8446 7.5 from RFC 5705 4 rather than merely observing that two
outputs differ; and flight-matching unit tests that reject wrong epoch, wrong type, wrong length, truncation,
and a foreign fragment packed beside a real one.

Disclosure

This work was produced with generative-AI assistance, per the contributing guidelines. Every design decision,
every security finding and its fix, and every test were reviewed; the RFC citations above were checked
against the specification text rather than recalled.

Refs #1468.