[ssegura]

Related: None. The last of the three rules of fault the plan's version 2.2.0 takes from requests RESTest already sends, after #403.

What you can do now that you could not before

A run now tells you when the API took a request and never answered it: the connection was open, the request went out, and nothing came back before RESTest stopped waiting — half a minute by default. Until now such a request was only one more noReply in a count, and a run whose only problem was an operation that hangs ended with no faults found.

Try it yourself

With only Java and git. The "API" is a stand-in of a dozen lines on the small HTTP server that comes with Java: /pets answers at once, /kennels takes a minute. The run waits two seconds for a reply instead of thirty, so that it is quick.

git fetch origin && git switch feat/m3-1c-no-reply-in-time
./mvnw -q install -DskipTests
mkdir -p try
cat > try/Stand.java <<'EOF'
import com.sun.net.httpserver.HttpServer;
import java.net.InetSocketAddress;
import java.util.concurrent.Executors;

/** A stand-in API: /pets answers at once, /kennels takes a minute. */
public class Stand {
    public static void main(String[] args) throws Exception {
        HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 8765), 0);
        server.createContext("/pets", exchange -> {
            byte[] body = "[{\"name\": \"Rex\"}]".getBytes();
            exchange.getResponseHeaders().add("Content-Type", "application/json");
            exchange.sendResponseHeaders(200, body.length);
            exchange.getResponseBody().write(body);
            exchange.close();
        });
        server.createContext("/kennels", exchange -> {
            try { Thread.sleep(60_000); } catch (InterruptedException stopped) { return; }
            exchange.sendResponseHeaders(200, -1);
            exchange.close();
        });
        server.setExecutor(Executors.newVirtualThreadPerTaskExecutor());
        server.start();
    }
}
EOF
cat > try/kennels.yaml <<'EOF'
openapi: 3.0.3
info: {title: Kennels, version: "1"}
paths:
  /pets:
    get:
      operationId: listPets
      responses:
        "200": {description: the pets}
  /kennels:
    get:
      operationId: listKennels
      responses:
        "200": {description: "the kennels, after a minute"}
EOF
java try/Stand.java &
./restest run try/kennels.yaml --url http://127.0.0.1:8765 --budget 10s --seed 1 --set engine.readTimeout=2s

Real output, from a fresh clone of this branch (the first two of the faults printed, then the end); the run ends with 1:

RESTest testing Kennels at http://127.0.0.1:8765

2 of 2 operations can be tested, seed 1, budget 10s
  what it sends depends on the API's own replies, so the seed alone does not repeat this run; --store keeps what it sent
  2 setting(s) are not what RESTest does by default; --print-settings lists them with where each came from

F312  Call Timeout
      listKennels - GET http://127.0.0.1:8765/kennels  ->  no reply
      no reply within 2.0s: a connection to the API was open, and RESTest stopped waiting before a reply's status line and headers came back
      curl -i -X GET 'http://127.0.0.1:8765/kennels' -H 'Accept: */*' -H 'User-Agent: RESTest/2.0'

F312  Call Timeout
      listKennels - GET http://127.0.0.1:8765/kennels  ->  no reply
      no reply within 2.0s: a connection to the API was open, and RESTest stopped waiting before a reply's status line and headers came back
      curl -i -X GET 'http://127.0.0.1:8765/kennels' -H 'Accept: */*' -H 'User-Agent: RESTest/2.0'
...
82 requests to 2 operations in 14.5s, 3% of it idle
  opening lap: 2 requests in 2.0s, 1 of 2 operations answered 2xx
  41 2xx, 41 no reply
41 faults:
  41 x F312  Call Timeout
report written to restest-out/report.json (13.5 KiB)
the run itself was not kept; pass --store to keep every request and reply

Every request to /kennels is a fault; every one to /pets was answered and is not. The check switches off in one line, and the run then reports nothing, as master does — the requests are still counted, under no reply:

./restest run try/kennels.yaml --url http://127.0.0.1:8765 --budget 10s --seed 1 --set engine.readTimeout=2s --set checks.noReplyInTime=false

  3 setting(s) are not what RESTest does by default; --print-settings lists them with where each came from

99 requests to 2 operations in 12.4s, 3% of it idle
  opening lap: 2 requests in 2.0s, 1 of 2 operations answered 2xx
  50 2xx, 49 no reply
no faults found

That run ends with 0. Stop the stand-in with kill %1 and remove try/ and restest-out/.

The run lasts longer than its budget because it waits, at the end, for the answers to requests it has already sent; that is not new. A run against an address where nothing answers anything is tested end to end by RestestTest:

./mvnw -pl restest-cli -am test -Dtest='RestestTest#an_address_that_never_answers*' -Dsurefire.failIfNoSpecifiedTests=false | grep "Tests run:" | tail -2

[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.891 s -- in io.restest.cli.RestestTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

How it works

RESTest gives each request a set time: half a minute for anything to arrive (engine.readTimeout), a minute for the whole request (engine.callTimeout). Until now, when the time ran out, the part that sends requests wrote down only a sentence such as SocketTimeoutException: timeout, which said the same as a connection that was refused or cut. It now also writes down what it was still waiting for: a connection, when none could be made in time, so the request never reached the API; or a reply, when a connection to the API was open and the status line and headers that begin a reply had not come back. It knows which because it notes, at the last step before the request goes on the wire, that a reply is awaited, and crosses the note out when one starts to arrive.

A new rule reads that note: a request that timed out waiting for a reply is a fault, F312, which the shared catalogue of faults (Web Fuzzing Commons, Faults 0.9.0) calls Call Timeout. A connection that could not be made in time is not one — a wrong or firewalled address looks exactly the same — and nor is a reply that began and then stopped half-way, which is what an endpoint that streams does. The rule has a switch, checks.noReplyInTime, on by default.

What was waited for is kept with the request in report.json and in the stored run, as "timedOutWaitingFor": "reply", so a stored run can be judged again later. A run in which nothing answered still ends with 3 ("nothing was tested"), since something that accepts connections and never speaks HTTP does exactly the same; its last line now says how many requests were sent on an open connection and never answered, and that an API that is only slow can be given longer.

What changed in the code

• restest-core: InteractionOutcome.TransportFailure gains Optional<Wait> timedOutWaitingFor, with Wait either CONNECTION or REPLY; Interaction gains a transportFailure(...) that takes it, and timedOutWaitingForAReply(), the one check the rule and the run loop share. InteractionDocument writes timedOutWaitingFor only when there is one and reads it back, so a document written before reads as no timeout, and refuses a word it does not know. Secrets, which hides a key handed over with --auth, keeps the wait in both places it rebuilds a failure. CheckSettings, SettingKey and Settings: checks.noReplyInTime. Ninety-four settings in nine groups.
• restest-exec: WireCapture.Slot.awaitingAReply, set just before the network interceptor passes a request on and cleared when the status line and headers come back. OkHttp runs that interceptor only once a connection is open, and again for each address a redirection sends it to. OkHttpEngine.timedOutWaitingFor(failure, awaitingAReply): a SocketTimeoutException, or an InterruptedIOException on a thread that is not interrupted (OkHttp's limit on the whole request), is a timeout; anything else is not.
• restest-oracles: NoReplyInTimeOracle, registered in module-info.java, the services file and last in Oracles.standard: 100, 121, 200, 206, 312.
• restest-cli: RunLoop counts the requests that timed out waiting for a reply (Outcome.noReplyInTime); RunCommand says so in the line a run ends with when nothing was answered.
• Tests: TimedOutWaitingForTest (9 cases) names each kind of failure; EngineFaultsTest makes the real ones happen against real sockets — a read timeout, a status line that trickles in until the whole request runs out, a TLS handshake that never finishes (CONNECTION), a redirection to such an address (CONNECTION), a body that stalls after its headers (still a malformed reply). NoReplyInTimeOracleTest (14); InteractionTest, InteractionDocumentTest, SecretsTest, JsonReportTest; OraclesTest and SettingsTest over five switches; CheckSwitchesCommandTest with a stand-in operation that answers after four seconds, five kinds of fault, and the switch off; RestestTest, an address that takes connections and never answers, with the check on and off.
• Documentation: docs/faults.md (the kind, with a real fault from a run, what is and is not reported, and why), docs/switches.md, docs/settings.md, docs/report.md (timedOutWaitingFor), docs/known-limitations.md (a hang the run's pace provoked, a pooled connection something in between dropped, a redirection followed to an address that hangs, a slow reply), docs/DESIGN.md, the README, the manual's chapters 1, 10 and 11. Amendments to ADR-0005 (what a missing reply records), ADR-0016 (F312, how other tools treat it, the measurements) and ADR-0025 (the switch).
• Left for later: a reply that arrives but slower than a limit (what Schemathesis reports as 312); splitting noReply in report.json by cause; report.json's JSON Schema, which is the next change to report.json's.

Evidence

• Tests: about fifty new cases, in two new classes and nine changed ones; ./mvnw verify green on all nine modules, 4,422 tests — CI: CI_LINK
• Architecture tests: unchanged, all green (the documentation rules hold the settings, switches and command-line pages to the code) — none weakened
• Mutation score, restest-oracles: 97% (195 of 201; 194 of 200 before), and every mutant in the new rule and in Oracles killed
• Per-request overhead: the rule is one look at how the attempt ended, and the engine adds two writes of a flag per request. On the smoke APIs, three runs each, 92,267–98,525 requests on the pet shop (86,250 before) and 52,479–61,166 on the scouts' catalogue (53,268–62,072 before): no difference beyond noise
• Smoke run, the pet shop and the scouts' catalogue, 10 s, seed 20260914, three runs: operations tested 19 and 49 (19 and 49), answered 2xx in the opening lap 14 and 18 (14 and 18), operations answering 500 16 and 12 (16 and 12), kinds of fault 3 and 2–3 (3 and 2–3), operations with faults 17 and 23 (17 and 23). No request went unanswered on either
• Idle time: 3.9–4.3% on both smoke APIs (4.5% before)
• Artefact: fifteen APIs of the Web Fuzzing Dataset run locally for a minute each before the rule, their stored runs read back: two had requests that timed out on an open connection with no reply, 42 on six operations, all at the 30-second read timeout — catwatch's fetch, which gathers data from GitHub and waits on it (29), and kafka-rest's five operations on a broker's configuration, which hang for a cluster that does not exist instead of answering 404 (13). Run again with the rule, a minute each, seed 42: catwatch 30 F312, all on fetch; kafka-rest 15 on six operations, the same five and one that sends a batch of records, which hung before any reply in this run. kafka-rest's 13 connections cut with nothing sent back, and its streaming endpoint that answered 200 and then sent nothing, were not reported. Both APIs also answered 500, so neither changes its exit code
• Size: no dependency added; the archive and the image grow by the new classes only

Decisions taken

• What counts, decided with the maintainer: a request on an open connection to which no status line and headers came back, whichever of the read, write or whole-request limits ran out. Not a connection that could not be made in time, and not a reply whose status line and headers arrived and whose body stalled, which stays a malformed reply counted under its status code. A request still being sent when time ran out counts too: OkHttp throws the same exception for a stalled write and a stalled read, and the bodies RESTest sends are far smaller than what the operating system buffers, so a write stalls only when the API stops reading a connection it holds open.
• On by default, chosen by the maintainer, as CATS and RESTler report it; EvoMaster has its own equivalent, code 960, off by default; Schemathesis reports a request with no reply as an error of the run and uses 312 for a reply slower than a limit, which is off unless a limit is given. ADR-0016's amendment has the details, checked in each tool's code.
• The code is F312, the catalogue's Call Timeout. The catalogue files it among its security kinds, but it is not one of the security oracles docs/DESIGN.md keeps out of scope: it sends nothing and judges an outcome every run already records; the plan's row for it is the approval.
• The exit codes are unchanged: a run where nothing answered ends with 3 even when it reported F312, and says how many requests sat on an open connection beside the advice; the line names both causes and how to give a slow API longer, and the manual's chapter 10 says the same. The count comes from the run loop, not from the faults reported, so it holds with the check off.
• A redirection followed to an address that takes the request and never answers is reported on the operation, and documented as a known limitation, rather than told apart: redirections are followed only when a run asks for it (engine.followRedirects, off by default).
• report.json has no JSON Schema yet, by the maintainer's choice: it is left to the next change of report.json's shape, which changes it more than one optional member.
• Departures from the approved plan. The closing line's count comes from the run loop instead of from the faults reported, found by the review of the code's design. The one-argument constructor of TransportFailure was dropped, having no caller, and Interaction.timedOutWaitingForAReply() was added so that the rule and the run loop share one check. The amendments are dated 17 October 2026, the day they were written; the decisions they record were taken on the 15th. The tests that need an address that takes connections and never answers use a socket that listens and is never read, whose connections the operating system completes on its own.
• Review findings not acted on. A method on TransportFailure to rebuild it with a new reason, which only Secrets would call. One shared way of writing a number of seconds: the three that exist are a line each, in modules that cannot see one another. A shorter whole-request limit in the test of a status line that trickles in, which would save under a second at the price of the margin a busy build machine needs.

Open questions

None.

---

• Targets master
• English throughout, including comments and test names
• The hard rules in AGENTS.md hold
• Nothing from "Out of scope" in docs/DESIGN.md without explicit approval

🤖 Generated with Claude Code