[ssegura]

Related: None

What you can do now that you could not before

Every fault a run reports is now an error — the API fell over, never answered, or accepted what its own document forbids — or a warning — the API answered something its document does not describe, where the document may be the one out of date. Each kind has a title in plain words, and the summary says under each kind what to look at.

A run ends with 1 for errors only. An API whose document leaves out the ways it refuses a request no longer turns a build red. --set checks.failOnWarnings=true makes warnings fail the run too.

Try it yourself

You need only Java and git. The "API" is a stand-in of a few lines, on the HTTP server that comes with Java. It answers every request with 201 and an empty JSON list, while its document declares only 200: every reply is a warning, and nothing is an error.

git fetch origin && git switch feat/m3-10-errors-and-warnings
./mvnw -q install -DskipTests
mkdir -p try
cat > try/Stand.java <<'EOF'
import com.sun.net.httpserver.HttpServer;
import java.net.InetSocketAddress;

/** A stand-in that answers every request with 201 and an empty JSON list. */
public class Stand {
    public static void main(String[] args) throws Exception {
        HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 18766), 0);
        server.createContext("/", exchange -> {
            byte[] body = "[]".getBytes();
            exchange.getResponseHeaders().add("Content-Type", "application/json");
            exchange.sendResponseHeaders(201, body.length);
            exchange.getResponseBody().write(body);
            exchange.close();
        });
        server.start();
    }
}
EOF
cat > try/pets.yaml <<'EOF'
openapi: 3.0.3
info: {title: Pets, version: "1"}
paths:
  /pets:
    get:
      operationId: listPets
      responses:
        "200":
          description: every pet
          content:
            application/json:
              schema: {type: array, items: {type: object}}
EOF
java try/Stand.java &
./restest run try/pets.yaml --url http://127.0.0.1:18766 --budget 10s --seed 1
echo "exit $?"

Real output, from a fresh clone of this branch. Ten warnings are printed whole, then only counted:

RESTest testing Pets at http://127.0.0.1:18766

1 of 1 operations can be tested, seed 1, budget 10s
  what it sends depends on the API's own replies, so the seed alone does not repeat this run; --store keeps what it sent

warning  Undeclared status code  (F920)
         listPets - GET http://127.0.0.1:18766/pets  ->  201
         the API answered 201, which the document does not declare for this operation; it declares 200
         curl -i -X GET 'http://127.0.0.1:18766/pets' -H 'Accept: application/json' -H 'User-Agent: RESTest/2.0'

... (nine more like it)

... more warnings are being found; every one of them is counted in the run's report and in the total below

595353 requests to 1 operations in 10.0s, 3% of it idle
  opening lap: 1 request in 72ms, 1 of 1 operations answered 2xx
  595353 2xx
595353 warnings, on 1 operation, which do not fail the run (checks.failOnWarnings):
  595353 x Undeclared status code  (F920)
        declare the status code for the operation in the document, or stop the API answering it
report written to restest-out/report.json (15.9 KiB)
the run itself was not kept; pass --store to keep every request and reply
exit 0

On master the same run ends with 1, and its fault is printed as F920 A Status Code The Document Does Not Declare.

The same run, asked to fail on warnings:

./restest run try/pets.yaml --url http://127.0.0.1:18766 --budget 10s --seed 1 --set checks.failOnWarnings=true
echo "exit $?"
jq -c '.totals' restest-out/report.json
kill %1

624951 warnings, on 1 operation, which fail the run, as checks.failOnWarnings asks:
  624951 x Undeclared status code  (F920)
        declare the status code for the operation in the document, or stop the API answering it
...
exit 1
{"requests":624951,"operations":1,"faults":624951,"faultKinds":1,"operationsWithFaults":1,"errors":0,"warnings":624951,"operationsWithErrors":0,"operationsWithWarnings":1,"faultsWrittenInFull":5,"faultsCountedOnly":624946,"elapsed":"PT10.00061S"}

What each kind is, its severity, its title and its advice are in docs/faults.md, under Errors and warnings.

How it works

One rule decides the severity. An error is the API failing at its job: it falls over, never answers, or accepts what its own document forbids. A warning is the API answering something its document does not describe, where the document may be the one out of date.

Code	Title	Severity
F100	Internal server error	error
F121	Other server error	error
F206	Invalid data accepted	error
F312	No reply in time	error
F200	Reply off schema	warning
F205	Not implemented	warning
F920	Undeclared status code	warning
F921	Undeclared content type	warning

Each kind's severity, title and advice line are in one table, in one place. The screen, report.json and the exit code all read that table.

On the screen a fault starts with its severity, then its title, then the catalogue's number in brackets.

• The title is also the name of the switch that turns the check off: Reply off schema is checks.replyOffSchema.
• The summary counts the errors first, then the warnings, each on how many operations, and says whether the warnings failed the run.
• report.json writes severity and title beside the number in every row that counts faults, and advice once for each kind. It also counts errors and warnings apart in totals.
• The catalogue's own names stay in report.json, for comparing tools.

The screen keeps room for errors. It prints at most fifty faults, as before, and now at most ten of those may be warnings. Before, a document that leaves out its refusals filled all fifty places with warnings from the first requests on, and one API's 10,902 replies of 500 never reached the screen.

What changed in the code

restest-core

• New oracle.Severity (ERROR, WARNING).
• New oracle.FaultReading: one table, keyed by the kind's number, giving each kind RESTest reports its severity, title and advice.
  • A kind the table does not know — one somebody else's rule reports — is an error under its own name, or under its number if it has none.
  • FaultCategory, WfcFault, RestestFault, Finding and the stored run are unchanged.
• CheckSettings gains failOnWarnings, last in the group.
• ReportSettings gains warningsShownOnTheConsole.
• SettingKey and Settings know both, which makes 99 settings.

restest-report

• ConsoleReport: the new fault block; the allowance per severity, said once each; and the summary in two sections.
  • It keeps one record per kind found, whose reading is worked out once.
  • It answers errors() and warnings() in place of faults().
  • The factory that took no check settings is gone, so that the screen cannot contradict the exit code.
• JsonReport: severity and title in every fault row; advice in faultsByCategory; four new members in totals.

restest-cli

• ExitCode: NO_FAULTS/FAULTS_FOUND become NO_ERRORS/ERRORS_FOUND, still 0 and 1. of(...) takes errors, warnings and whether warnings fail. Every earlier question keeps its order.
• The help's text for 0 and 1 changes. RunCommand passes the setting to the console and the counts to ExitCode.

Decision records

• New ADR-0037, Errors and warnings.
• Amendments to ADR-0015: the first change to what an exit code means in 2.x, why, and what it costs.
• Amendments to ADR-0016: the screen leads with RESTest's words; severity is a third classification.
• Amendments to ADR-0014: why a reply not checked still never fails a run.
• Amendments to ADR-0025: the two settings.

Documentation

• docs/faults.md: a section Errors and warnings, and new columns in the table of kinds. The headings are renamed to the titles, and every link to them is fixed.
• Other reference pages: report.md, settings.md (template regenerated), switches.md ("Part of" uses the titles), command-line.md (the rows for 0 and 1, and the exception in What frozen means) and known-limitations.md.
• The manual: chapters 1, 4, 9 and 11; README.md's exit-code paragraph; the docs index; DESIGN.md.
• Example output is from real runs of this branch.

Tests

• New: FaultReadingTest.
• Extended:
  • ConsoleReportTest, JsonReportTest, ExitCodeTest, SettingsTest;
  • CheckSwitchesCommandTest: a run whose only faults are F200 ends with 0, and with 1 when asked;
  • DocumentedReportTest: the faults page's two tables are held to FaultReading;
  • DocumentedSwitchesTest: each check's title, written as one word, is its switch, and is what the check is "part of".
• Changed because the meaning changed:
  • AReplyUnreadIsNotUndeclaredTest: its two runs, whose only faults are F920, end with 0, no longer 1;
  • WholeRunTest and CheckSwitchesCommandTest: they now look for the new titles.

Left for later

• 2.2.0's release pull request owes the whole-run walkthroughs. Until then, the README's first run and the manual's chapter 3 still show the old screen (F100 HTTP Status 500, <n> faults:). That pull request also owes a line in the release notes.
• A level between error and warning, when the oracles that need one arrive.
• Raising or lowering a kind on one operation. It will need the severity decided per fault, in both reports.
• Colours on the terminal.
• A verdict member in report.json.

Evidence

• Tests: 32 new test methods, none removed. ./mvnw verify is green on all nine modules, 4,648 tests (4,594 before). CI: added below once it has run.

• Architecture tests: two added (the faults page's tables against FaultReading, and each check's title against its switch); none weakened.

• Mutation score, restest-oracles: 97% (294 of 304), as on master; its code is unchanged.

• Per-request overhead: measured on swagger-petstore, restcountries and rest-scs. Three one-minute runs of master and of this branch, alternated, with the same seed for each pair, on mains power.

  • Every fault now costs one look-up of its kind; the request loop is not touched.
  • On average this branch sent 2.3% more requests on one API and 1.6% and 2.2% fewer on the other two.
  • The nine pairs differed by −20% to +23%: four up, five down. Three runs do not tell that apart from how much two runs of one build differ.

  API	Requests, master	Requests, this branch	Idle, both
  swagger-petstore	562,358; 449,424; 495,508	448,049; 551,816; 541,534	0.7–1.0%
  restcountries	246,500; 239,692; 255,614	245,641; 248,808; 235,570	0.5–0.6%
  rest-scs	305,983; 323,948; 297,693	297,241; 312,396; 297,937	0.5–0.6%

• Smoke run, two containerised APIs: the same two APIs run as the smoke job runs them (10 s, seed 20260914, no key), three runs of master and of this branch, alternated.

  • Pet shop. Both: 19 operations tested, 14–16 answering 500, 3 kinds of fault, 16–17 operations with faults. master 33,569–36,374 faults; this branch 30,899–36,300, of which 25,525–31,711 errors and 4,589–5,374 warnings.
  • Activity catalogue. Both: 49 operations, 3 answering 500, 3 kinds, 30 operations with faults. master 35,767–41,317 faults; this branch 37,649–41,783, of which 2,557–2,827 errors and 35,092–38,956 warnings.
  • Every run ended with 1, on both builds.

• Idle time: 3.9–4.5% in the smoke runs on both builds; 0.5–1.0% in the one-minute runs.

• Artefact: the fifteen APIs of the Web Fuzzing Dataset used for the earlier rules, run locally with this branch, a minute each, seed 42.

  • A replay of their stored runs predicted, before any code was written, that only bibliothek would change its exit code. It did: from 1 to 0, because all 622,908 of its faults are warnings.
  • On all fifteen, errors and warnings add up to the faults counted.
  • On spring-rest-example, whose 10,902 replies of 500 never reached the screen before, the screen printed ten warnings and then forty errors, 36 of them replies of 500.

  API	Exit code, master	Exit code, this branch	Errors, on operations	Warnings, on operations
  bibliothek	1	0	0 on 0	622,908 on 7
  catwatch	1	1	288 on 12	85 on 7
  features-service	1	1	26,046 on 11	3,066 on 5
  gestaohospital	1	1	10,649 on 4	112,583 on 19
  kafka-rest	1	1	30 on 10	508 on 39
  market	1	1	774 on 9	683 on 7
  petclinic-rest	1	1	26,787 on 30	71,001 on 39
  rest-ncs	1	1	10,523 on 6	302,211 on 6
  rest-news	1	1	305 on 1	285,297 on 6
  rest-scs	1	1	5,532 on 3	28,489 on 11
  restcountries	1	1	4,912 on 3	0 on 0
  scout-api	1	1	23,180 on 23	65,912 on 16
  spring-rest-example	1	1	12,976 on 2	309,487 on 9
  swagger-petstore	1	1	247,271 on 16	27,559 on 8
  user-management	1	1	880 on 12	26,871 on 21

  • Pet-clinic in Docker (springcommunity/spring-petclinic-rest:4.0.2, as the manual starts it), 30 seconds:
    • with the four checks that report errors switched off, only warnings were left (F920, F921, F200), and the run ended with 0;
    • the same with --set checks.failOnWarnings=true ended with 1, saying so under the warnings.

Decisions taken

Decided with the maintainer before any code was written:

• Two severities, by the rule above. F206 is an error, because accepting what the document forbids is the API failing at a job its document gives it. F205 is a warning. A third level, F206 as a "high" warning, was weighed and kept for when the oracles that need it arrive.
• Warnings fail a run only when asked, with a setting rather than a command-line option.
• A reply no check could judge never fails a run, not even with checks.failOnWarnings.
• The severity word comes first on the screen, then the title, then the code in brackets.
• Severity is fixed per kind.
• One name per kind everywhere: the switch's name in words. So F200 is Reply off schema, and the switches page and the faults page's headings use the same words.
• The screen keeps report.faultsShownOnTheConsole's meaning, and adds report.warningsShownOnTheConsole.
• This breaks ADR-0015's freeze once, deliberately, as the maintainer chose: 0 meant no fault and means no error. Against 2.1.0, a run whose only faults are F200 ended with 1 and ends with 0; with --set checks.failOnWarnings=true it ends with 1 again. ADR-0015's amendment says so, and so does docs/command-line.md.

Taken while building it:

• The help's text for 0 ends "…counted in its summary and in its report." It does not end "…in report.json." because the command-line framework breaks long lines after a full stop and printed "report. json". DocumentedCommandLineTest caught it.
• What to look at is said once per kind: in the summary and in faultsByCategory. It is not repeated under each of up to a thousand findings written out.
• The examples have to show what reaches the screen.
  • The faults page's F205 example comes from a run of an API with checks.undeclaredStatusCode off.
  • The manual's F200 example comes from a pet-clinic run with both undeclared checks off.
  • The reason in both cases: otherwise the first ten warnings are other kinds. The text says so.
  • The F312 example now comes from kafka-rest-proxy: catwatch's F312s did not reach the screen in this run.
• FaultReading.isKnown was removed in simplification: it was public and used only by tests.

Findings not acted on:

• Two tests take checks.failOnWarnings out of "the checks" by name.
  • CheckSwitchesCommandTest does it for the run with every check off, DocumentedSwitchesTest for counting the checks' rows.
  • A general "is this a check" in the production code would be API used only by tests.
• DocumentedSwitchesTest.kindNumbered looks a kind up by number next to WfcFault.byCode. Sharing DocumentedReportTest's map would mean widening a private helper, for little gain.

Open questions

• Should a reply off its schema stay a warning? It is one here, as planned; making it an error is one line in FaultReading, and the exit codes above would not change, since every API with an F200 also answered 500.

---

• Targets master
• English throughout, including comments and test names
• The hard rules in AGENTS.md hold
• Nothing from "Out of scope" in docs/DESIGN.md without explicit approval

🤖 Generated with Claude Code