From ba134013382d2f0e707fa1139aa1731ca3d6cc65 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 25 Sep 2026 17:49:36 +0800 Subject: [PATCH 01/43] docs: point progress.md at the workspace items still open --- progress.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/progress.md b/progress.md index 65519d9..14320f6 100644 --- a/progress.md +++ b/progress.md @@ -2,7 +2,7 @@ Outstanding work only. When an item is done, delete it in the same commit and add a `#done` entry to `docs/updates/` (format and query commands: `docs/updates/README.md`). No finished items, no history, no rules (rules live in `CLAUDE.md`). Item numbers (`#n`) are never reused. Tags: [DECIDE] needs the owner's decision, [BLOCKED] waits on something else, [UNVERIFIED] observed but not confirmed. -Cross-repo and workspace items live in `D:\Codes\progress.md` (relevant here: X-1, X-12, X-16). +Cross-repo and workspace items live in `D:\Codes\progress.md` (relevant here: X-12, X-13). ## Open From 08ad36630ad53a1235036c6a395051c1ef773272 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 25 Sep 2026 18:09:20 +0800 Subject: [PATCH 02/43] Wait 7 days before Dependabot proposes a new release --- .github/dependabot.yml | 7 +++++++ docs/updates/2026-09.md | 9 +++++++++ docs/updates/README.md | 3 ++- test/test_workflow_actions.py | 10 ++++++++++ 4 files changed, 28 insertions(+), 1 deletion(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 8058307..201868a 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -12,6 +12,11 @@ updates: target-branch: "dev" schedule: interval: "daily" + # A new release waits 7 days before Dependabot proposes it, so a + # compromised version is more likely to be found and yanked first. + # Dependabot's own default is 3 days. Security updates are not delayed. + cooldown: + default-days: 7 # Workflow actions are pinned to commit SHAs with the version as a comment # (test_workflow_actions.py); Dependabot moves both together. - package-ecosystem: "github-actions" @@ -19,3 +24,5 @@ updates: target-branch: "dev" schedule: interval: "weekly" + cooldown: + default-days: 7 diff --git a/docs/updates/2026-09.md b/docs/updates/2026-09.md index 9c02b0d..b7bb359 100644 --- a/docs/updates/2026-09.md +++ b/docs/updates/2026-09.md @@ -192,3 +192,12 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Tests**: `test/test_workflow_actions.py` gains a check that every checkout step sets `persist-credentials` explicitly. The previous workflows fail it. - **Result / numbers**: zizmor reports no warnings or errors for these workflows any more; the test file passes. - **Files**: `.github/workflows/ci.yml`, `.github/workflows/publish.yml`, `test/test_workflow_actions.py`. + +## U-20260925-01 · 2026-09-25 · Dependabot waits 7 days before proposing a new release · #ci #security #deps + +- **What**: `.github/dependabot.yml` sets `cooldown: default-days: 7` on all 2 update blocks. Dependabot now proposes a version only after it has been public for 7 days, so a compromised release is more likely to be found and yanked before a PR exists. Without the setting Dependabot waits 3 days (its default since 2026-07-14). Security updates ignore the cooldown and still open at once. `github-actions` accepts only `default-days`, so no per-semver values are set. +- **Why now**: zizmor 1.30.1 (`zizmor --offline .github`) reported `dependabot-cooldown` (medium) on every block; the workflow scan earlier this week only covered `.github/workflows`. +- **Tests**: `test/test_workflow_actions.py` checks that every update block waits at least 7 days. The previous config fails it. +- **Result / numbers**: zizmor medium findings 0. The workflow-actions test file passes. +- **Sources**: https://docs.zizmor.sh/audits/#dependabot-cooldown ; https://docs.github.com/en/code-security/dependabot/working-with-dependabot/dependabot-options-reference#cooldown- ; https://github.blog/changelog/2026-07-14-dependabot-version-updates-introduce-default-package-cooldown/ +- **Files**: `.github/dependabot.yml`, `test/test_workflow_actions.py`. diff --git a/docs/updates/README.md b/docs/updates/README.md index d23a243..4fdddca 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20260925-01 | 2026-09-25 | Dependabot waits 7 days before proposing a new release | #ci #security #deps | [2026-09](2026-09.md) | | U-20260924-02 | 2026-09-24 | Keep checkout credentials only in the job that pushes | #ci #security | [2026-09](2026-09.md) | | U-20260924-01 | 2026-09-24 | Move CI to Node 24 actions pinned by commit | #ci #security #deps | [2026-09](2026-09.md) | | U-20260923-14 | 2026-09-23 | MCP server runs on mcp 1.x and 2.x; extra allows <3 | #done #mcp #compat | [2026-09](2026-09.md) | @@ -83,4 +84,4 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-09.md](2026-09.md) | 2026-09 | 20 | +| [2026-09.md](2026-09.md) | 2026-09 | 21 | diff --git a/test/test_workflow_actions.py b/test/test_workflow_actions.py index 46b9b4b..6e0190c 100644 --- a/test/test_workflow_actions.py +++ b/test/test_workflow_actions.py @@ -65,6 +65,16 @@ def test_dependabot_keeps_pins_current_on_dev(): for block in blocks) +def test_dependabot_waits_a_week_before_proposing_a_release(): + # A compromised release is usually found and yanked within days. Dependabot's + # own default wait is 3 days, and zizmor's dependabot-cooldown audit asks + # for 7. The wait never delays security updates. + text = (_ROOT / ".github" / "dependabot.yml").read_text(encoding="utf-8") + blocks = re.split(r"^\s*-\s*package-ecosystem:", text, flags=re.MULTILINE)[1:] + days = [re.search(r"^\s*default-days:\s*(\d+)", block, re.MULTILINE) for block in blocks] + assert blocks and all(match and int(match.group(1)) >= 7 for match in days) + + def _checkout_steps(path: Path) -> list[tuple[int, str]]: """Return ``(line number, step text)`` for each ``actions/checkout`` step.""" lines = path.read_text(encoding="utf-8").splitlines() From 28ede7372c86a294f10b4e8bd0969087a5478c1c Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 25 Sep 2026 18:29:11 +0800 Subject: [PATCH 03/43] docs: CLAUDE.md: look up SonarCloud/Codacy findings through their APIs with the env keys; never reveal a key --- CLAUDE.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CLAUDE.md b/CLAUDE.md index 2214123..311d14c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -211,6 +211,7 @@ Workspace rule shared by every repository under `D:\Codes` (full text: `D:\Codes - Stage only the files that stage touched (`git add `, never `git add -A`), follow this file's commit-message rules, and never add AI attribution. - Committing is not pushing: push or open a PR only as this project's branch flow says or when asked. - **Commit and push frequently.** After each big feature — a self-contained stage that passes this project's checks — commit and push to the remote; do not pile up a large batch of work before committing or pushing. Smaller batches collide less with other sessions, let CI catch problems earlier, and are easier to revert. Follow this project's normal branch flow (usually `dev`). + - **SonarCloud / Codacy findings.** When a PR or commit fails a SonarCloud or Codacy check, look the findings up through their APIs instead of guessing. The keys are in environment variables: `SonarCloudToken` (SonarCloud, e.g. `curl -s -u "$SonarCloudToken:" "https://sonarcloud.io/api/issues/search?componentKeys=&pullRequest=&resolved=false"`) and `CODACY_PROJECT_TOKEN` (Codacy; for a public repository `https://app.codacy.com/api/v3/analysis/organizations/gh//repositories//pull-requests//issues?status=new` also answers without a key). **Never reveal a key or any personal credential while doing so**: refer to the variables by name only, never echo or print their values, and never put them in files, commit messages, PR or issue text, logs, or any output that leaves the machine. - **`progress.md`** (repository root, tracked) holds outstanding work only: no finished items, no history, no rules. - **`docs/updates/`** records finished work: one batch file per month (`YYYY-MM.md`), one entry per piece of work headed `## U-YYYYMMDD-NN · date · title · #tags`, and an index with query commands in `docs/updates/README.md`. When a `progress.md` item is done, delete it and add a `#done` entry plus its index row in the same commit. - **`architecture.md`** (repository root) is the short architecture overview: layers, entry points, main flows, extension points, cross-project boundaries. Update it in the same commit whenever a change alters any of those. From b6bd677db3e4072b6dbc8f260f9130d0d80a8295 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 25 Sep 2026 18:32:27 +0800 Subject: [PATCH 04/43] docs: CLAUDE.md: the Codacy project token only works for its own project; query public repos without it --- CLAUDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index 311d14c..d092b44 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -211,7 +211,7 @@ Workspace rule shared by every repository under `D:\Codes` (full text: `D:\Codes - Stage only the files that stage touched (`git add `, never `git add -A`), follow this file's commit-message rules, and never add AI attribution. - Committing is not pushing: push or open a PR only as this project's branch flow says or when asked. - **Commit and push frequently.** After each big feature — a self-contained stage that passes this project's checks — commit and push to the remote; do not pile up a large batch of work before committing or pushing. Smaller batches collide less with other sessions, let CI catch problems earlier, and are easier to revert. Follow this project's normal branch flow (usually `dev`). - - **SonarCloud / Codacy findings.** When a PR or commit fails a SonarCloud or Codacy check, look the findings up through their APIs instead of guessing. The keys are in environment variables: `SonarCloudToken` (SonarCloud, e.g. `curl -s -u "$SonarCloudToken:" "https://sonarcloud.io/api/issues/search?componentKeys=&pullRequest=&resolved=false"`) and `CODACY_PROJECT_TOKEN` (Codacy; for a public repository `https://app.codacy.com/api/v3/analysis/organizations/gh//repositories//pull-requests//issues?status=new` also answers without a key). **Never reveal a key or any personal credential while doing so**: refer to the variables by name only, never echo or print their values, and never put them in files, commit messages, PR or issue text, logs, or any output that leaves the machine. + - **SonarCloud / Codacy findings.** When a PR or commit fails a SonarCloud or Codacy check, look the findings up through their APIs instead of guessing. The keys are in environment variables: `SonarCloudToken` (SonarCloud, e.g. `curl -s -u "$SonarCloudToken:" "https://sonarcloud.io/api/issues/search?componentKeys=&pullRequest=&resolved=false"`) and `CODACY_PROJECT_TOKEN` (a Codacy project token, valid only for its own project: any other repository answers "Bad credentials", so for a public repository query `https://app.codacy.com/api/v3/analysis/organizations/gh//repositories//pull-requests//issues?status=new` without a key). **Never reveal a key or any personal credential while doing so**: refer to the variables by name only, never echo or print their values, and never put them in files, commit messages, PR or issue text, logs, or any output that leaves the machine. - **`progress.md`** (repository root, tracked) holds outstanding work only: no finished items, no history, no rules. - **`docs/updates/`** records finished work: one batch file per month (`YYYY-MM.md`), one entry per piece of work headed `## U-YYYYMMDD-NN · date · title · #tags`, and an index with query commands in `docs/updates/README.md`. When a `progress.md` item is done, delete it and add a `#done` entry plus its index row in the same commit. - **`architecture.md`** (repository root) is the short architecture overview: layers, entry points, main flows, extension points, cross-project boundaries. Update it in the same commit whenever a change alters any of those. From ad809406345dd522d08fdd2a2058bf9b7f0f25c6 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 25 Sep 2026 19:03:45 +0800 Subject: [PATCH 05/43] Pin the dev environment to the package's PySide6 6.11.2 --- dev_requirements.txt | 2 +- docs/updates/2026-09.md | 7 ++++ docs/updates/README.md | 3 +- test/test_utils/test_requirement_pins.py | 51 ++++++++++++++++++++++++ 4 files changed, 61 insertions(+), 2 deletions(-) create mode 100644 test/test_utils/test_requirement_pins.py diff --git a/dev_requirements.txt b/dev_requirements.txt index 8ff5dae..85f5ba8 100644 --- a/dev_requirements.txt +++ b/dev_requirements.txt @@ -11,5 +11,5 @@ sphinx-rtd-theme pytest pytest-asyncio pytest-timeout -Pyside6 == 6.11.0 +PySide6==6.11.2 qt_material \ No newline at end of file diff --git a/docs/updates/2026-09.md b/docs/updates/2026-09.md index b7bb359..58093db 100644 --- a/docs/updates/2026-09.md +++ b/docs/updates/2026-09.md @@ -201,3 +201,10 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Result / numbers**: zizmor medium findings 0. The workflow-actions test file passes. - **Sources**: https://docs.zizmor.sh/audits/#dependabot-cooldown ; https://docs.github.com/en/code-security/dependabot/working-with-dependabot/dependabot-options-reference#cooldown- ; https://github.blog/changelog/2026-07-14-dependabot-version-updates-introduce-default-package-cooldown/ - **Files**: `.github/dependabot.yml`, `test/test_workflow_actions.py`. + +## U-20260925-02 · 2026-09-25 · dev_requirements.txt pins the package's PySide6 · #deps #tests + +- **What**: `dev_requirements.txt` still pinned `Pyside6 == 6.11.0` after the `gui` extra in `pyproject.toml` (and `dev.toml`, `.github/requirements/ci.in`) moved to `PySide6==6.11.2` on 2026-09-23, so a local dev environment ran a different Qt than users install. It now pins `PySide6==6.11.2`. +- **Tests**: new `test/test_utils/test_requirement_pins.py`: every `name==version` pin in `requirements.txt` or `dev_requirements.txt` that `pyproject.toml` also pins with `==` (dependencies and every extra) must name the same version. Names are compared in their PEP 503 form, so `Pyside6` and `PySide6` are one. The old file fails it. `tomllib` needs 3.11, so it skips on 3.10 like `test_dev_toml_parity.py`. +- **Result / numbers**: `python -m pytest test/` 411 passed, 4 skipped. +- **Files**: `dev_requirements.txt`, `test/test_utils/test_requirement_pins.py` (new). diff --git a/docs/updates/README.md b/docs/updates/README.md index 4fdddca..4d416cf 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20260925-02 | 2026-09-25 | dev_requirements.txt pins the package's PySide6 | #deps #tests | [2026-09](2026-09.md) | | U-20260925-01 | 2026-09-25 | Dependabot waits 7 days before proposing a new release | #ci #security #deps | [2026-09](2026-09.md) | | U-20260924-02 | 2026-09-24 | Keep checkout credentials only in the job that pushes | #ci #security | [2026-09](2026-09.md) | | U-20260924-01 | 2026-09-24 | Move CI to Node 24 actions pinned by commit | #ci #security #deps | [2026-09](2026-09.md) | @@ -84,4 +85,4 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-09.md](2026-09.md) | 2026-09 | 21 | +| [2026-09.md](2026-09.md) | 2026-09 | 22 | diff --git a/test/test_utils/test_requirement_pins.py b/test/test_utils/test_requirement_pins.py new file mode 100644 index 0000000..c3610b4 --- /dev/null +++ b/test/test_utils/test_requirement_pins.py @@ -0,0 +1,51 @@ +"""Versions pinned with ``==`` in the requirements files match the ones the package pins. + +``requirements.txt`` and ``dev_requirements.txt`` set up a working copy, while +``pyproject.toml`` is what users install. The two drifted: ``dev_requirements.txt`` +kept an older PySide6 after ``pyproject.toml`` moved to 6.11.2, so a developer +tested against a different Qt than the release ships. For example +``_pins(["PySide6==6.11.2"])`` is ``{"pyside6": "6.11.2"}``. +""" +from __future__ import annotations + +import re +from pathlib import Path + +import pytest + +tomllib = pytest.importorskip("tomllib") # stdlib from 3.11; CI also runs 3.10 + +REPO_ROOT = Path(__file__).resolve().parents[2] +_PIN = re.compile(r"^\s*([A-Za-z0-9._-]+)\s*==\s*([^\s;#,]+)") + + +def _normalise(name: str) -> str: + """Return the PEP 503 form of a distribution name (``Pyside6`` and ``PySide6`` are one).""" + return re.sub(r"[-_.]+", "-", name).lower() + + +def _pins(requirements: list[str]) -> dict[str, str]: + """Map each ``name==version`` requirement to its version, keyed by normalised name.""" + return {_normalise(match.group(1)): match.group(2) + for match in (_PIN.match(line) for line in requirements) if match} + + +def _package_requirements() -> list[str]: + """Return the dependencies and every optional-dependency group of ``pyproject.toml``.""" + with (REPO_ROOT / "pyproject.toml").open("rb") as handle: + project = tomllib.load(handle)["project"] + requirements = list(project.get("dependencies", [])) + for group in project.get("optional-dependencies", {}).values(): + requirements.extend(group) + return requirements + + +@pytest.mark.parametrize("name", ["requirements.txt", "dev_requirements.txt"]) +def test_requirements_file_pins_what_the_package_pins(name): + path = REPO_ROOT / name + if not path.is_file(): + pytest.skip(f"{name} does not exist") + local = _pins(path.read_text(encoding="utf-8").splitlines()) + package = _pins(_package_requirements()) + assert {dist: (local[dist], package[dist]) for dist in local.keys() & package.keys() + if local[dist] != package[dist]} == {} From e7fbbf13c0605b63adf4710da71a75a7e4885feb Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 25 Sep 2026 19:46:03 +0800 Subject: [PATCH 06/43] List Python 3.11 to 3.14 in the classifiers, as CI tests them --- dev.toml | 4 +++ docs/updates/2026-09.md | 6 ++++ docs/updates/README.md | 3 +- pyproject.toml | 4 +++ test/test_utils/test_python_classifiers.py | 35 ++++++++++++++++++++++ 5 files changed, 51 insertions(+), 1 deletion(-) create mode 100644 test/test_utils/test_python_classifiers.py diff --git a/dev.toml b/dev.toml index 12e7039..6b0a7d4 100644 --- a/dev.toml +++ b/dev.toml @@ -18,6 +18,10 @@ dependencies = [ ] classifiers = [ "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Programming Language :: Python :: 3.14", "Development Status :: 2 - Pre-Alpha", "Environment :: Win32 (MS Windows)", "Environment :: MacOS X", diff --git a/docs/updates/2026-09.md b/docs/updates/2026-09.md index 58093db..20e0541 100644 --- a/docs/updates/2026-09.md +++ b/docs/updates/2026-09.md @@ -208,3 +208,9 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Tests**: new `test/test_utils/test_requirement_pins.py`: every `name==version` pin in `requirements.txt` or `dev_requirements.txt` that `pyproject.toml` also pins with `==` (dependencies and every extra) must name the same version. Names are compared in their PEP 503 form, so `Pyside6` and `PySide6` are one. The old file fails it. `tomllib` needs 3.11, so it skips on 3.10 like `test_dev_toml_parity.py`. - **Result / numbers**: `python -m pytest test/` 411 passed, 4 skipped. - **Files**: `dev_requirements.txt`, `test/test_utils/test_requirement_pins.py` (new). + +## U-20260925-03 · 2026-09-25 · Python classifiers list every version CI tests · #packaging #tests + +- **What**: the package metadata (`pyproject.toml`, `dev.toml`) listed only `Programming Language :: Python :: 3.10`, while CI runs every test on 3.10, 3.11, 3.12, 3.13 and 3.14, so PyPI showed the package as 3.10-only. The classifiers for 3.11 to 3.14 are added to both files. +- **Tests**: new `test/test_utils/test_python_classifiers.py` checks that each metadata file's Python classifiers are exactly the versions in the workflows' `python-version` matrices, so adding a version to CI without the classifier (or the other way round) fails. The old metadata fails it. +- **Files**: `pyproject.toml`, `dev.toml`, `test/test_utils/test_python_classifiers.py` (new). diff --git a/docs/updates/README.md b/docs/updates/README.md index 4d416cf..10ae024 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20260925-03 | 2026-09-25 | Python classifiers list every version CI tests | #packaging #tests | [2026-09](2026-09.md) | | U-20260925-02 | 2026-09-25 | dev_requirements.txt pins the package's PySide6 | #deps #tests | [2026-09](2026-09.md) | | U-20260925-01 | 2026-09-25 | Dependabot waits 7 days before proposing a new release | #ci #security #deps | [2026-09](2026-09.md) | | U-20260924-02 | 2026-09-24 | Keep checkout credentials only in the job that pushes | #ci #security | [2026-09](2026-09.md) | @@ -85,4 +86,4 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-09.md](2026-09.md) | 2026-09 | 22 | +| [2026-09.md](2026-09.md) | 2026-09 | 23 | diff --git a/pyproject.toml b/pyproject.toml index 97faffa..c522cee 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -18,6 +18,10 @@ dependencies = [ ] classifiers = [ "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Programming Language :: Python :: 3.14", "Development Status :: 2 - Pre-Alpha", "Environment :: Win32 (MS Windows)", "Environment :: MacOS X", diff --git a/test/test_utils/test_python_classifiers.py b/test/test_utils/test_python_classifiers.py new file mode 100644 index 0000000..d9baa75 --- /dev/null +++ b/test/test_utils/test_python_classifiers.py @@ -0,0 +1,35 @@ +"""The PyPI classifiers name exactly the Python versions CI tests. + +The package metadata listed only 3.10 while CI ran 3.10 to 3.14, so PyPI showed +the package as 3.10-only. A workflow matrix ``python-version: ["3.10", "3.11"]`` +needs ``Programming Language :: Python :: 3.10`` and ``:: 3.11``, nothing more. +""" +from __future__ import annotations + +import re +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[2] +_MATRIX = re.compile(r"python-version:\s*\[([^\]]*)\]") +_CLASSIFIER = re.compile(r"Programming Language :: Python :: (3\.\d+)") + + +def _tested_versions() -> set[str]: + """Return every Python version named in a workflow's ``python-version`` matrix.""" + versions: set[str] = set() + for workflow in (REPO_ROOT / ".github" / "workflows").glob("*.yml"): + for match in _MATRIX.finditer(workflow.read_text(encoding="utf-8")): + versions.update(re.findall(r"3\.\d+", match.group(1))) + return versions + + +def test_ci_has_a_python_matrix(): + assert _tested_versions() + + +@pytest.mark.parametrize("metadata", ["pyproject.toml", "dev.toml"]) +def test_classifiers_match_the_ci_matrix(metadata): + declared = set(_CLASSIFIER.findall((REPO_ROOT / metadata).read_text(encoding="utf-8"))) + assert declared == _tested_versions() From 10dd122d0f00dfd7f29f598818c29acfd03d0e91 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 00:25:23 +0800 Subject: [PATCH 07/43] docs: progress.md: record the eight planned features (#6-#13) --- progress.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/progress.md b/progress.md index 14320f6..cedb50c 100644 --- a/progress.md +++ b/progress.md @@ -7,3 +7,11 @@ Cross-repo and workspace items live in `D:\Codes\progress.md` (relevant here: X- ## Open - **#3** [DECIDE] The PyPI classifier still says `Development Status :: 2 - Pre-Alpha`. +- **#6** GUI redesign: restructure and restyle. `gui/main_widget.py` is one 605-line widget (over the 500-line limit) with six flat tabs. It should become a sidebar-navigated window with a request/response split view, and history and environment panels docked (`gui/history_panel.py`, `gui/env_manager_model.py` exist but are not wired in). It needs a refreshed theme (`gui/main_window.py:21` hard-codes `dark_amber.xml`), pages for the newer features (#8–#13), and all four language wrappers (English, Traditional Chinese, Simplified Chinese, Japanese). `APITestkaWidget` stays the embeddable entry point (PyBreeze contract, `architecture.md` §6). +- **#7** `records_to_openapi` has no CLI subcommand (`cli/cli_main.py:101` `build_parser`). The MCP tool (`mcp_server/tool_definitions.py:74`) reads only the in-memory record, so a fresh MCP process gets an empty spec. Neither can read a saved JSON report or write a file. `spec/records_to_openapi.py:48` also labels a body-less response `application/json` → `text/plain`, and the last record per operation overwrites earlier status codes. +- **#8** Mock server: gRPC and WebSocket endpoints. `utils/mock_server/grpc_stub.py` serves unary calls only and is not reachable from `apitestka mock`, the executor or `flask_mock_server_instance`. There is no WebSocket mock endpoint. +- **#9** Pact-style bidirectional contract testing. `diff/contract_diff.py` only compares two OpenAPI documents. Missing: a consumer contract recorded from the consumer's tests (interactions plus matchers), provider verification replaying it against a running provider, and a cross-check of the consumer contract against the provider's OpenAPI. +- **#10** Reference AI backend on the Anthropic API. `ai/backend.py:1` ships only `NoOpAIBackend` and `StaticAIBackend`. Missing: an optional `ai` extra, an `AnthropicAIBackend` (key from the environment, never stored), and a way to select it (CLI, executor). +- **#11** Load-test bridge to LoadDensity. No way to turn APITestka actions into a LoadDensity run. It needs a converter to LoadDensity's action JSON (`LD_start_test` with `tasks`) and a runner that uses LoadDensity's `--execute_file` CLI. That is a new cross-project contract, so it also needs §6 in both repos and the root `AI_CONTEXT.md`. +- **#12** Response-time trends and anomaly detection. `utils/generate_report/trend_store.py:13` stores only run totals (success, failure, average elapsed). Missing: per-endpoint latency history, a baseline (median/MAD or EWMA), anomaly flags, and a trend report. +- **#13** Test-as-spec full loop. The pieces exist separately (`cli/import_specs.py` spec → actions, `spec/records_to_openapi.py` records → spec, `diff/contract_diff.py`, `spec/openapi_changelog.py`), but no single flow runs the tests, infers the spec, checks it against the committed spec, fails on drift, and generates actions for uncovered operations. Depends on #7. From 481074387317f57a1113f64f89085bad84fac3e4 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 00:36:03 +0800 Subject: [PATCH 08/43] feat: infer OpenAPI from saved reports via CLI, MCP and executor Add `apitestka openapi` (--report / --run, -o, --title, --api-version), a report_paths argument on the apitestka_records_to_openapi MCP tool, and AT_export_openapi. spec/openapi_export.py reads the JSON success report back into records. records_to_openapi now leaves body-less responses without content instead of a malformed application/json entry, keeps one response per status code instead of replacing the operation, and records query parameters and request bodies. --- README.md | 18 +++- README/README_zh-CN.md | 17 ++- README/README_zh-TW.md | 17 ++- architecture.md | 5 +- docs/source/Eng/doc/cli/cli_doc.rst | 48 +++++++++ docs/source/Eng/doc/diff/diff_doc.rst | 19 ++++ docs/source/Eng/doc/mcp/mcp_doc.rst | 6 +- docs/source/Zh/Zh_index.rst | 2 +- docs/source/Zh/doc/cli/cli_doc.rst | 46 ++++++++ docs/source/Zh/doc/diff/diff_doc.rst | 18 ++++ docs/source/Zh/doc/mcp/mcp_doc.rst | 2 +- docs/updates/2026-10.md | 15 +++ docs/updates/README.md | 2 + je_api_testka/cli/cli_main.py | 44 +++++++- je_api_testka/cli/completion.py | 4 +- je_api_testka/mcp_server/tool_definitions.py | 23 +++- je_api_testka/spec/__init__.py | 4 + je_api_testka/spec/openapi_export.py | 88 +++++++++++++++ je_api_testka/spec/records_to_openapi.py | 102 ++++++++++++------ .../utils/executor/action_executor.py | 2 + progress.md | 1 - test/test_cli/test_cli_main.py | 42 ++++++++ test/test_cli/test_completion.py | 7 ++ test/test_executor_integration.py | 2 +- test/test_mcp_server/test_tool_definitions.py | 16 +++ test/test_spec/test_openapi_export.py | 88 +++++++++++++++ test/test_spec/test_records_to_openapi.py | 57 ++++++++++ 27 files changed, 639 insertions(+), 56 deletions(-) create mode 100644 docs/updates/2026-10.md create mode 100644 je_api_testka/spec/openapi_export.py create mode 100644 test/test_spec/test_openapi_export.py diff --git a/README.md b/README.md index a868d1c..77dc0e6 100644 --- a/README.md +++ b/README.md @@ -305,6 +305,8 @@ apitestka summary # ANSI-coloured summary apitestka scaffold https://api/x out.json apitestka completion bash # source >> ~/.bashrc apitestka mcp # MCP server over stdio +apitestka openapi --report run_success.json -o openapi.json # infer a spec from a saved run +apitestka openapi --run actions.json # run first, print the inferred spec ``` ### Security Probes @@ -326,12 +328,22 @@ run_pip_audit() # delegates to pip-audit ### OpenAPI Inference ```python -from je_api_testka.spec import infer_schema, records_to_openapi, openapi_changelog +from je_api_testka.spec import ( + infer_schema, records_to_openapi, build_openapi, export_openapi, load_report_records, openapi_changelog, +) -records_to_openapi(title="Recovered", version="0.1.0") +records_to_openapi(title="Recovered", version="0.1.0") # from the in-memory test record +build_openapi(["run_success.json"]) # record plus saved JSON reports +export_openapi("openapi.json", ["run_success.json"]) # same, written as UTF-8 JSON openapi_changelog(prev_spec, current_spec) # markdown diff ``` +Records with the same method and path merge into one operation: every status code seen gets +a response entry, query parameter names become `in: query` parameters, and JSON or text +bodies become response and request-body schemas. A saved report is the +`_success.json` file from `generate_json_report`; the `AT_export_openapi` command and +`apitestka openapi` read it too. + ### GUI ```bash @@ -378,7 +390,7 @@ MCP-compatible client) can drive the framework. Eight tools are exposed: | `apitestka_curl_to_action` | cURL → action JSON | | `apitestka_har_import` | HAR file → action list | | `apitestka_render_markdown` | Markdown report from current records | -| `apitestka_records_to_openapi` | Reconstruct an OpenAPI document | +| `apitestka_records_to_openapi` | Reconstruct an OpenAPI document from the record or saved reports (`report_paths`) | | `apitestka_clear_records` | Wipe the test record | | `apitestka_get_records` | Return successes / failures | diff --git a/README/README_zh-CN.md b/README/README_zh-CN.md index 4a8b37e..a96b3ab 100644 --- a/README/README_zh-CN.md +++ b/README/README_zh-CN.md @@ -302,6 +302,8 @@ apitestka summary # ANSI 彩色摘要 apitestka scaffold https://api/x out.json apitestka completion bash # source >> ~/.bashrc apitestka mcp # 走 stdio 启动 MCP server +apitestka openapi --report run_success.json -o openapi.json # 从保存的运行结果反推规格 +apitestka openapi --run actions.json # 先运行再打印反推的规格 ``` ### 安全检测 @@ -323,12 +325,21 @@ run_pip_audit() ### OpenAPI 反推 ```python -from je_api_testka.spec import infer_schema, records_to_openapi, openapi_changelog +from je_api_testka.spec import ( + infer_schema, records_to_openapi, build_openapi, export_openapi, load_report_records, openapi_changelog, +) -records_to_openapi(title="Recovered", version="0.1.0") +records_to_openapi(title="Recovered", version="0.1.0") # 从内存中的测试记录 +build_openapi(["run_success.json"]) # 测试记录加上保存的 JSON 报告 +export_openapi("openapi.json", ["run_success.json"]) # 同上,并写成 UTF-8 JSON openapi_changelog(prev_spec, current_spec) # markdown changelog ``` +同一个 method 与路径的记录会合并成一个 operation:每个出现过的状态码各有一条 response, +查询参数名称变成 `in: query` 参数,JSON 或文本内容变成响应与请求体的 schema。 +保存的报告是 `generate_json_report` 生成的 `_success.json`;`AT_export_openapi` +命令与 `apitestka openapi` 也能读取它。 + ### GUI ```bash @@ -373,7 +384,7 @@ MCP-compatible client 可以直接驱动本框架。共暴露八个工具: | `apitestka_curl_to_action` | cURL → action JSON | | `apitestka_har_import` | HAR 文件 → action list | | `apitestka_render_markdown` | 从当前记录生成 Markdown 报告 | -| `apitestka_records_to_openapi` | 反推 OpenAPI 文档 | +| `apitestka_records_to_openapi` | 从测试记录或保存的报告(`report_paths`)反推 OpenAPI 文档 | | `apitestka_clear_records` | 清空测试记录 | | `apitestka_get_records` | 拿当前的成功 / 失败记录 | diff --git a/README/README_zh-TW.md b/README/README_zh-TW.md index 5ad42a9..c8d9a19 100644 --- a/README/README_zh-TW.md +++ b/README/README_zh-TW.md @@ -302,6 +302,8 @@ apitestka summary # ANSI 彩色摘要 apitestka scaffold https://api/x out.json apitestka completion bash # source >> ~/.bashrc apitestka mcp # 走 stdio 啟動 MCP server +apitestka openapi --report run_success.json -o openapi.json # 從存下的執行結果反推規格 +apitestka openapi --run actions.json # 先執行再印出反推的規格 ``` ### 安全檢測 @@ -323,12 +325,21 @@ run_pip_audit() ### OpenAPI 反推 ```python -from je_api_testka.spec import infer_schema, records_to_openapi, openapi_changelog +from je_api_testka.spec import ( + infer_schema, records_to_openapi, build_openapi, export_openapi, load_report_records, openapi_changelog, +) -records_to_openapi(title="Recovered", version="0.1.0") +records_to_openapi(title="Recovered", version="0.1.0") # 從記憶體中的測試紀錄 +build_openapi(["run_success.json"]) # 測試紀錄加上存下的 JSON 報告 +export_openapi("openapi.json", ["run_success.json"]) # 同上,並寫成 UTF-8 JSON openapi_changelog(prev_spec, current_spec) # markdown changelog ``` +同一個 method 與路徑的紀錄會合併成一個 operation:每個出現過的狀態碼各有一筆 response, +查詢參數名稱變成 `in: query` 參數,JSON 或文字內容變成回應與請求內容的 schema。 +存下的報告是 `generate_json_report` 產生的 `_success.json`;`AT_export_openapi` +指令與 `apitestka openapi` 也讀得到它。 + ### GUI ```bash @@ -373,7 +384,7 @@ MCP-compatible client 可以直接驅動本框架。共曝露八個工具: | `apitestka_curl_to_action` | cURL → action JSON | | `apitestka_har_import` | HAR 檔 → action list | | `apitestka_render_markdown` | 從目前紀錄產 Markdown 報告 | -| `apitestka_records_to_openapi` | 反推 OpenAPI 文件 | +| `apitestka_records_to_openapi` | 從測試紀錄或存下的報告(`report_paths`)反推 OpenAPI 文件 | | `apitestka_clear_records` | 清空測試紀錄 | | `apitestka_get_records` | 拿目前的成功 / 失敗紀錄 | diff --git a/architecture.md b/architecture.md index 126e539..c75d9a2 100644 --- a/architecture.md +++ b/architecture.md @@ -32,7 +32,7 @@ CLI, a TCP socket server, an MCP server, a pytest plugin and an optional PySide6 | `je_api_testka/utils/{json,xml,file_process,logging,exception,retry,observability}/` | JSON and XML I/O, directory listing, `apitestka_logger` (file at `$APITESTKA_LOG_FILE` or `~/.je_api_testka/logs/APITestka.log`, opened on first use; the root logger is left alone), exception hierarchy, `RetryPolicy`, OpenTelemetry hooks | | `je_api_testka/data/` | Variable store, template rendering, env profiles, fake-data helpers, data rows | | `je_api_testka/connection/` | Connection options, DNS override, record/replay cassettes | -| `je_api_testka/diff/`, `spec/` | Response and contract diff, SLA checks; schema inference, records → OpenAPI, OpenAPI changelog | +| `je_api_testka/diff/`, `spec/` | Response and contract diff, SLA checks; schema inference, records → OpenAPI (`records_to_openapi`; `openapi_export` reads saved JSON reports and writes the spec), OpenAPI changelog | | `je_api_testka/security/` | Auth header helpers, CORS/SSRF/rate-limit probes, header scan, fuzzing, `pip-audit` wrapper | | `je_api_testka/runner/` | Parallel runner, tag filter, dependency ordering of actions | | `je_api_testka/integrations/` | cURL and HAR import, webhook notify, GitHub PR comment | @@ -57,7 +57,8 @@ CLI, a TCP socket server, an MCP server, a pytest plugin and an optional PySide6 `-c/--create_project` and `--execute_str`. On `win32`/`cygwin`/`msys`, `--execute_str` is decoded with `json.loads` twice. Errors print `repr(error)` to stderr and exit with code 1. - **Subcommand CLI**: `apitestka` (`je_api_testka.cli.cli_main:main`) with `run`, `create`, `mock`, - `import`, `repl`, `summary`, `scaffold`, `completion` and `mcp`. + `import`, `repl`, `summary`, `scaffold`, `completion`, `mcp` and `openapi`. `cli/completion.py` + `SUBCOMMANDS` lists the same names (a test compares them with the parser). - **MCP**: `apitestka-mcp`, `python -m je_api_testka.mcp_server` or `apitestka mcp` (stdio, needs the `mcp` extra, mcp 1.x or 2.x: `build_server` registers the handlers with the 1.x `list_tools()`/`call_tool()` decorators or passes them to the 2.x `Server(on_list_tools=..., on_call_tool=...)`). Tools are the `apitestka_*` `MCPToolSpec` entries in `APITESTKA_TOOLS` diff --git a/docs/source/Eng/doc/cli/cli_doc.rst b/docs/source/Eng/doc/cli/cli_doc.rst index 12752a7..a67bfe5 100644 --- a/docs/source/Eng/doc/cli/cli_doc.rst +++ b/docs/source/Eng/doc/cli/cli_doc.rst @@ -48,3 +48,51 @@ CLI Flags - Execute a JSON string directly * - ``-c``, ``--create_project`` - Create a project directory with template files + +Subcommand CLI +-------------- + +The ``apitestka`` command (installed with the package) groups the everyday tasks: + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - Subcommand + - Description + * - ``run PATH`` + - Execute an action JSON file or every JSON file in a directory + * - ``create PATH`` + - Scaffold a project directory + * - ``mock [--host] [--port]`` + - Start the Flask mock server + * - ``import INPUT OUTPUT [--format openapi|postman]`` + - Convert an OpenAPI document or Postman collection into action JSON + * - ``repl`` + - Interactive JSON-action REPL + * - ``summary`` + - Print a terminal summary of the latest run + * - ``scaffold URL OUTPUT [--method]`` + - Write a starter action JSON for one URL + * - ``completion SHELL`` + - Print a completion script for bash, zsh, fish or PowerShell + * - ``mcp`` + - Run the MCP server over stdio + * - ``openapi`` + - Infer an OpenAPI document from recorded traffic (below) + +Inferring an OpenAPI document +----------------------------- + +``apitestka openapi`` builds an OpenAPI 3.1 document from successful requests. Give it saved +JSON success reports (the ``_success.json`` file from ``generate_json_report``), action +files to run first, or both; each option can be repeated. + +.. code-block:: bash + + apitestka openapi --report run_success.json -o openapi.json + apitestka openapi --run tests/ --title "Shop API" --api-version 2.0 + +Without ``-o`` the document is printed to stdout. The command exits with 2 when no source is +given or a ``--run`` path does not exist, and with 1 (writing nothing) when no successful +record was found. diff --git a/docs/source/Eng/doc/diff/diff_doc.rst b/docs/source/Eng/doc/diff/diff_doc.rst index 5353eca..3a3fcc4 100644 --- a/docs/source/Eng/doc/diff/diff_doc.rst +++ b/docs/source/Eng/doc/diff/diff_doc.rst @@ -48,6 +48,23 @@ Render the same drift as a markdown changelog: print(openapi_changelog(prev_spec, current_spec)) +Inferring OpenAPI from recorded traffic +--------------------------------------- + +``records_to_openapi`` turns test records into an OpenAPI 3.1 document. Records with the same +method and path merge into one operation: every status code seen gets a response entry, query +parameter names become ``in: query`` parameters, and JSON or text bodies become response and +request-body schemas. ``build_openapi`` adds records read back from saved JSON reports, and +``export_openapi`` writes the result as UTF-8 JSON. + +.. code-block:: python + + from je_api_testka.spec import build_openapi, export_openapi, load_report_records + + records = load_report_records("run_success.json") # from generate_json_report + spec = build_openapi(["run_success.json"], title="Shop API") + export_openapi("openapi.json", ["run_success.json"]) + Response time SLA ----------------- @@ -68,3 +85,5 @@ Executor commands * ``AT_diff_openapi_specs`` * ``AT_assert_sla`` * ``AT_openapi_changelog`` +* ``AT_records_to_openapi`` +* ``AT_export_openapi`` diff --git a/docs/source/Eng/doc/mcp/mcp_doc.rst b/docs/source/Eng/doc/mcp/mcp_doc.rst index 6dae065..0ff8756 100644 --- a/docs/source/Eng/doc/mcp/mcp_doc.rst +++ b/docs/source/Eng/doc/mcp/mcp_doc.rst @@ -1,6 +1,6 @@ -================= +=================== MCP Server (Claude) -================= +=================== APITestka ships a Model Context Protocol server so Claude Code (or any other MCP-compatible client) can drive the framework as a tool. The implementation @@ -48,7 +48,7 @@ Tools exposed * - ``apitestka_render_markdown`` - Markdown report of the current test record * - ``apitestka_records_to_openapi`` - - Reconstruct an OpenAPI 3.x document + - Reconstruct an OpenAPI 3.x document from the record or saved JSON reports (``report_paths``) * - ``apitestka_clear_records`` - Wipe success / failure records * - ``apitestka_get_records`` diff --git a/docs/source/Zh/Zh_index.rst b/docs/source/Zh/Zh_index.rst index c0b8a4c..e373119 100644 --- a/docs/source/Zh/Zh_index.rst +++ b/docs/source/Zh/Zh_index.rst @@ -18,7 +18,7 @@ HAR / cURL 匯入器、record-replay proxy、安全檢測、平行 runner、給 * - 類別 - 內容 * - **後端** - - ``requests``(同步、session)、``httpx``(同步 + 非同步、HTTP/2)、 + - ``requests``\ (同步、session)、``httpx``\ (同步 + 非同步、HTTP/2)、 WebSocket、SSE、GraphQL * - **資料層** - 變數儲存、``{{var}}`` 模板、CSV/JSON 資料驅動、環境設定檔、假資料 diff --git a/docs/source/Zh/doc/cli/cli_doc.rst b/docs/source/Zh/doc/cli/cli_doc.rst index 6802889..6917149 100644 --- a/docs/source/Zh/doc/cli/cli_doc.rst +++ b/docs/source/Zh/doc/cli/cli_doc.rst @@ -48,3 +48,49 @@ CLI 參數 - 直接執行 JSON 字串 * - ``-c``, ``--create_project`` - 建立專案目錄及範本檔案 + +子命令 CLI +-------------- + +安裝套件後會有 ``apitestka`` 指令，把常用工作集中在一起： + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - 子命令 + - 說明 + * - ``run PATH`` + - 執行一個動作 JSON 檔，或目錄裡每個 JSON 檔 + * - ``create PATH`` + - 建立專案目錄 + * - ``mock [--host] [--port]`` + - 啟動 Flask 模擬伺服器 + * - ``import INPUT OUTPUT [--format openapi|postman]`` + - 把 OpenAPI 文件或 Postman collection 轉成動作 JSON + * - ``repl`` + - 互動式 JSON 動作 REPL + * - ``summary`` + - 在終端機印出最近一次執行的摘要 + * - ``scaffold URL OUTPUT [--method]`` + - 為一個 URL 寫出起始的動作 JSON + * - ``completion SHELL`` + - 印出 bash、zsh、fish 或 PowerShell 的補全腳本 + * - ``mcp`` + - 以 stdio 啟動 MCP server + * - ``openapi`` + - 從錄下的流量反推 OpenAPI 文件（見下節） + +反推 OpenAPI 文件 +---------------------- + +``apitestka openapi`` 用成功的請求建出 OpenAPI 3.1 文件。來源可以是存下的 JSON 成功報告 +（``generate_json_report`` 產生的 ``_success.json``）、先執行的動作檔，或兩者都給；每個選項都可以重複。 + +.. code-block:: bash + + apitestka openapi --report run_success.json -o openapi.json + apitestka openapi --run tests/ --title "Shop API" --api-version 2.0 + +沒有 ``-o`` 時文件印到 stdout。沒給任何來源、或 ``--run`` 的路徑不存在時結束碼是 2； +找不到任何成功紀錄時結束碼是 1，而且不寫檔。 diff --git a/docs/source/Zh/doc/diff/diff_doc.rst b/docs/source/Zh/doc/diff/diff_doc.rst index a7d47bd..e1b24ea 100644 --- a/docs/source/Zh/doc/diff/diff_doc.rst +++ b/docs/source/Zh/doc/diff/diff_doc.rst @@ -47,6 +47,22 @@ OpenAPI contract drift print(openapi_changelog(prev_spec, current_spec)) +從錄下的流量反推 OpenAPI +------------------------------ + +``records_to_openapi`` 把測試紀錄轉成 OpenAPI 3.1 文件。同一個 method 與路徑的紀錄會合併成一個 +operation：每個出現過的狀態碼各有一筆 response，查詢參數名稱變成 ``in: query`` 參數，JSON 或文字 +內容變成回應與請求內容的 schema。``build_openapi`` 再加上從存下的 JSON 報告讀回的紀錄， +``export_openapi`` 把結果寫成 UTF-8 JSON。 + +.. code-block:: python + + from je_api_testka.spec import build_openapi, export_openapi, load_report_records + + records = load_report_records("run_success.json") # generate_json_report 的輸出 + spec = build_openapi(["run_success.json"], title="Shop API") + export_openapi("openapi.json", ["run_success.json"]) + 回應時間 SLA ------------ @@ -67,3 +83,5 @@ Executor 命令 * ``AT_diff_openapi_specs`` * ``AT_assert_sla`` * ``AT_openapi_changelog`` +* ``AT_records_to_openapi`` +* ``AT_export_openapi`` diff --git a/docs/source/Zh/doc/mcp/mcp_doc.rst b/docs/source/Zh/doc/mcp/mcp_doc.rst index a1a4194..a49ab76 100644 --- a/docs/source/Zh/doc/mcp/mcp_doc.rst +++ b/docs/source/Zh/doc/mcp/mcp_doc.rst @@ -46,7 +46,7 @@ MCP-compatible client)能直接把這個框架當成 tool 使用。實作站在 * - ``apitestka_render_markdown`` - 從目前測試紀錄產 Markdown 報告 * - ``apitestka_records_to_openapi`` - - 反推 OpenAPI 3.x 文件 + - 從測試紀錄或存下的 JSON 報告（``report_paths``）反推 OpenAPI 3.x 文件 * - ``apitestka_clear_records`` - 清空測試紀錄 * - ``apitestka_get_records`` diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md new file mode 100644 index 0000000..a50791c --- /dev/null +++ b/docs/updates/2026-10.md @@ -0,0 +1,15 @@ +# 2026-10 update log + +Index and query commands: [README.md](README.md). New entries go at the end. + +--- + +## U-20261001-01 · 2026-10-01 · records_to_openapi reachable from the CLI and MCP; merges responses · #done #openapi #cli #mcp + +- **What** (`progress.md` #7): `apitestka openapi` infers an OpenAPI 3.1 document from saved JSON success reports (`--report`, repeatable), from action files it runs first (`--run`, repeatable), or both. It writes to `-o` or stdout and takes `--title` and `--api-version`. It exits with 2 when no source is given or a `--run` path is missing, and with 1 (writing nothing) when no successful record is found. The `apitestka_records_to_openapi` MCP tool takes `report_paths`, so a fresh MCP process can use an earlier run. New `spec/openapi_export.py`: `load_report_records` reads the `generate_json_report` success file (and turns its `"None"` / `b'...'` strings back into values), the MCP `{"successes": [...]}` dump, or a plain record list; `build_openapi` combines the in-memory record with those; `export_openapi` writes UTF-8 JSON and is registered as `AT_export_openapi`. +- **Fixes in `records_to_openapi`**: a response without a body used to get `content: {"application/json": {"text/plain": {}}}`, and it now has no `content`. Non-JSON text is `text/plain` with a string schema. A second record for the same operation used to replace the whole operation, and now each status code keeps its own response. Query parameter names become `in: query` parameters, and a JSON or text request body becomes `requestBody`. +- **Also**: `cli/completion.py` `SUBCOMMANDS` lacked `completion` and `mcp`; a test now compares it with the parser. Two older Sphinx warnings are fixed (MCP page title overline, an inline literal in `Zh_index.rst`), so `sphinx -W` builds cleanly. +- **Found, not fixed here**: `import`, `scaffold`, `curl_to_action`, `convert_har`, `generate_tests_from_openapi` and the `apitestka_test_api` MCP tool emit `AT_test_api_method_requests`, which the executor does not register. It is fixed in the next entry. +- **Tests**: `test/test_spec/test_records_to_openapi.py` (8 new, 2 of them regressions that fail on the old code), `test/test_spec/test_openapi_export.py` (new, including a round trip through `generate_json_report` against the local mock), `test/test_cli/test_cli_main.py` (5 new), `test/test_cli/test_completion.py` (parser parity), `test/test_mcp_server/test_tool_definitions.py` (3 new). +- **Result / numbers**: `pytest` 447 passed, 2 skipped (418 before); `sphinx -W` builds without warnings. +- **Files**: `je_api_testka/spec/{records_to_openapi,openapi_export,__init__}.py`, `je_api_testka/cli/{cli_main,completion}.py`, `je_api_testka/mcp_server/tool_definitions.py`, `je_api_testka/utils/executor/action_executor.py`; README ×3, `docs/source/{Eng,Zh}/doc/{cli,diff,mcp}`, `architecture.md` §2–§3. diff --git a/docs/updates/README.md b/docs/updates/README.md index 10ae024..798bffe 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261001-01 | 2026-10-01 | records_to_openapi reachable from the CLI and MCP | #done #openapi #cli #mcp | [2026-10](2026-10.md) | | U-20260925-03 | 2026-09-25 | Python classifiers list every version CI tests | #packaging #tests | [2026-09](2026-09.md) | | U-20260925-02 | 2026-09-25 | dev_requirements.txt pins the package's PySide6 | #deps #tests | [2026-09](2026-09.md) | | U-20260925-01 | 2026-09-25 | Dependabot waits 7 days before proposing a new release | #ci #security #deps | [2026-09](2026-09.md) | @@ -86,4 +87,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| +| [2026-10.md](2026-10.md) | 2026-10 | 1 | | [2026-09.md](2026-09.md) | 2026-09 | 23 | diff --git a/je_api_testka/cli/cli_main.py b/je_api_testka/cli/cli_main.py index 8608f62..82945bd 100644 --- a/je_api_testka/cli/cli_main.py +++ b/je_api_testka/cli/cli_main.py @@ -10,6 +10,8 @@ summary Print a terminal summary of the latest run. scaffold Generate a starter action JSON for a URL. completion Print a shell completion script. + mcp Run the MCP server over stdio. + openapi Infer an OpenAPI document from recorded traffic. """ from __future__ import annotations @@ -19,6 +21,8 @@ from pathlib import Path from typing import Optional, Sequence +from je_api_testka.spec.openapi_export import build_openapi +from je_api_testka.spec.records_to_openapi import DEFAULT_SPEC_TITLE, DEFAULT_SPEC_VERSION from je_api_testka.utils.executor.action_executor import execute_action, execute_files from je_api_testka.utils.file_process.get_dir_file_list import get_dir_files_as_list from je_api_testka.utils.json.json_file.json_file import read_action_json @@ -29,8 +33,7 @@ DEFAULT_MOCK_PORT: int = 8090 -def _cmd_run(args: argparse.Namespace) -> int: - target = Path(args.path) +def _run_path(target: Path) -> int: if target.is_dir(): execute_files(get_dir_files_as_list(str(target))) elif target.is_file(): @@ -41,6 +44,10 @@ def _cmd_run(args: argparse.Namespace) -> int: return 0 +def _cmd_run(args: argparse.Namespace) -> int: + return _run_path(Path(args.path)) + + def _cmd_create(args: argparse.Namespace) -> int: create_project_dir(args.path) return 0 @@ -97,6 +104,37 @@ def _cmd_mcp(_args: argparse.Namespace) -> int: return mcp_main() +def _cmd_openapi(args: argparse.Namespace) -> int: + if not args.report and not args.run: + apitestka_logger.error("cli openapi: give at least one --report or --run") + return 2 + for path in args.run: + status = _run_path(Path(path)) + if status: + return status + spec = build_openapi(args.report, title=args.title, version=args.api_version) + if not spec["paths"]: + apitestka_logger.error("cli openapi: no successful records to infer from") + return 1 + text = json.dumps(spec, indent=2, ensure_ascii=False) + if args.output: + Path(args.output).write_text(text, encoding="utf-8") + else: + print(text) + return 0 + + +def _configure_openapi_parser(openapi_parser: argparse.ArgumentParser) -> None: + openapi_parser.add_argument("--report", action="append", default=[], metavar="FILE", + help="Saved JSON success report (_success.json); repeatable") + openapi_parser.add_argument("--run", action="append", default=[], metavar="PATH", + help="Action JSON file or directory to execute first; repeatable") + openapi_parser.add_argument("-o", "--output", help="Destination file (default: stdout)") + openapi_parser.add_argument("--title", default=DEFAULT_SPEC_TITLE) + openapi_parser.add_argument("--api-version", default=DEFAULT_SPEC_VERSION, help="info.version of the spec") + openapi_parser.set_defaults(func=_cmd_openapi) + + def build_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser(prog="apitestka", description="APITestka command line") sub = parser.add_subparsers(dest="command", required=True) @@ -143,6 +181,8 @@ def build_parser() -> argparse.ArgumentParser: mcp_parser = sub.add_parser("mcp", help="Run the APITestka MCP server (stdio transport)") mcp_parser.set_defaults(func=_cmd_mcp) + + _configure_openapi_parser(sub.add_parser("openapi", help="Infer an OpenAPI document from recorded traffic")) return parser diff --git a/je_api_testka/cli/completion.py b/je_api_testka/cli/completion.py index 856e741..193f871 100644 --- a/je_api_testka/cli/completion.py +++ b/je_api_testka/cli/completion.py @@ -6,7 +6,9 @@ """ from __future__ import annotations -SUBCOMMANDS = ("run", "create", "mock", "import", "repl", "scaffold", "summary") +SUBCOMMANDS = ( + "run", "create", "mock", "import", "repl", "summary", "scaffold", "completion", "mcp", "openapi", +) SUPPORTED_SHELLS = ("bash", "zsh", "fish", "powershell") diff --git a/je_api_testka/mcp_server/tool_definitions.py b/je_api_testka/mcp_server/tool_definitions.py index 8160104..8cf18e6 100644 --- a/je_api_testka/mcp_server/tool_definitions.py +++ b/je_api_testka/mcp_server/tool_definitions.py @@ -19,7 +19,8 @@ from je_api_testka.integrations.curl_import import curl_to_action from je_api_testka.integrations.har_import import convert_har -from je_api_testka.spec.records_to_openapi import records_to_openapi +from je_api_testka.spec.openapi_export import build_openapi +from je_api_testka.spec.records_to_openapi import DEFAULT_SPEC_TITLE, DEFAULT_SPEC_VERSION from je_api_testka.utils.executor.action_executor import execute_action from je_api_testka.utils.exception.exceptions import APITesterException from je_api_testka.utils.generate_report.markdown_report import render_markdown @@ -72,9 +73,13 @@ def _handle_render_markdown(_arguments: Dict[str, Any]) -> str: def _handle_records_to_openapi(arguments: Dict[str, Any]) -> Any: - return records_to_openapi( - title=arguments.get("title", "APITestka Inferred"), - version=arguments.get("version", "0.1.0"), + report_paths = arguments.get("report_paths") or [] + if not isinstance(report_paths, list) or not all(isinstance(path, str) for path in report_paths): + raise APITesterException("'report_paths' must be a list of file paths") + return build_openapi( + report_paths, + title=arguments.get("title", DEFAULT_SPEC_TITLE), + version=arguments.get("version", DEFAULT_SPEC_VERSION), ) @@ -166,12 +171,20 @@ def _jsonable(value: Any) -> Any: ), MCPToolSpec( name="apitestka_records_to_openapi", - description="Reconstruct an OpenAPI 3.x document from the current test record.", + description=( + "Reconstruct an OpenAPI 3.x document from the current test record " + "plus any saved JSON reports (_success.json)." + ), input_schema={ "type": "object", "properties": { "title": {"type": "string"}, "version": {"type": "string"}, + "report_paths": { + "type": "array", + "items": {"type": "string"}, + "description": "Saved JSON success reports to read records from.", + }, }, }, handler=_handle_records_to_openapi, diff --git a/je_api_testka/spec/__init__.py b/je_api_testka/spec/__init__.py index 2696e5e..c2e1834 100644 --- a/je_api_testka/spec/__init__.py +++ b/je_api_testka/spec/__init__.py @@ -1,9 +1,13 @@ from je_api_testka.spec.openapi_changelog import openapi_changelog +from je_api_testka.spec.openapi_export import build_openapi, export_openapi, load_report_records from je_api_testka.spec.records_to_openapi import records_to_openapi from je_api_testka.spec.schema_inference import infer_schema __all__ = [ + "build_openapi", + "export_openapi", "infer_schema", + "load_report_records", "openapi_changelog", "records_to_openapi", ] diff --git a/je_api_testka/spec/openapi_export.py b/je_api_testka/spec/openapi_export.py new file mode 100644 index 0000000..ea0aab0 --- /dev/null +++ b/je_api_testka/spec/openapi_export.py @@ -0,0 +1,88 @@ +""" +Build and write an inferred OpenAPI document from recorded traffic. + +Records come from the in-memory test record and from saved JSON reports, so a +fresh process (the ``apitestka openapi`` CLI, the MCP server) can still infer a +spec from an earlier run. +""" +from __future__ import annotations + +import ast +import json +import re +from pathlib import Path +from typing import Any, List, Optional, Sequence + +from je_api_testka.spec.records_to_openapi import DEFAULT_SPEC_TITLE, DEFAULT_SPEC_VERSION, records_to_openapi +from je_api_testka.utils.exception.exceptions import APIJsonReportException +from je_api_testka.utils.test_record.test_record_class import test_record_instance + +# generate_json_report writes every field through str(): None becomes "None" +# and a bytes request body becomes its repr, b'...'. +_REPORT_NONE: str = "None" +_BYTES_REPR = re.compile(r"^b(['\"]).*\1$", re.DOTALL) + + +def _report_value(value: Any) -> Any: + if value == _REPORT_NONE: + return None + if isinstance(value, str) and _BYTES_REPR.match(value): + # literal_eval only builds literals; the regex restricts it to one bytes literal. + try: + return ast.literal_eval(value) + except (ValueError, SyntaxError): + return value + return value + + +def _record_entries(document: Any, path: str) -> List[Any]: + if isinstance(document, list): + return document + if isinstance(document, dict) and isinstance(document.get("successes"), list): + return document["successes"] + if isinstance(document, dict): + return list(document.values()) + raise APIJsonReportException(f"{path}: expected a JSON report object or a list of records") + + +def load_report_records(path: str) -> List[dict]: + """ + Read success records back from a saved report. + + Accepts the ``_success.json`` file written by ``generate_json_report``, + the ``{"successes": [...]}`` object returned by the ``apitestka_get_records`` + MCP tool, or a plain list of record objects. + + :raises APIJsonReportException: if the file is not JSON or an entry is not a record object. + """ + try: + document = json.loads(Path(path).read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as error: + raise APIJsonReportException(f"{path}: cannot read report: {error!r}") from error + records = [] + for entry in _record_entries(document, path): + if not isinstance(entry, dict) or "request_url" not in entry: + raise APIJsonReportException(f"{path}: every record needs a request_url") + records.append({key: _report_value(value) for key, value in entry.items()}) + return records + + +def build_openapi(report_paths: Optional[Sequence[str]] = None, title: str = DEFAULT_SPEC_TITLE, + version: str = DEFAULT_SPEC_VERSION) -> dict: + """Return an OpenAPI document inferred from the current test record plus each saved report.""" + records = list(test_record_instance.test_record_list) + for path in report_paths or (): + records.extend(load_report_records(path)) + return records_to_openapi(records, title=title, version=version) + + +def export_openapi(output_path: str, report_paths: Optional[Sequence[str]] = None, + title: str = DEFAULT_SPEC_TITLE, version: str = DEFAULT_SPEC_VERSION) -> str: + """ + Write the document from :func:`build_openapi` to ``output_path`` as UTF-8 JSON. + + :return: the path written. + """ + spec = build_openapi(report_paths, title=title, version=version) + Path(output_path).write_text(json.dumps(spec, indent=2, ensure_ascii=False), encoding="utf-8") + return str(output_path) diff --git a/je_api_testka/spec/records_to_openapi.py b/je_api_testka/spec/records_to_openapi.py index 629b7e4..ab4309e 100644 --- a/je_api_testka/spec/records_to_openapi.py +++ b/je_api_testka/spec/records_to_openapi.py @@ -2,55 +2,97 @@ Reconstruct an OpenAPI 3.x dictionary from the global test record. This is a coarse, "good enough to start" approach: it groups records by -(method, path) and infers response schemas via :func:`infer_schema`. Path -templating is not detected automatically. +(method, path), keeps one response entry per status code, and infers +response and request-body schemas via :func:`infer_schema`. Query parameter +names seen in the URLs become ``in: query`` parameters. Path templating is +not detected automatically. """ from __future__ import annotations import json -from typing import Iterable -from urllib.parse import urlparse +from typing import Any, Iterable, Optional, Tuple +from urllib.parse import parse_qsl, urlparse from je_api_testka.spec.schema_inference import infer_schema from je_api_testka.utils.test_record.test_record_class import test_record_instance DEFAULT_OPENAPI_VERSION: str = "3.1.0" +DEFAULT_SPEC_TITLE: str = "APITestka Inferred" +DEFAULT_SPEC_VERSION: str = "0.1.0" +JSON_MEDIA_TYPE: str = "application/json" +TEXT_MEDIA_TYPE: str = "text/plain" -def _decode_body(record: dict): - text = record.get("text") or "" - if not text: +def _media_sample(raw: Any) -> Optional[Tuple[str, Any]]: + """Return ``(media type, sample)`` for a body, or ``None`` when it is empty.""" + if isinstance(raw, (bytes, bytearray)): + try: + raw = raw.decode("utf-8") + except UnicodeDecodeError: + return None + if not isinstance(raw, str) or not raw: return None try: - return json.loads(text) + return JSON_MEDIA_TYPE, json.loads(raw) except json.JSONDecodeError: - return text + return TEXT_MEDIA_TYPE, raw -def records_to_openapi(records: Iterable[dict] = None, title: str = "APITestka Inferred", - version: str = "0.1.0") -> dict: - """Return an OpenAPI 3.x dictionary describing the supplied (or global) records.""" +def _content(raw: Any) -> Optional[dict]: + """Return an OpenAPI ``content`` map for a body, or ``None`` when it is empty.""" + sample = _media_sample(raw) + if sample is None: + return None + media_type, value = sample + return {media_type: {"schema": infer_schema(value)}} + + +def _response_entry(record: dict) -> dict: + entry: dict = {"description": "inferred"} + content = _content(record.get("text")) + if content is not None: + entry["content"] = content + return entry + + +def _merge_query_parameters(operation: dict, query: str) -> None: + """Add a ``query`` parameter for each name in ``query`` the operation does not list yet.""" + parameters = operation.setdefault("parameters", []) + known = {parameter["name"] for parameter in parameters} + for name, _value in parse_qsl(query, keep_blank_values=True): + if name not in known: + parameters.append({"name": name, "in": "query", "schema": {"type": "string"}}) + known.add(name) + if not parameters: + del operation["parameters"] + + +def _add_record(paths: dict, record: dict) -> None: + url = record.get("request_url") or "" + if not url: + return + parsed = urlparse(url) + method = (record.get("request_method") or "get").lower() + operation = paths.setdefault(parsed.path or "/", {}).setdefault(method, {"responses": {}}) + operation["responses"][str(record.get("status_code", 200))] = _response_entry(record) + _merge_query_parameters(operation, parsed.query) + request_content = _content(record.get("request_body")) + if request_content is not None: + operation["requestBody"] = {"content": request_content} + + +def records_to_openapi(records: Optional[Iterable[dict]] = None, title: str = DEFAULT_SPEC_TITLE, + version: str = DEFAULT_SPEC_VERSION) -> dict: + """ + Return an OpenAPI 3.x dictionary describing the supplied (or global) records. + + Records for the same method and path merge into one operation: each status + code keeps the latest response seen, and query parameter names accumulate. + """ source = list(records) if records is not None else list(test_record_instance.test_record_list) paths: dict = {} for record in source: - url = record.get("request_url") or "" - method = (record.get("request_method") or "get").lower() - if not url: - continue - parsed = urlparse(url) - path = parsed.path or "/" - body = _decode_body(record) - operation = { - "responses": { - str(record.get("status_code", 200)): { - "description": "inferred", - "content": { - "application/json": {"schema": infer_schema(body)} if body is not None else {"text/plain": {}} - }, - } - } - } - paths.setdefault(path, {})[method] = operation + _add_record(paths, record) return { "openapi": DEFAULT_OPENAPI_VERSION, "info": {"title": title, "version": version}, diff --git a/je_api_testka/utils/executor/action_executor.py b/je_api_testka/utils/executor/action_executor.py index 177f9e5..cc46a15 100644 --- a/je_api_testka/utils/executor/action_executor.py +++ b/je_api_testka/utils/executor/action_executor.py @@ -36,6 +36,7 @@ from je_api_testka.security.rate_limit_probe import probe_rate_limit from je_api_testka.security.ssrf_check import probe_ssrf from je_api_testka.spec.openapi_changelog import openapi_changelog +from je_api_testka.spec.openapi_export import export_openapi from je_api_testka.spec.records_to_openapi import records_to_openapi from je_api_testka.spec.schema_inference import infer_schema from je_api_testka.sse_wrapper.sse_method import test_api_method_sse @@ -141,6 +142,7 @@ def __init__(self): # Spec inference "AT_infer_schema": infer_schema, "AT_records_to_openapi": records_to_openapi, + "AT_export_openapi": export_openapi, "AT_openapi_changelog": openapi_changelog, # AI integrations "AT_classify_failures": classify_failures, diff --git a/progress.md b/progress.md index cedb50c..29fb5f3 100644 --- a/progress.md +++ b/progress.md @@ -8,7 +8,6 @@ Cross-repo and workspace items live in `D:\Codes\progress.md` (relevant here: X- - **#3** [DECIDE] The PyPI classifier still says `Development Status :: 2 - Pre-Alpha`. - **#6** GUI redesign: restructure and restyle. `gui/main_widget.py` is one 605-line widget (over the 500-line limit) with six flat tabs. It should become a sidebar-navigated window with a request/response split view, and history and environment panels docked (`gui/history_panel.py`, `gui/env_manager_model.py` exist but are not wired in). It needs a refreshed theme (`gui/main_window.py:21` hard-codes `dark_amber.xml`), pages for the newer features (#8–#13), and all four language wrappers (English, Traditional Chinese, Simplified Chinese, Japanese). `APITestkaWidget` stays the embeddable entry point (PyBreeze contract, `architecture.md` §6). -- **#7** `records_to_openapi` has no CLI subcommand (`cli/cli_main.py:101` `build_parser`). The MCP tool (`mcp_server/tool_definitions.py:74`) reads only the in-memory record, so a fresh MCP process gets an empty spec. Neither can read a saved JSON report or write a file. `spec/records_to_openapi.py:48` also labels a body-less response `application/json` → `text/plain`, and the last record per operation overwrites earlier status codes. - **#8** Mock server: gRPC and WebSocket endpoints. `utils/mock_server/grpc_stub.py` serves unary calls only and is not reachable from `apitestka mock`, the executor or `flask_mock_server_instance`. There is no WebSocket mock endpoint. - **#9** Pact-style bidirectional contract testing. `diff/contract_diff.py` only compares two OpenAPI documents. Missing: a consumer contract recorded from the consumer's tests (interactions plus matchers), provider verification replaying it against a running provider, and a cross-check of the consumer contract against the provider's OpenAPI. - **#10** Reference AI backend on the Anthropic API. `ai/backend.py:1` ships only `NoOpAIBackend` and `StaticAIBackend`. Missing: an optional `ai` extra, an `AnthropicAIBackend` (key from the environment, never stored), and a way to select it (CLI, executor). diff --git a/test/test_cli/test_cli_main.py b/test/test_cli/test_cli_main.py index 2a2a59e..417035d 100644 --- a/test/test_cli/test_cli_main.py +++ b/test/test_cli/test_cli_main.py @@ -38,3 +38,45 @@ def test_main_invokes_import(tmp_path): def test_main_run_with_missing_path_returns_2(tmp_path): rc = main(["run", str(tmp_path / "does_not_exist.json")]) assert rc == 2 + + +def _write_report(path, url="https://x.invalid/users"): + path.write_text(json.dumps({"Success_Test1": { + "request_url": url, "request_method": "GET", "status_code": "200", "text": '{"id": 1}', + }}), encoding="utf-8") + return str(path) + + +def test_openapi_from_report_writes_file(tmp_path): + report = _write_report(tmp_path / "run_success.json") + out_path = tmp_path / "spec.json" + rc = main(["openapi", "--report", report, "-o", str(out_path), "--title", "Shop", "--api-version", "2.0"]) + spec = json.loads(out_path.read_text(encoding="utf-8")) + assert rc == 0 + assert spec["info"] == {"title": "Shop", "version": "2.0"} + assert "/users" in spec["paths"] + + +def test_openapi_runs_actions_then_prints(tmp_path, mock_url, capsys): + actions = tmp_path / "actions.json" + actions.write_text(json.dumps([["AT_test_api_method", { + "http_method": "get", "test_url": f"{mock_url}/get", "timeout": 30, + }]]), encoding="utf-8") + rc = main(["openapi", "--run", str(actions)]) + assert rc == 0 + assert "/get" in json.loads(capsys.readouterr().out)["paths"] + + +def test_openapi_needs_a_source(): + assert main(["openapi"]) == 2 + + +def test_openapi_with_missing_run_path_returns_2(tmp_path): + assert main(["openapi", "--run", str(tmp_path / "missing.json")]) == 2 + + +def test_openapi_without_records_returns_1(tmp_path): + empty = tmp_path / "empty_success.json" + empty.write_text("{}", encoding="utf-8") + assert main(["openapi", "--report", str(empty), "-o", str(tmp_path / "spec.json")]) == 1 + assert not (tmp_path / "spec.json").exists() diff --git a/test/test_cli/test_completion.py b/test/test_cli/test_completion.py index bb3c263..a6fd85b 100644 --- a/test/test_cli/test_completion.py +++ b/test/test_cli/test_completion.py @@ -3,9 +3,16 @@ import pytest +from je_api_testka.cli.cli_main import build_parser from je_api_testka.cli.completion import SUBCOMMANDS, generate_completion_script +def test_subcommands_match_the_parser(): + # Regression: completion missed "completion" and "mcp", so the list must follow build_parser. + subparsers = next(action for action in build_parser()._actions if action.dest == "command") + assert set(SUBCOMMANDS) == set(subparsers.choices) + + def test_bash_script_lists_subcommands(): script = generate_completion_script("bash") for command in SUBCOMMANDS: diff --git a/test/test_executor_integration.py b/test/test_executor_integration.py index f2c122b..f6c8086 100644 --- a/test/test_executor_integration.py +++ b/test/test_executor_integration.py @@ -40,7 +40,7 @@ "AT_basic_auth_header", "AT_bearer_token_header", "AT_build_jwt", "AT_aws_sigv4_headers", # Spec - "AT_infer_schema", "AT_records_to_openapi", "AT_openapi_changelog", + "AT_infer_schema", "AT_records_to_openapi", "AT_export_openapi", "AT_openapi_changelog", # AI "AT_classify_failures", "AT_generate_fake_payload", "AT_generate_tests_from_openapi", diff --git a/test/test_mcp_server/test_tool_definitions.py b/test/test_mcp_server/test_tool_definitions.py index 3c5aa8b..a54a129 100644 --- a/test/test_mcp_server/test_tool_definitions.py +++ b/test/test_mcp_server/test_tool_definitions.py @@ -66,6 +66,22 @@ def test_records_to_openapi_tool(): test_record_instance.clean_record() +def test_records_to_openapi_tool_reads_saved_reports(tmp_path): + # A fresh MCP process has an empty record; report_paths lets it use an earlier run. + report = tmp_path / "run_success.json" + report.write_text(json.dumps({"Success_Test1": { + "request_url": "https://x.invalid/saved", "request_method": "GET", "status_code": "200", "text": "", + }}), encoding="utf-8") + result = dispatch_tool("apitestka_records_to_openapi", {"report_paths": [str(report)]}) + assert "/saved" in result["paths"] + + +@pytest.mark.parametrize("report_paths", ["one.json", [1]]) +def test_records_to_openapi_tool_validates_report_paths(report_paths): + with pytest.raises(APITesterException): + dispatch_tool("apitestka_records_to_openapi", {"report_paths": report_paths}) + + def test_run_action_tool_dispatches(): test_record_instance.clean_record() result = dispatch_tool("apitestka_run_action", {"actions": [["AT_fake_uuid"]]}) diff --git a/test/test_spec/test_openapi_export.py b/test/test_spec/test_openapi_export.py new file mode 100644 index 0000000..078621a --- /dev/null +++ b/test/test_spec/test_openapi_export.py @@ -0,0 +1,88 @@ +"""Tests for building and writing OpenAPI documents from saved reports.""" +from __future__ import annotations + +import json + +import pytest + +from je_api_testka import execute_action, generate_json_report +from je_api_testka.spec.openapi_export import build_openapi, export_openapi, load_report_records +from je_api_testka.utils.exception.exceptions import APIJsonReportException +from je_api_testka.utils.test_record.test_record_class import test_record_instance + + +def _write(path, document): + path.write_text(json.dumps(document), encoding="utf-8") + return str(path) + + +def test_round_trip_through_generate_json_report(mock_url, tmp_path): + execute_action([["AT_test_api_method", { + "http_method": "post", "test_url": f"{mock_url}/post?page=1", "json": {"name": "a"}, "timeout": 30, + }]]) + generate_json_report(str(tmp_path / "run")) + test_record_instance.clean_record() + + records = load_report_records(str(tmp_path / "run_success.json")) + + assert len(records) == 1 + assert records[0]["request_body"] == b'{"name": "a"}' + spec = build_openapi([str(tmp_path / "run_success.json")]) + operation = spec["paths"]["/post"]["post"] + assert operation["parameters"][0]["name"] == "page" + assert "application/json" in operation["requestBody"]["content"] + assert "application/json" in operation["responses"]["200"]["content"] + + +def test_report_none_strings_become_none(tmp_path): + path = _write(tmp_path / "r.json", {"Success_Test1": { + "request_url": "https://x.invalid/a", "request_method": "GET", "status_code": "200", + "text": "", "request_body": "None", + }}) + assert load_report_records(path)[0]["request_body"] is None + + +def test_accepts_mcp_get_records_shape(tmp_path): + path = _write(tmp_path / "r.json", {"successes": [{"request_url": "https://x.invalid/a"}], "failures": []}) + assert load_report_records(path) == [{"request_url": "https://x.invalid/a"}] + + +def test_accepts_plain_record_list(tmp_path): + path = _write(tmp_path / "r.json", [{"request_url": "https://x.invalid/a"}]) + assert len(load_report_records(path)) == 1 + + +@pytest.mark.parametrize("document", [[1], {"Success_Test1": {"text": "no url"}}, "text"]) +def test_rejects_entries_that_are_not_records(tmp_path, document): + with pytest.raises(APIJsonReportException): + load_report_records(_write(tmp_path / "r.json", document)) + + +def test_rejects_unreadable_file(tmp_path): + (tmp_path / "bad.json").write_text("{not json", encoding="utf-8") + with pytest.raises(APIJsonReportException): + load_report_records(str(tmp_path / "bad.json")) + with pytest.raises(APIJsonReportException): + load_report_records(str(tmp_path / "missing.json")) + + +def test_build_combines_current_record_and_reports(tmp_path): + test_record_instance.test_record_list.append({"request_url": "https://x.invalid/live", "status_code": 200}) + path = _write(tmp_path / "r.json", [{"request_url": "https://x.invalid/saved"}]) + spec = build_openapi([path], title="T", version="9") + assert set(spec["paths"]) == {"/live", "/saved"} + assert spec["info"] == {"title": "T", "version": "9"} + + +def test_export_writes_utf8_json(tmp_path): + path = _write(tmp_path / "r.json", [{"request_url": "https://x.invalid/a", "text": '{"name": "測試"}'}]) + written = export_openapi(str(tmp_path / "spec.json"), [path]) + spec = json.loads((tmp_path / "spec.json").read_text(encoding="utf-8")) + assert written == str(tmp_path / "spec.json") + assert "/a" in spec["paths"] + + +def test_export_through_executor(tmp_path): + test_record_instance.test_record_list.append({"request_url": "https://x.invalid/a", "status_code": 200}) + execute_action([["AT_export_openapi", {"output_path": str(tmp_path / "spec.json")}]]) + assert "/a" in json.loads((tmp_path / "spec.json").read_text(encoding="utf-8"))["paths"] diff --git a/test/test_spec/test_records_to_openapi.py b/test/test_spec/test_records_to_openapi.py index 9f66644..a3360b0 100644 --- a/test/test_spec/test_records_to_openapi.py +++ b/test/test_spec/test_records_to_openapi.py @@ -39,3 +39,60 @@ def test_default_response_when_body_empty(): }] spec = records_to_openapi(records=records) assert spec["paths"]["/health"]["get"]["responses"]["200"] + + +def _record(url="https://x.invalid/users", method="GET", status=200, text="", body=None): + return {"request_url": url, "request_method": method, "status_code": status, + "text": text, "request_body": body} + + +def test_body_less_response_has_no_content(): + # Regression: an empty body used to produce content {"application/json": {"text/plain": {}}}. + spec = records_to_openapi(records=[_record(text="")]) + assert spec["paths"]["/users"]["get"]["responses"]["200"] == {"description": "inferred"} + + +def test_json_response_schema_under_application_json(): + spec = records_to_openapi(records=[_record(text='{"id": 1}')]) + content = spec["paths"]["/users"]["get"]["responses"]["200"]["content"] + assert content == {"application/json": {"schema": { + "type": "object", "properties": {"id": {"type": "integer"}}, "required": ["id"], + }}} + + +def test_plain_text_response_under_text_plain(): + spec = records_to_openapi(records=[_record(text="pong")]) + content = spec["paths"]["/users"]["get"]["responses"]["200"]["content"] + assert content == {"text/plain": {"schema": {"type": "string"}}} + + +def test_status_codes_of_one_operation_accumulate(): + # Regression: the last record used to replace the whole operation. + spec = records_to_openapi(records=[_record(status=200), _record(status=404)]) + assert set(spec["paths"]["/users"]["get"]["responses"]) == {"200", "404"} + + +def test_query_parameter_names_accumulate_once(): + spec = records_to_openapi(records=[ + _record(url="https://x.invalid/users?page=1&size=5"), + _record(url="https://x.invalid/users?page=2&sort=name"), + ]) + parameters = spec["paths"]["/users"]["get"]["parameters"] + assert [parameter["name"] for parameter in parameters] == ["page", "size", "sort"] + assert all(parameter["in"] == "query" for parameter in parameters) + + +def test_operation_without_query_has_no_parameters_key(): + spec = records_to_openapi(records=[_record()]) + assert "parameters" not in spec["paths"]["/users"]["get"] + + +def test_json_request_body_bytes_become_request_body_schema(): + spec = records_to_openapi(records=[_record(method="POST", status=201, body=b'{"name": "a"}')]) + request_body = spec["paths"]["/users"]["post"]["requestBody"] + assert request_body["content"]["application/json"]["schema"]["properties"] == {"name": {"type": "string"}} + + +def test_undecodable_request_body_is_skipped(): + spec = records_to_openapi(records=[_record(method="POST", body=b"\xff\xfe")]) + assert "requestBody" not in spec["paths"]["/users"]["post"] From 2771ababa8dd2d4779efd5e0dcffe49427f580bc Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 00:44:28 +0800 Subject: [PATCH 09/43] fix: make imported, scaffolded and AI-generated actions runnable OpenAPI/Postman/cURL/HAR import and the AI fallback returned {"AT_test_api_method_requests": {...}} dicts, which execute_action cannot run, and scaffold plus the apitestka_test_api MCP tool used that unregistered command name. All converters now build ["AT_test_api_method", {...}] through build_request_action. The executor strips runner metadata (id, depends_on, tags) before the call, and assert_sla accepts a mapping SLA and defaults to the shared record, so ["AT_assert_sla", {"sla": {...}}] works from JSON. --- README.md | 5 ++ README/README_zh-CN.md | 5 ++ README/README_zh-TW.md | 5 ++ architecture.md | 5 +- docs/source/Eng/doc/diff/diff_doc.rst | 9 +- .../Eng/doc/integrations/integrations_doc.rst | 6 ++ docs/source/Eng/doc/runner/runner_doc.rst | 11 +-- docs/source/Zh/doc/diff/diff_doc.rst | 8 +- .../Zh/doc/integrations/integrations_doc.rst | 5 ++ docs/source/Zh/doc/runner/runner_doc.rst | 9 +- docs/updates/2026-10.md | 10 +++ docs/updates/README.md | 3 +- je_api_testka/ai/test_generator.py | 22 ++--- je_api_testka/cli/import_specs.py | 37 +++----- je_api_testka/cli/scaffold_test.py | 13 ++- je_api_testka/diff/sla_check.py | 25 ++++-- je_api_testka/integrations/curl_import.py | 14 ++- je_api_testka/integrations/har_import.py | 21 ++--- je_api_testka/mcp_server/tool_definitions.py | 17 ++-- .../utils/executor/action_executor.py | 3 + .../utils/executor/request_action.py | 33 +++++++ test/test_ai/test_test_generator.py | 6 +- test/test_cli/test_import_specs.py | 6 +- test/test_cli/test_scaffold.py | 2 +- test/test_diff/test_sla_check.py | 20 +++++ test/test_executor_integration.py | 2 +- test/test_integrations/test_curl_import.py | 9 +- test/test_integrations/test_har_import.py | 4 +- test/test_mcp_server/test_tool_definitions.py | 2 +- test/test_utils/test_generated_actions_run.py | 89 +++++++++++++++++++ 30 files changed, 294 insertions(+), 112 deletions(-) create mode 100644 je_api_testka/utils/executor/request_action.py create mode 100644 test/test_utils/test_generated_actions_run.py diff --git a/README.md b/README.md index 77dc0e6..cf9fb8e 100644 --- a/README.md +++ b/README.md @@ -197,6 +197,7 @@ check_jsonpath(payload, "$.data.id", expected=7) assert_snapshot("user-by-id", payload, ignore_keys=["timestamp"]) diff = diff_payloads(prod_response, staging_response, ignore_paths=["server_time"]) assert_sla(records, ResponseSLA(max_ms=2000, p95_ms=1500)) +assert_sla(sla={"max_ms": 2000}) # the run recorded so far; JSON: ["AT_assert_sla", {"sla": {...}}] ``` ### Connection Layer @@ -293,6 +294,10 @@ actions = convert_har("traffic.har") actions = convert_spec_file("openapi.json", spec_format="openapi") ``` +Each importer returns `["AT_test_api_method", {...}]` actions that `execute_action` runs +as they are. Runner metadata (`id`, `depends_on`, `tags`) may stay in the kwargs; the +executor strips it before the call. + ### CLI / Developer Experience ```bash diff --git a/README/README_zh-CN.md b/README/README_zh-CN.md index a96b3ab..5f15c25 100644 --- a/README/README_zh-CN.md +++ b/README/README_zh-CN.md @@ -194,6 +194,7 @@ check_jsonpath(payload, "$.data.id", expected=7) assert_snapshot("user-by-id", payload, ignore_keys=["timestamp"]) diff = diff_payloads(prod_response, staging_response, ignore_paths=["server_time"]) assert_sla(records, ResponseSLA(max_ms=2000, p95_ms=1500)) +assert_sla(sla={"max_ms": 2000}) # 目前录下的运行;JSON:["AT_assert_sla", {"sla": {...}}] ``` ### 连接层 @@ -290,6 +291,10 @@ actions = convert_har("traffic.har") actions = convert_spec_file("openapi.json", spec_format="openapi") ``` +每个导入工具都返回 `["AT_test_api_method", {...}]` 形式的 action,`execute_action` +可以直接执行。runner 的元数据(`id`、`depends_on`、`tags`)可以留在 kwargs 里, +executor 调用前会先去掉。 + ### CLI / 开发体验 ```bash diff --git a/README/README_zh-TW.md b/README/README_zh-TW.md index c8d9a19..e14f612 100644 --- a/README/README_zh-TW.md +++ b/README/README_zh-TW.md @@ -194,6 +194,7 @@ check_jsonpath(payload, "$.data.id", expected=7) assert_snapshot("user-by-id", payload, ignore_keys=["timestamp"]) diff = diff_payloads(prod_response, staging_response, ignore_paths=["server_time"]) assert_sla(records, ResponseSLA(max_ms=2000, p95_ms=1500)) +assert_sla(sla={"max_ms": 2000}) # 目前錄下的執行;JSON:["AT_assert_sla", {"sla": {...}}] ``` ### 連線層 @@ -290,6 +291,10 @@ actions = convert_har("traffic.har") actions = convert_spec_file("openapi.json", spec_format="openapi") ``` +每個匯入工具都回傳 `["AT_test_api_method", {...}]` 形式的 action,`execute_action` +可以直接執行。runner 的中繼資料(`id`、`depends_on`、`tags`)可以留在 kwargs 裡, +executor 呼叫前會先拿掉。 + ### CLI / 開發體驗 ```bash diff --git a/architecture.md b/architecture.md index c75d9a2..c82789d 100644 --- a/architecture.md +++ b/architecture.md @@ -52,7 +52,10 @@ CLI, a TCP socket server, an MCP server, a pytest plugin and an optional PySide6 `generate_{html,json,xml}_report`, `flask_mock_server_instance`, `start_apitestka_socket_server`, `callback_executor`, `create_project_dir`. - **Action format**: an action is `[name]`, `[name, {kwargs}]` or `[name, [args]]`. A file holds a - list of actions or `{"api_testka": [...]}`. + list of actions or `{"api_testka": [...]}`. The executor drops runner metadata (`id`, `depends_on`, `tags`) from the + kwargs before the call. Converters (OpenAPI/Postman/cURL/HAR import, AI test generation, scaffold, + the `apitestka_test_api` MCP tool) build request actions with + `utils/executor/request_action.build_request_action` → `["AT_test_api_method", {...}]`. - **Legacy CLI**: `python -m je_api_testka` with `-e/--execute_file`, `-d/--execute_dir`, `-c/--create_project` and `--execute_str`. On `win32`/`cygwin`/`msys`, `--execute_str` is decoded with `json.loads` twice. Errors print `repr(error)` to stderr and exit with code 1. diff --git a/docs/source/Eng/doc/diff/diff_doc.rst b/docs/source/Eng/doc/diff/diff_doc.rst index 3a3fcc4..df6c81a 100644 --- a/docs/source/Eng/doc/diff/diff_doc.rst +++ b/docs/source/Eng/doc/diff/diff_doc.rst @@ -69,7 +69,9 @@ Response time SLA ----------------- ``ResponseSLA`` carries ``max_ms`` and ``p95_ms`` thresholds; ``assert_sla`` -walks a list of records and raises ``APIAssertException`` on breach. +walks a list of records and raises ``APIAssertException`` on breach. Without +``records`` it checks the successes in the shared test record, and ``sla`` may also +be a plain mapping, which is the form JSON actions use. .. code-block:: python @@ -77,6 +79,11 @@ walks a list of records and raises ``APIAssertException`` on breach. sla = ResponseSLA(max_ms=2000, p95_ms=1500) assert_sla(records, sla) + assert_sla(sla={"max_ms": 2000}) # the run recorded so far + +.. code-block:: json + + [["AT_assert_sla", {"sla": {"max_ms": 2000, "p95_ms": 1500}}]] Executor commands ----------------- diff --git a/docs/source/Eng/doc/integrations/integrations_doc.rst b/docs/source/Eng/doc/integrations/integrations_doc.rst index e5f49e5..a4a50d4 100644 --- a/docs/source/Eng/doc/integrations/integrations_doc.rst +++ b/docs/source/Eng/doc/integrations/integrations_doc.rst @@ -42,6 +42,12 @@ Shell-quote-aware parser for ``-X``, ``-H``, ``-d`` / ``--data``. action = curl_to_action( "curl -X POST https://api/x -H 'Content-Type: application/json' -d '{\"a\":1}'" ) + # ["AT_test_api_method", {"http_method": "post", "test_url": "https://api/x", + # "headers": {...}, "json": {"a": 1}}] + +Every importer on this page returns actions in the ``[command, kwargs]`` form, so +``execute_action`` runs the result as it is. A JSON body (object or array) is sent as +``json``; any other body as ``data``. HAR import ---------- diff --git a/docs/source/Eng/doc/runner/runner_doc.rst b/docs/source/Eng/doc/runner/runner_doc.rst index 36092e8..9d52717 100644 --- a/docs/source/Eng/doc/runner/runner_doc.rst +++ b/docs/source/Eng/doc/runner/runner_doc.rst @@ -3,9 +3,10 @@ Runner ====== Three orthogonal helpers for organising large action lists. They all expect -the same action-list shape used by ``execute_action`` and strip -runner-specific metadata (``id``, ``depends_on``, ``tags``) before invoking -the underlying ``AT_`` function. +the same action-list shape used by ``execute_action``. Runner-specific metadata +(``id``, ``depends_on``, ``tags``) may stay in the kwargs: the executor strips it +before invoking the underlying ``AT_`` function, so filtered or ordered lists run +as they are. Parallel runner --------------- @@ -14,8 +15,8 @@ Parallel runner from je_api_testka.runner import run_actions_parallel - actions = [["AT_test_api_method_requests", {"http_method": "get", - "test_url": f"https://x.invalid/{i}"}] + actions = [["AT_test_api_method", {"http_method": "get", + "test_url": f"https://x.invalid/{i}"}] for i in range(20)] results = run_actions_parallel(actions, max_workers=8) diff --git a/docs/source/Zh/doc/diff/diff_doc.rst b/docs/source/Zh/doc/diff/diff_doc.rst index e1b24ea..0af5f81 100644 --- a/docs/source/Zh/doc/diff/diff_doc.rst +++ b/docs/source/Zh/doc/diff/diff_doc.rst @@ -67,7 +67,8 @@ operation：每個出現過的狀態碼各有一筆 response，查詢參數名 ------------ ``ResponseSLA`` 帶 ``max_ms`` 與 ``p95_ms`` 兩個門檻;``assert_sla`` 走過 -records,超過就丟 ``APIAssertException``。 +records,超過就丟 ``APIAssertException``。沒給 ``records`` 時檢查共用測試紀錄裡的成功紀錄; +``sla`` 也可以是一般的 mapping,也就是 JSON action 用的形式。 .. code-block:: python @@ -75,6 +76,11 @@ records,超過就丟 ``APIAssertException``。 sla = ResponseSLA(max_ms=2000, p95_ms=1500) assert_sla(records, sla) + assert_sla(sla={"max_ms": 2000}) # 目前為止錄下的執行 + +.. code-block:: json + + [["AT_assert_sla", {"sla": {"max_ms": 2000, "p95_ms": 1500}}]] Executor 命令 ------------- diff --git a/docs/source/Zh/doc/integrations/integrations_doc.rst b/docs/source/Zh/doc/integrations/integrations_doc.rst index 23ba77a..f9b9226 100644 --- a/docs/source/Zh/doc/integrations/integrations_doc.rst +++ b/docs/source/Zh/doc/integrations/integrations_doc.rst @@ -42,6 +42,11 @@ cURL → action action = curl_to_action( "curl -X POST https://api/x -H 'Content-Type: application/json' -d '{\"a\":1}'" ) + # ["AT_test_api_method", {"http_method": "post", "test_url": "https://api/x", + # "headers": {...}, "json": {"a": 1}}] + +本頁每個匯入工具都回傳 ``[command, kwargs]`` 形式的 action，``execute_action`` 可以直接執行。 +JSON 內容（物件或陣列）以 ``json`` 送出，其他內容以 ``data`` 送出。 HAR 匯入 -------- diff --git a/docs/source/Zh/doc/runner/runner_doc.rst b/docs/source/Zh/doc/runner/runner_doc.rst index 364ca95..057f552 100644 --- a/docs/source/Zh/doc/runner/runner_doc.rst +++ b/docs/source/Zh/doc/runner/runner_doc.rst @@ -3,8 +3,9 @@ Runner ====== 三個正交的工具,專門處理大型 action list。它們都吃 ``execute_action`` 同樣 -的 action 格式,並會在呼叫底層 ``AT_`` 函式前自動拿掉 runner 專用的 -metadata(``id``、``depends_on``、``tags``)。 +的 action 格式。runner 專用的 metadata(``id``、``depends_on``、``tags``)可以留在 +kwargs 裡:executor 在呼叫底層 ``AT_`` 函式前會自動拿掉,所以過濾或排序後的 list +可以直接執行。 平行 Runner ----------- @@ -13,8 +14,8 @@ metadata(``id``、``depends_on``、``tags``)。 from je_api_testka.runner import run_actions_parallel - actions = [["AT_test_api_method_requests", {"http_method": "get", - "test_url": f"https://x.invalid/{i}"}] + actions = [["AT_test_api_method", {"http_method": "get", + "test_url": f"https://x.invalid/{i}"}] for i in range(20)] results = run_actions_parallel(actions, max_workers=8) diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index a50791c..2df32c7 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -13,3 +13,13 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Tests**: `test/test_spec/test_records_to_openapi.py` (8 new, 2 of them regressions that fail on the old code), `test/test_spec/test_openapi_export.py` (new, including a round trip through `generate_json_report` against the local mock), `test/test_cli/test_cli_main.py` (5 new), `test/test_cli/test_completion.py` (parser parity), `test/test_mcp_server/test_tool_definitions.py` (3 new). - **Result / numbers**: `pytest` 447 passed, 2 skipped (418 before); `sphinx -W` builds without warnings. - **Files**: `je_api_testka/spec/{records_to_openapi,openapi_export,__init__}.py`, `je_api_testka/cli/{cli_main,completion}.py`, `je_api_testka/mcp_server/tool_definitions.py`, `je_api_testka/utils/executor/action_executor.py`; README ×3, `docs/source/{Eng,Zh}/doc/{cli,diff,mcp}`, `architecture.md` §2–§3. + +## U-20261001-02 · 2026-10-01 · Imported, scaffolded and AI-generated actions run as they are · #bugfix #executor + +- **What**: every converter produced actions the executor could not run. OpenAPI/Postman import (`apitestka import`), `curl_to_action`, `convert_har` and the deterministic `generate_tests_from_openapi` fallback returned `{"AT_test_api_method_requests": {...}}` dicts, which `execute_action` rejects (`KeyError(0)`). `scaffold` and the `apitestka_test_api` MCP tool used the list form, but that command name was never registered (`'NoneType' object is not callable`). All of them now go through the new `utils/executor/request_action.build_request_action`, which returns `["AT_test_api_method", {...}]` and sends a dict/list body as `json` and any other body as `data` (Postman raw text used to go out as `json`). +- **Executor**: `_execute_event` strips runner metadata (`id`, `depends_on`, `tags`) before the call, as only `run_actions_parallel` did. A tagged action used to reach `requests` as an unknown keyword. No registered command takes those names. +- **`assert_sla`**: `sla` may be a mapping (`{"max_ms": ..., "p95_ms": ...}`) and `records` defaults to the shared record's successes, so `["AT_assert_sla", {"sla": {...}}]` works from JSON. Before, JSON could only pass a dict, which failed on `sla.max_ms`, and `records` was required. The scaffold now emits that form instead of `"records": []`, which always passed. +- **Compatibility**: `curl_to_action` now returns a list and `convert_har`, `convert_openapi`, `convert_postman`, `convert_spec_file` return lists of lists. The old shapes were never runnable. No other workspace repo imports these functions. +- **Tests**: new `test/test_utils/test_generated_actions_run.py` runs each converter's output against the local mock and checks for success records and no errors; 3 new cases in `test/test_diff/test_sla_check.py`. Checked in a worktree of `4810743`: 11 of the new tests fail on the old code. The existing tests that pinned the dict shape now assert the list form. +- **Result / numbers**: `pytest` 460 passed, 2 skipped (447 before); `sphinx -W` clean. +- **Files**: `je_api_testka/utils/executor/{request_action,action_executor}.py`, `je_api_testka/{cli/import_specs,cli/scaffold_test,integrations/curl_import,integrations/har_import,ai/test_generator,mcp_server/tool_definitions,diff/sla_check}.py`; README ×3, `docs/source/{Eng,Zh}/doc/{integrations,diff,runner}`, `architecture.md` §3. diff --git a/docs/updates/README.md b/docs/updates/README.md index 798bffe..6958192 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261001-02 | 2026-10-01 | Imported, scaffolded and AI-generated actions run as they are | #bugfix #executor | [2026-10](2026-10.md) | | U-20261001-01 | 2026-10-01 | records_to_openapi reachable from the CLI and MCP | #done #openapi #cli #mcp | [2026-10](2026-10.md) | | U-20260925-03 | 2026-09-25 | Python classifiers list every version CI tests | #packaging #tests | [2026-09](2026-09.md) | | U-20260925-02 | 2026-09-25 | dev_requirements.txt pins the package's PySide6 | #deps #tests | [2026-09](2026-09.md) | @@ -87,5 +88,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 1 | +| [2026-10.md](2026-10.md) | 2026-10 | 2 | | [2026-09.md](2026-09.md) | 2026-09 | 23 | diff --git a/je_api_testka/ai/test_generator.py b/je_api_testka/ai/test_generator.py index ab4f517..3ff8546 100644 --- a/je_api_testka/ai/test_generator.py +++ b/je_api_testka/ai/test_generator.py @@ -10,13 +10,14 @@ from typing import Any, List from je_api_testka.ai.backend import NoOpAIBackend, ai_backend +from je_api_testka.utils.executor.request_action import build_request_action from je_api_testka.utils.logging.loggin_instance import apitestka_logger HTTP_VERBS = ("get", "put", "post", "patch", "delete", "options", "head") -def _deterministic_actions(spec: dict) -> List[dict]: - actions: List[dict] = [] +def _deterministic_actions(spec: dict) -> List[list]: + actions: List[list] = [] base = "" servers = spec.get("servers") or [] if servers: @@ -25,18 +26,17 @@ def _deterministic_actions(spec: dict) -> List[dict]: for verb in HTTP_VERBS: if not (item or {}).get(verb): continue - actions.append({ - "AT_test_api_method_requests": { - "http_method": verb, - "test_url": f"{base}{path}", - "result_check_dict": {"status_code": 200}, - }, - }) + actions.append(build_request_action(verb, f"{base}{path}", result_check_dict={"status_code": 200})) return actions -def generate_tests_from_openapi(spec: dict) -> List[dict]: - """Return action dicts via the AI backend if configured, otherwise deterministic fallback.""" +def generate_tests_from_openapi(spec: dict) -> List[Any]: + """ + Return executor actions via the AI backend if configured, otherwise a deterministic fallback. + + The fallback is one ``["AT_test_api_method", {...}]`` happy-path action per operation. + A backend reply that is a JSON list is returned as it is. + """ backend = ai_backend() if isinstance(backend, NoOpAIBackend): return _deterministic_actions(spec) diff --git a/je_api_testka/cli/import_specs.py b/je_api_testka/cli/import_specs.py index ac74ff7..b8ed202 100644 --- a/je_api_testka/cli/import_specs.py +++ b/je_api_testka/cli/import_specs.py @@ -1,8 +1,8 @@ """ OpenAPI 3.x and Postman 2.1 collection importers. -Both functions return a list of action dictionaries usable with the existing -JSON-driven executor (``execute_action``). +Both functions return a list of ``["AT_test_api_method", {...}]`` actions that +``execute_action`` runs as they are. """ from __future__ import annotations @@ -11,39 +11,26 @@ from typing import Iterable, List from je_api_testka.utils.exception.exceptions import APITesterException +from je_api_testka.utils.executor.request_action import build_request_action from je_api_testka.utils.logging.loggin_instance import apitestka_logger UNSUPPORTED_SPEC_FORMAT: str = "Unsupported spec format. Choose 'openapi' or 'postman'." HTTP_VERBS: tuple = ("get", "put", "post", "patch", "delete", "options", "head") -def _action_for_request(method: str, url: str, headers: dict = None, body: dict = None) -> dict: - payload = { - "AT_test_api_method_requests": { - "http_method": method.lower(), - "test_url": url, - } - } - if headers: - payload["AT_test_api_method_requests"]["headers"] = headers - if body is not None: - payload["AT_test_api_method_requests"]["json"] = body - return payload - - -def convert_openapi(spec: dict, base_url: str = "") -> List[dict]: - """Convert an OpenAPI 3.x dict into a list of executor action dicts.""" +def convert_openapi(spec: dict, base_url: str = "") -> List[list]: + """Convert an OpenAPI 3.x dict into a list of executor actions, one per operation.""" apitestka_logger.info("import_specs convert_openapi") if not base_url: servers = spec.get("servers") or [] base_url = servers[0].get("url", "") if servers else "" - actions: List[dict] = [] + actions: List[list] = [] for path, item in (spec.get("paths") or {}).items(): for verb in HTTP_VERBS: operation = item.get(verb) if not operation: continue - actions.append(_action_for_request(verb, f"{base_url}{path}")) + actions.append(build_request_action(verb, f"{base_url}{path}")) return actions @@ -55,10 +42,10 @@ def _iterate_postman_items(items: Iterable[dict]) -> Iterable[dict]: yield entry -def convert_postman(collection: dict) -> List[dict]: - """Convert a Postman 2.1 collection dict into action dicts.""" +def convert_postman(collection: dict) -> List[list]: + """Convert a Postman 2.1 collection dict into executor actions; a non-JSON raw body is sent as ``data``.""" apitestka_logger.info("import_specs convert_postman") - actions: List[dict] = [] + actions: List[list] = [] for entry in _iterate_postman_items(collection.get("item") or []): request = entry["request"] method = request.get("method", "GET") @@ -74,11 +61,11 @@ def convert_postman(collection: dict) -> List[dict]: body = json.loads(body_field["raw"]) except json.JSONDecodeError: body = body_field["raw"] - actions.append(_action_for_request(method, url, headers=headers or None, body=body)) + actions.append(build_request_action(method, url, headers=headers, body=body)) return actions -def convert_spec_file(input_path: str, spec_format: str) -> List[dict]: +def convert_spec_file(input_path: str, spec_format: str) -> List[list]: """Read ``input_path`` and dispatch to the matching converter.""" text = Path(input_path).read_text(encoding="utf-8") document = json.loads(text) diff --git a/je_api_testka/cli/scaffold_test.py b/je_api_testka/cli/scaffold_test.py index 6c7a95f..5915538 100644 --- a/je_api_testka/cli/scaffold_test.py +++ b/je_api_testka/cli/scaffold_test.py @@ -10,17 +10,14 @@ from pathlib import Path from typing import List +from je_api_testka.utils.executor.request_action import build_request_action + def scaffold_action_list(url: str, method: str = "GET") -> List[list]: - """Return a starter action list.""" + """Return a starter action list: one tagged smoke request, then an SLA check over the recorded run.""" return [ - ["AT_test_api_method_requests", { - "http_method": method.lower(), - "test_url": url, - "result_check_dict": {"status_code": 200}, - "tags": ["smoke"], - }], - ["AT_assert_sla", {"records": [], "sla": {"max_ms": 2000, "p95_ms": 1500}}], + build_request_action(method, url, result_check_dict={"status_code": 200}, tags=["smoke"]), + ["AT_assert_sla", {"sla": {"max_ms": 2000, "p95_ms": 1500}}], ] diff --git a/je_api_testka/diff/sla_check.py b/je_api_testka/diff/sla_check.py index b30ba11..dcbc213 100644 --- a/je_api_testka/diff/sla_check.py +++ b/je_api_testka/diff/sla_check.py @@ -2,18 +2,22 @@ Response-time SLA assertions. These do not load-test the endpoint - they only assert that a single response -record meets a target. Multiple records can be batched via :func:`assert_sla`. +record meets a target. Multiple records can be batched via :func:`assert_sla`, +which checks the shared test record when no records are given. """ from __future__ import annotations from dataclasses import dataclass -from typing import Iterable, Sequence +from typing import Iterable, Mapping, Optional, Sequence, Union from je_api_testka.utils.exception.exceptions import APIAssertException from je_api_testka.utils.logging.loggin_instance import apitestka_logger +from je_api_testka.utils.test_record.test_record_class import test_record_instance DEFAULT_TIMEOUT_MS: float = 5000.0 SECONDS_TO_MS: int = 1000 +# A numeric ``elapsed`` below this is taken as seconds, otherwise as milliseconds. +ELAPSED_SECONDS_LIMIT: float = 1000.0 @dataclass @@ -27,7 +31,7 @@ class ResponseSLA: def _record_ms(record: dict) -> float: elapsed = record.get("elapsed") if isinstance(elapsed, (int, float)): - return float(elapsed) * SECONDS_TO_MS if elapsed < 1000 else float(elapsed) + return float(elapsed) * SECONDS_TO_MS if elapsed < ELAPSED_SECONDS_LIMIT else float(elapsed) seconds = record.get("request_time_sec") if isinstance(seconds, (int, float)): return float(seconds) * SECONDS_TO_MS @@ -42,9 +46,20 @@ def _percentile(values: Sequence[float], percentile: float) -> float: return ordered[min(rank, len(ordered) - 1)] -def assert_sla(records: Iterable[dict], sla: ResponseSLA) -> None: - """Raise :class:`APIAssertException` if records breach ``sla``.""" +def assert_sla(records: Optional[Iterable[dict]] = None, + sla: Union[ResponseSLA, Mapping[str, float], None] = None) -> None: + """ + Raise :class:`APIAssertException` if records breach ``sla``. + + ``records`` defaults to the successes in the shared test record. ``sla`` may be a + :class:`ResponseSLA` or a mapping with ``max_ms`` / ``p95_ms`` (the JSON action form); + ``None`` uses the defaults. + """ apitestka_logger.info(f"sla_check assert_sla sla: {sla}") + if not isinstance(sla, ResponseSLA): + sla = ResponseSLA(**(sla or {})) + if records is None: + records = list(test_record_instance.test_record_list) timings = [_record_ms(record) for record in records] if not timings: return diff --git a/je_api_testka/integrations/curl_import.py b/je_api_testka/integrations/curl_import.py index 5445ff8..54e5952 100644 --- a/je_api_testka/integrations/curl_import.py +++ b/je_api_testka/integrations/curl_import.py @@ -1,5 +1,5 @@ """ -Convert a ``curl`` command line into an executor action dict. +Convert a ``curl`` command line into an executor action. Supports the most common flags: ``-X``, ``-H``, ``-d`` / ``--data``, and the implicit GET / POST behaviour. Bash quoting is parsed via ``shlex``. @@ -11,6 +11,7 @@ from typing import Dict, Optional, Tuple from je_api_testka.utils.exception.exceptions import APITesterException +from je_api_testka.utils.executor.request_action import build_request_action CURL_TOKEN: str = "curl" _METHOD_FLAGS = ("-X", "--request") @@ -65,15 +66,10 @@ def _parse_tokens(tokens: list) -> Tuple[str, str, Dict[str, str], Optional[obje return method, state["url"], state["headers"], state["body"] -def curl_to_action(curl_command: str) -> dict: - """Return ``{"AT_test_api_method_requests": {...}}`` for the given curl line.""" +def curl_to_action(curl_command: str) -> list: + """Return ``["AT_test_api_method", {...}]`` for the given curl line.""" tokens = shlex.split(curl_command) if not tokens or tokens[0] != CURL_TOKEN: raise APITesterException("input does not start with 'curl'") method, url, headers, body = _parse_tokens(tokens) - payload: dict = {"http_method": method.lower(), "test_url": url} - if headers: - payload["headers"] = headers - if body is not None: - payload["json" if isinstance(body, (dict, list)) else "data"] = body - return {"AT_test_api_method_requests": payload} + return build_request_action(method, url, headers=headers, body=body) diff --git a/je_api_testka/integrations/har_import.py b/je_api_testka/integrations/har_import.py index 09c1af3..0006f4d 100644 --- a/je_api_testka/integrations/har_import.py +++ b/je_api_testka/integrations/har_import.py @@ -10,6 +10,8 @@ from pathlib import Path from typing import List, Optional +from je_api_testka.utils.executor.request_action import build_request_action + def _decode_text_body(text: Optional[str]) -> Optional[object]: if not text: @@ -20,18 +22,7 @@ def _decode_text_body(text: Optional[str]) -> Optional[object]: return text -def _build_payload(method: str, url: str, headers: dict, body_value: object) -> dict: - payload: dict = {"http_method": method.lower(), "test_url": url} - if headers: - payload["headers"] = headers - if isinstance(body_value, (dict, list)): - payload["json"] = body_value - elif body_value is not None: - payload["data"] = body_value - return payload - - -def _entry_to_action(entry: dict) -> Optional[dict]: +def _entry_to_action(entry: dict) -> Optional[list]: request = entry.get("request") or {} url = request.get("url") if not url: @@ -41,11 +32,11 @@ def _entry_to_action(entry: dict) -> Optional[dict]: for item in request.get("headers", []) if "name" in item and "value" in item} body_value = _decode_text_body((request.get("postData") or {}).get("text")) - return {"AT_test_api_method_requests": _build_payload(method, url, headers, body_value)} + return build_request_action(method, url, headers=headers, body=body_value) -def convert_har(file_path: str) -> List[dict]: - """Return a list of action dicts parsed from a HAR JSON document.""" +def convert_har(file_path: str) -> List[list]: + """Return a list of executor actions parsed from a HAR JSON document.""" document = json.loads(Path(file_path).read_text(encoding="utf-8")) entries = (document.get("log") or {}).get("entries") or [] actions = [action for action in (_entry_to_action(entry) for entry in entries) diff --git a/je_api_testka/mcp_server/tool_definitions.py b/je_api_testka/mcp_server/tool_definitions.py index 8cf18e6..1a0c508 100644 --- a/je_api_testka/mcp_server/tool_definitions.py +++ b/je_api_testka/mcp_server/tool_definitions.py @@ -23,6 +23,7 @@ from je_api_testka.spec.records_to_openapi import DEFAULT_SPEC_TITLE, DEFAULT_SPEC_VERSION from je_api_testka.utils.executor.action_executor import execute_action from je_api_testka.utils.exception.exceptions import APITesterException +from je_api_testka.utils.executor.request_action import build_request_action from je_api_testka.utils.generate_report.markdown_report import render_markdown from je_api_testka.utils.test_record.test_record_class import test_record_instance @@ -46,17 +47,11 @@ def _handle_run_action(arguments: Dict[str, Any]) -> Any: def _handle_test_api(arguments: Dict[str, Any]) -> Any: - method = arguments.get("method", "GET") - url = arguments["url"] - timeout = arguments.get("timeout", 30) - body = arguments.get("body") - headers = arguments.get("headers") - payload: dict = {"http_method": method.lower(), "test_url": url, "timeout": timeout} - if headers: - payload["headers"] = headers - if body is not None: - payload["json"] = body - record = execute_action([["AT_test_api_method_requests", payload]]) + action = build_request_action( + arguments.get("method", "GET"), arguments["url"], headers=arguments.get("headers"), + body=arguments.get("body"), timeout=arguments.get("timeout", 30), + ) + record = execute_action([action]) return {key: _jsonable(value) for key, value in record.items()} diff --git a/je_api_testka/utils/executor/action_executor.py b/je_api_testka/utils/executor/action_executor.py index cc46a15..f21365d 100644 --- a/je_api_testka/utils/executor/action_executor.py +++ b/je_api_testka/utils/executor/action_executor.py @@ -21,6 +21,7 @@ from je_api_testka.integrations.notify import notify_via_webhook from je_api_testka.requests_wrapper.request_method import test_api_method_requests from je_api_testka.runner.dependency_runner import order_actions +from je_api_testka.runner.metadata import strip_runner_metadata from je_api_testka.runner.parallel_runner import run_actions_parallel from je_api_testka.runner.tag_filter import filter_actions_by_tag from je_api_testka.security.auth_helpers import ( @@ -192,6 +193,8 @@ def _execute_event(self, action: list) -> Any: :return: 事件回傳值 / Event return value """ apitestka_logger.info(f"Executor _execute_event action: {action}") + # Runner metadata (id, depends_on, tags) orders and filters actions; the command never sees it. + action = strip_runner_metadata(action) event: Callable = self.event_dict.get(action[0]) if len(action) == 2: if isinstance(action[1], dict): diff --git a/je_api_testka/utils/executor/request_action.py b/je_api_testka/utils/executor/request_action.py new file mode 100644 index 0000000..476403a --- /dev/null +++ b/je_api_testka/utils/executor/request_action.py @@ -0,0 +1,33 @@ +""" +Build one requests-backend call in executor action form. + +Every converter that turns outside input (OpenAPI, Postman, cURL, HAR, AI +output, scaffolds) into actions goes through :func:`build_request_action`, so +they all emit the ``[command, kwargs]`` shape and a command name that +``Executor.event_dict`` registers. +""" +from __future__ import annotations + +from typing import Mapping, Optional + +REQUEST_COMMAND: str = "AT_test_api_method" + + +def build_request_action(http_method: str, test_url: str, headers: Optional[Mapping[str, str]] = None, + body: object = None, **options: object) -> list: + """ + Return ``["AT_test_api_method", {...}]`` for one request. + + A ``dict`` or ``list`` body is sent as ``json``; any other non-``None`` body + as ``data``. ``options`` (``result_check_dict``, ``timeout``, runner ``tags``...) + are copied into the kwargs unchanged. + """ + kwargs: dict = {"http_method": http_method.lower(), "test_url": test_url} + if headers: + kwargs["headers"] = dict(headers) + if isinstance(body, (dict, list)): + kwargs["json"] = body + elif body is not None: + kwargs["data"] = body + kwargs.update(options) + return [REQUEST_COMMAND, kwargs] diff --git a/test/test_ai/test_test_generator.py b/test/test_ai/test_test_generator.py index aec0277..a7c7624 100644 --- a/test/test_ai/test_test_generator.py +++ b/test/test_ai/test_test_generator.py @@ -21,14 +21,14 @@ def test_noop_backend_uses_deterministic_fallback(): set_ai_backend(NoOpAIBackend()) actions = generate_tests_from_openapi(_SPEC) - methods = sorted(a["AT_test_api_method_requests"]["http_method"] for a in actions) + methods = sorted(a[1]["http_method"] for a in actions) assert methods == ["get", "post"] def test_real_backend_response_used_when_valid_json(): - set_ai_backend(StaticAIBackend(response=json.dumps([{"AT_test_api_method_requests": {"http_method": "head", "test_url": "x"}}]))) + set_ai_backend(StaticAIBackend(response=json.dumps([["AT_test_api_method", {"http_method": "head", "test_url": "x"}]]))) actions = generate_tests_from_openapi(_SPEC) - assert actions[0]["AT_test_api_method_requests"]["http_method"] == "head" + assert actions[0] == ["AT_test_api_method", {"http_method": "head", "test_url": "x"}] set_ai_backend(NoOpAIBackend()) diff --git a/test/test_cli/test_import_specs.py b/test/test_cli/test_import_specs.py index 30d343a..293ad12 100644 --- a/test/test_cli/test_import_specs.py +++ b/test/test_cli/test_import_specs.py @@ -25,9 +25,9 @@ def test_convert_openapi_extracts_path_methods(): }, } actions = convert_openapi(spec) - methods = sorted(a["AT_test_api_method_requests"]["http_method"] for a in actions) + methods = sorted(a[1]["http_method"] for a in actions) assert methods == ["delete", "get", "post"] - urls = {a["AT_test_api_method_requests"]["test_url"] for a in actions} + urls = {a[1]["test_url"] for a in actions} assert "https://api.example.invalid/users" in urls @@ -51,7 +51,7 @@ def test_convert_postman_handles_nested_folders(): } actions = convert_postman(collection) assert len(actions) == 1 - body = actions[0]["AT_test_api_method_requests"] + body = actions[0][1] assert body["http_method"] == "get" assert body["headers"] == {"X-Test": "1"} diff --git a/test/test_cli/test_scaffold.py b/test/test_cli/test_scaffold.py index b52a4de..96e4e38 100644 --- a/test/test_cli/test_scaffold.py +++ b/test/test_cli/test_scaffold.py @@ -8,7 +8,7 @@ def test_scaffold_action_list_includes_smoke_check(): actions = scaffold_action_list("https://x.invalid", method="POST") - assert actions[0][0] == "AT_test_api_method_requests" + assert actions[0][0] == "AT_test_api_method" assert actions[0][1]["http_method"] == "post" assert actions[0][1]["tags"] == ["smoke"] diff --git a/test/test_diff/test_sla_check.py b/test/test_diff/test_sla_check.py index 504c848..f139e6e 100644 --- a/test/test_diff/test_sla_check.py +++ b/test/test_diff/test_sla_check.py @@ -5,6 +5,8 @@ from je_api_testka.diff.sla_check import ResponseSLA, assert_sla from je_api_testka.utils.exception.exceptions import APIAssertException +from je_api_testka.utils.executor.action_executor import execute_action +from je_api_testka.utils.test_record.test_record_class import test_record_instance def test_assert_sla_passes_under_thresholds(): @@ -36,3 +38,21 @@ def test_diff_payloads_via_executor(): ]) diff_value = next(iter(record.values())) assert diff_value.changed == {"a": (1, 2)} + + +def test_assert_sla_accepts_json_mapping(): + # Regression: JSON actions can only pass a dict, which used to fail on sla.max_ms. + with pytest.raises(APIAssertException): + assert_sla([{"request_time_sec": 1.0}], {"max_ms": 500, "p95_ms": 2000}) + + +def test_assert_sla_defaults_to_the_shared_record(): + test_record_instance.test_record_list.append({"request_time_sec": 1.0}) + with pytest.raises(APIAssertException): + assert_sla(sla={"max_ms": 500}) + + +def test_assert_sla_via_executor_checks_recorded_run(): + test_record_instance.test_record_list.append({"request_time_sec": 1.0}) + record = execute_action([["AT_assert_sla", {"sla": {"max_ms": 500}}]]) + assert "APIAssertException" in next(iter(record.values())) diff --git a/test/test_executor_integration.py b/test/test_executor_integration.py index f6c8086..70867e3 100644 --- a/test/test_executor_integration.py +++ b/test/test_executor_integration.py @@ -90,7 +90,7 @@ def test_curl_import_then_dispatch(tmp_path): ["AT_curl_to_action", {"curl_command": "curl https://api.invalid/v1/health"}], ]) action = next(iter(record.values())) - body = action["AT_test_api_method_requests"] + body = action[1] assert body["http_method"] == "get" assert body["test_url"] == "https://api.invalid/v1/health" diff --git a/test/test_integrations/test_curl_import.py b/test/test_integrations/test_curl_import.py index ea6b235..af6c188 100644 --- a/test/test_integrations/test_curl_import.py +++ b/test/test_integrations/test_curl_import.py @@ -9,7 +9,8 @@ def test_simple_get(): action = curl_to_action("curl https://api.invalid/x") - body = action["AT_test_api_method_requests"] + assert action[0] == "AT_test_api_method" + body = action[1] assert body["http_method"] == "get" assert body["test_url"] == "https://api.invalid/x" @@ -19,7 +20,7 @@ def test_post_with_json_body(): "curl -X POST https://api.invalid/x -H 'Content-Type: application/json' " "-d '{\"a\":1}'" ) - body = action["AT_test_api_method_requests"] + body = action[1] assert body["http_method"] == "post" assert body["json"] == {"a": 1} assert body["headers"]["Content-Type"] == "application/json" @@ -27,13 +28,13 @@ def test_post_with_json_body(): def test_post_with_text_body_falls_back_to_data(): action = curl_to_action("curl -X POST https://api.invalid/x -d 'plain text'") - body = action["AT_test_api_method_requests"] + body = action[1] assert body["data"] == "plain text" def test_implicit_post_when_data_present(): action = curl_to_action("curl https://api.invalid/x -d '{\"a\":1}'") - assert action["AT_test_api_method_requests"]["http_method"] == "post" + assert action[1]["http_method"] == "post" def test_non_curl_input_raises(): diff --git a/test/test_integrations/test_har_import.py b/test/test_integrations/test_har_import.py index 99f92a5..6ad7132 100644 --- a/test/test_integrations/test_har_import.py +++ b/test/test_integrations/test_har_import.py @@ -32,10 +32,10 @@ def test_convert_har_basic(tmp_path): path.write_text(json.dumps(har), encoding="utf-8") actions = convert_har(str(path)) assert len(actions) == 2 - first = actions[0]["AT_test_api_method_requests"] + first = actions[0][1] assert first["http_method"] == "get" assert first["headers"]["Accept"] == "application/json" - second = actions[1]["AT_test_api_method_requests"] + second = actions[1][1] assert second["json"] == {"name": "alice"} diff --git a/test/test_mcp_server/test_tool_definitions.py b/test/test_mcp_server/test_tool_definitions.py index a54a129..b9f5332 100644 --- a/test/test_mcp_server/test_tool_definitions.py +++ b/test/test_mcp_server/test_tool_definitions.py @@ -42,7 +42,7 @@ def test_get_records_tool_returns_lists(): def test_curl_to_action_tool(): result = dispatch_tool("apitestka_curl_to_action", {"curl_command": "curl https://x.invalid"}) - assert "AT_test_api_method_requests" in result + assert result[0] == "AT_test_api_method" def test_render_markdown_tool_returns_string(): diff --git a/test/test_utils/test_generated_actions_run.py b/test/test_utils/test_generated_actions_run.py new file mode 100644 index 0000000..49a2ee0 --- /dev/null +++ b/test/test_utils/test_generated_actions_run.py @@ -0,0 +1,89 @@ +""" +Actions produced by the converters must run through ``execute_action`` as they are. + +Regression: OpenAPI/Postman import, cURL and HAR import, AI test generation, +``scaffold`` and the ``apitestka_test_api`` MCP tool emitted +``AT_test_api_method_requests`` (never registered) and, except scaffold and +MCP, the dict shape ``{"AT_...": {...}}`` the executor cannot run. +""" +from __future__ import annotations + +import json + +from je_api_testka.ai.backend import NoOpAIBackend, set_ai_backend +from je_api_testka.ai.test_generator import generate_tests_from_openapi +from je_api_testka.cli.import_specs import convert_openapi, convert_postman +from je_api_testka.cli.scaffold_test import scaffold_action_list +from je_api_testka.integrations.curl_import import curl_to_action +from je_api_testka.integrations.har_import import convert_har +from je_api_testka.mcp_server.tool_definitions import dispatch_tool +from je_api_testka.utils.executor.action_executor import execute_action, executor +from je_api_testka.utils.executor.request_action import REQUEST_COMMAND, build_request_action +from je_api_testka.utils.test_record.test_record_class import test_record_instance + + +def _run_cleanly(actions: list) -> None: + record = execute_action(actions) + errors = [value for value in record.values() if isinstance(value, str) and "Error(" in value] + assert errors == [] + assert test_record_instance.error_record_list == [] + assert len(test_record_instance.test_record_list) == sum(1 for a in actions if a[0] == REQUEST_COMMAND) + + +def test_request_command_is_registered(): + assert REQUEST_COMMAND in executor.event_dict + + +def test_build_request_action_body_kinds(): + assert build_request_action("GET", "u") == [REQUEST_COMMAND, {"http_method": "get", "test_url": "u"}] + assert build_request_action("POST", "u", body={"a": 1})[1]["json"] == {"a": 1} + assert build_request_action("POST", "u", body="a=1")[1]["data"] == "a=1" + action = build_request_action("GET", "u", headers={"X": "1"}, timeout=3) + assert action[1]["headers"] == {"X": "1"} and action[1]["timeout"] == 3 + + +def test_openapi_import_runs(mock_url): + spec = {"servers": [{"url": mock_url}], "paths": {"/get": {"get": {"responses": {}}}}} + _run_cleanly(convert_openapi(spec)) + + +def test_postman_import_runs_and_sends_text_as_data(mock_url): + collection = {"item": [{"request": { + "method": "POST", "url": {"raw": f"{mock_url}/post"}, "body": {"mode": "raw", "raw": "a=1"}, + }}]} + actions = convert_postman(collection) + assert actions[0][1]["data"] == "a=1" + _run_cleanly(actions) + + +def test_curl_import_runs(mock_url): + _run_cleanly([curl_to_action(f"curl {mock_url}/get -H 'Accept: application/json'")]) + + +def test_har_import_runs(mock_url, tmp_path): + har = {"log": {"entries": [{"request": { + "method": "POST", "url": f"{mock_url}/post", "headers": [], "postData": {"text": '{"a": 1}'}, + }}]}} + path = tmp_path / "t.har" + path.write_text(json.dumps(har), encoding="utf-8") + _run_cleanly(convert_har(str(path))) + + +def test_ai_fallback_actions_run(mock_url): + set_ai_backend(NoOpAIBackend()) + spec = {"servers": [{"url": mock_url}], "paths": {"/get": {"get": {"responses": {}}}}} + _run_cleanly(generate_tests_from_openapi(spec)) + + +def test_scaffold_runs_including_tags_and_sla(mock_url): + # The request carries runner tags; the executor strips them before calling requests. + _run_cleanly(scaffold_action_list(f"{mock_url}/get")) + + +def test_runner_metadata_is_stripped_before_the_call(mock_url): + _run_cleanly([build_request_action("GET", f"{mock_url}/get", id="a", depends_on=[], tags=["smoke"])]) + + +def test_mcp_test_api_tool_sends_the_request(mock_url): + dispatch_tool("apitestka_test_api", {"url": f"{mock_url}/get", "timeout": 30}) + assert len(test_record_instance.test_record_list) == 1 From 39557ec7a7489f39734f75d64684636ddcae6b8a Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 00:56:44 +0800 Subject: [PATCH 10/43] feat: serve WebSocket and gRPC endpoints from the mock server WebSocketMockServer runs scripted routes (greeting, reply map, {{message}} fallback, handler) in a background thread and refuses unknown paths with 404. GrpcStubServer gains fixed unary responses, server streams, status-code errors, request capture and port 0. JSON actions drive one of each through AT_mock_start/stop_*_server and AT_mock_*_received, and `apitestka mock --config` reads http, websocket and grpc sections from one file. Adds the grpc extra and puts websockets and grpcio in the CI lock so these tests run in CI. --- .github/requirements/ci.in | 4 +- .github/requirements/ci.txt | 325 ++++++++++++++++++ README.md | 42 +++ README/README_zh-CN.md | 41 +++ README/README_zh-TW.md | 41 +++ architecture.md | 17 +- dev.toml | 1 + docs/source/Eng/doc/cli/cli_doc.rst | 4 +- .../Eng/doc/mock_server/mock_server_doc.rst | 81 +++++ docs/source/Zh/doc/cli/cli_doc.rst | 4 +- .../Zh/doc/mock_server/mock_server_doc.rst | 77 +++++ docs/updates/2026-10.md | 14 + docs/updates/README.md | 3 +- je_api_testka/cli/cli_main.py | 12 +- .../utils/executor/action_executor.py | 15 + je_api_testka/utils/mock_server/grpc_stub.py | 206 ++++++++--- .../utils/mock_server/mock_config.py | 113 ++++++ .../utils/mock_server/protocol_mocks.py | 114 ++++++ .../utils/mock_server/websocket_mock.py | 165 +++++++++ progress.md | 1 - pyproject.toml | 1 + test/test_executor_integration.py | 2 + test/test_mock_server/test_grpc_stub.py | 99 +++++- test/test_mock_server/test_mock_config.py | 101 ++++++ test/test_mock_server/test_protocol_mocks.py | 82 +++++ test/test_mock_server/test_websocket_mock.py | 109 ++++++ uv.lock | 81 ++++- 27 files changed, 1696 insertions(+), 59 deletions(-) create mode 100644 je_api_testka/utils/mock_server/mock_config.py create mode 100644 je_api_testka/utils/mock_server/protocol_mocks.py create mode 100644 je_api_testka/utils/mock_server/websocket_mock.py create mode 100644 test/test_mock_server/test_mock_config.py create mode 100644 test/test_mock_server/test_protocol_mocks.py create mode 100644 test/test_mock_server/test_websocket_mock.py diff --git a/.github/requirements/ci.in b/.github/requirements/ci.in index 391064a..9235366 100644 --- a/.github/requirements/ci.in +++ b/.github/requirements/ci.in @@ -1,5 +1,5 @@ # What the CI test job installs; ci.txt is generated from this file (the command is in ci.yml). -# Runtime dependencies (pyproject.toml), the gui and mcp extras, and the test tools. +# Runtime dependencies (pyproject.toml), the gui, mcp, websocket and grpc extras, and the test tools. requests Flask httpx @@ -7,6 +7,8 @@ defusedxml PySide6==6.11.2 qt-material mcp>=1.28.1,<3 +websockets>=12.0 +grpcio>=1.60 pytest pytest-asyncio pytest-timeout diff --git a/.github/requirements/ci.txt b/.github/requirements/ci.txt index 2444565..357cf67 100644 --- a/.github/requirements/ci.txt +++ b/.github/requirements/ci.txt @@ -382,6 +382,69 @@ flask==3.1.3 \ --hash=sha256:0ef0e52b8a9cd932855379197dd8f94047b359ca0a78695144304cb45f87c9eb \ --hash=sha256:f4bcbefc124291925f1a26446da31a5178f9483862233b23c0c96a20701f670c # via -r .github/requirements/ci.in +grpcio==1.84.0 \ + --hash=sha256:026d757df86c5b7a41de8200b9a2cda454aaa5004cb0c7e3374c66eb82f61499 \ + --hash=sha256:06619ba1515e5ee69fb2a514e95dd8be05ce74cb3928d5b34f87f87c86fe3c27 \ + --hash=sha256:08735e3d08d24ab3132cf87e2e5dea8746cabcc7d676c2b0b7362f195feef9d9 \ + --hash=sha256:0d532ade4486dad9b302ffa4d4683d67561051c26d17c4023322845e9fa10140 \ + --hash=sha256:158c1c11cfb61b4849c3caf4d52de6f5ecd376e14446feb4a90dc95a90d616f5 \ + --hash=sha256:15bb76489e337fc492685c9758e2fd4d4ab516b901ad830dc5a91987decf00be \ + --hash=sha256:19aaf172fc2edbefccce3f6e92c5150975dbe56c45744e9e87cf72ebdf85bfbe \ + --hash=sha256:209414080da8c20af94df1395b635da52dd57b5edc9e917e1deca0dc1c4bb55e \ + --hash=sha256:210e4c32f907045eb8158273e60c6ab69a3947697df6245dbda381f26c59485b \ + --hash=sha256:23e6e8e8a75cff88e0a793bfd3becea03a13e2763ae90c1ff573bc19ca5b429a \ + --hash=sha256:27b8b36200a9fbee6e120246f4a8a41657549107ef19fb2c819c4b2fd524f39a \ + --hash=sha256:28d2609691da93051e998495108bbddd2a9f7a561253bae94828d81290f30c15 \ + --hash=sha256:2c024da73b296f040b8360e60bd73a659b230093684a438da0e1260f34cc724e \ + --hash=sha256:393d8a78bff6731ecc5ad2151a821f8fbc1709b137ebb9c25a4ef399fbdcc914 \ + --hash=sha256:3d6a82c4fc6c85f2fb7572c86bdb86f84c97b6580e5f6599f711800bac48a5d8 \ + --hash=sha256:3de427b05f244ba2c2a9bdc67e7a6731c8340811524ecc4435466549f8af1d17 \ + --hash=sha256:406583b4e8fb2282ebd392e12b963e601c1f82e07125a8c2cb5b144e7e024796 \ + --hash=sha256:4119efa6519871719ad81f33bc95ab87857dcb1c5801f30a6e592f2c41164169 \ + --hash=sha256:42959bd50dd660ffc3f2a9bec15a6da4f9aaa0dda555d59ff2d2e80b908456a8 \ + --hash=sha256:455ed6083353b8e938f1d58c765eab2fbb165731e5b507be30fee344915a2a11 \ + --hash=sha256:465eef3d17e59ad22a556fc0138f7c7c799df426734344daec42c797d49fda99 \ + --hash=sha256:47ecf0d9b81d981f07b61bd89eced9d2582f5eaacc3aaa36ad27f81aef70a27f \ + --hash=sha256:49717e857899f4136d7657bf5aded61ac479110a075438290923a4d86af7cd02 \ + --hash=sha256:4aaeceeb7fa7d824c322d1ec3208c8495c88478a927295553235435fc49043ad \ + --hash=sha256:57dc36a5ab0e676f5f6e171de2917fd0aef73f32a9aaf23956bfe19997a30bd1 \ + --hash=sha256:5933a052946873d01a42119a05420d669bdca436aeba2d1851988ccb12b421c0 \ + --hash=sha256:5deda5b4bf62769eb98c119cca43d40e1231e34846b19db5cdea821d446a2253 \ + --hash=sha256:61386101ecaa096b694d0dd278caf99a56aeec78440cc17e918eef0b50f2d567 \ + --hash=sha256:659728f20fc7a0933ed7b1945435e31014b97ab8a5a7edcbaa70da4794aeb191 \ + --hash=sha256:70bb4ce8be0c5606bec259cbd7152374470396413b7863a658a08c849e6b29ff \ + --hash=sha256:71fd60e6e426d293d0a2f685115ad0a0845117602cf13605a4be7524fb5f7bba \ + --hash=sha256:756ea5c2da00fa65c930284892d2a9706828704ca3ba40b4c51c4834eb39fcfd \ + --hash=sha256:800b7e00d92553313c0463c200087930aa78678ec1d528193aeb50906f55989b \ + --hash=sha256:82da34ae4f639c73ac46e521e00c0a49bf86f717b9fb1f405f133e98731e38dc \ + --hash=sha256:8e1a45d174b6b8589f51dce1cea804aa6c1f72c9c80cba91ae2caabeb6d90540 \ + --hash=sha256:8e3f508d0e9e6236ba2f08d56e33355e434e785e813149a1b8477d3edf69779d \ + --hash=sha256:986e9751d416d7a6eaa2fecdac38da63153d63a4b340ba7d624889c490451500 \ + --hash=sha256:9b73836ba0e16fcbb57c31cf6cbc2907c8d8c790b83679df454b74bd15e0be04 \ + --hash=sha256:9bab4cf571653a8afffb83ce21aa27b51dfe629b526b7b6adec35491fe1fc2ea \ + --hash=sha256:a71d24f40b0cc6798feaa978c7411dc1135b7018e9fc0442db611c139bf58344 \ + --hash=sha256:a9383401d9f116f98cacd4eba6c505a6edb80ba65badfc8e8ed8ae64983bcc44 \ + --hash=sha256:b44f0a0fc7bc6677d38cc80bca1a32814ce6c8f200fb8b3c1a61c9d77eaefbf3 \ + --hash=sha256:b5c6f20d657ae09ae4e30d9d3a21edd13f1219d58cc6f999b9d1bb63be9c1baa \ + --hash=sha256:b61692f0069b3eee2fc8a3a1b7f6c044df9e03fede6ce69b3ca832e1c39f26c5 \ + --hash=sha256:b8c62888c3e49debf37ad9773e3c02f77b0c1e811f8fb0962f2b6c3bbab5b97a \ + --hash=sha256:bd8ea8eb3817b226057cc1c0e7ec4b378dcda52043b972b6ff12b1152178967d \ + --hash=sha256:c5559b492007dc09b4de9b95dab05f0b5e53547aad230cf07e46c7dd017a3be5 \ + --hash=sha256:d0fdd25faece8a1f95e8a3a8006e29701b5cf8dadb4a8132e68f3134637004a5 \ + --hash=sha256:e094dd21f077af8194923fc263cad872eaa1802bb0156fd7e5ae18e99cd86715 \ + --hash=sha256:e41c3993eee896c617dbd8a505085d28b6e84a0445ed9a1f40f95808473cf678 \ + --hash=sha256:e88d304f094f4937bc27ec6a435e218a084168f11ec630c8d5d39b431d08d81d \ + --hash=sha256:e90e3bdf7b5eac005fef631adae9cafde16f922def207b80a7c46b253c18ad20 \ + --hash=sha256:ed2c1493c44d0932f1e55fdb5d1ead658c68288ec5d51b8c4928422d98633ef9 \ + --hash=sha256:edb6f87fc60ff438557291501b3e16c7a77c3b01a52d782cf276dccc7c5dd89c \ + --hash=sha256:efb29f8633bf6630dc89de4fe0353ac3d7e4b70ef7b6e29fb40f00e68c127fa5 \ + --hash=sha256:f6c972474ce691aca74e58d17625450cef153dc4760364cadeb167983ea6d589 \ + --hash=sha256:f6d178ba6dc8e82976c184b65fddde172d054c17237993a3e083efe4f134d55b \ + --hash=sha256:f9a456bdbed52a01c9ab8423bdebab04a5363c78676edc55ab9b58bd13bdf9e1 \ + --hash=sha256:fbdbcd06986ede3ce584083b1dc2afe6808e8943e5cf50ad11183c03aceda25a \ + --hash=sha256:fc66cb50c93554b86db0b6625ab5c6e9051dbf8847c08d93c84918e02e413fb7 \ + --hash=sha256:fff5ef3fe1bba7d6147e5f19e01e5e122ac2c076486887ddcb8d42e663400fbe + # via -r .github/requirements/ci.in h11==0.16.0 \ --hash=sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1 \ --hash=sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86 @@ -1095,6 +1158,7 @@ typing-extensions==4.16.0 \ # anyio # cryptography # exceptiongroup + # grpcio # httpx2 # mcp # mcp-types @@ -1121,6 +1185,267 @@ uvicorn==0.53.0 ; sys_platform != 'emscripten' \ --hash=sha256:a9356f0cb89b3b8621529c5d5eebd69bfe154f4c3f68b4cf2de47e45fa855c2e \ --hash=sha256:e8dca71ec86dce5f04e333f0d56cdedf942446e6643b9cea1af0d6d3a02cb03e # via mcp +websockets==16.1.1 ; python_full_version < '3.11' \ + --hash=sha256:01fbdcbac298efe19360b94bc0039c8f746f0220ba570f327577bfee81059175 \ + --hash=sha256:024193f8551a2b0eafbdd160911012c4e6c228c28430c84433253299a9e42d6a \ + --hash=sha256:04fd29a0e2fe9414a95b00e92c67ae51bf900c50c0f8a4b2dafdad621f49ea1d \ + --hash=sha256:056ae37939ed7e9974f364f5864e76e49182622d8f9751ac1903c0d09b013985 \ + --hash=sha256:0f62863e8a00a6d33c3d6566ec0b89f23787b747ffe0c3bc71ec0e76b82c94b1 \ + --hash=sha256:0ffd3031ea8bda8d61762e84220186105ba3b748b3c8da2ae4f7816fac03e573 \ + --hash=sha256:1214e673c404684b9bf7154f5cf43b45025b1a6160fac3a9e438e9c1a97e22cb \ + --hash=sha256:125f22dbefaf1554fea66fc83851490edb284ce4f501d37ffed2752f418332d9 \ + --hash=sha256:130937b167a52af203c8d58e78d67705874e82759862e3b9671a452fec4abc87 \ + --hash=sha256:1427fb4cf0d72f66333e2cacc3ff5f575bf2d7008166ce991a4a470b21d51a22 \ + --hash=sha256:195c978b065fa40910582464f99d6b15c8b314c68e0546549a55ed83f4735328 \ + --hash=sha256:1d27fa8462ad6a1cb36206a3d0640b2333340def181fae11ed7f9adeaa5c0747 \ + --hash=sha256:1db4de4a0e95673f7545d393c49eeb0c2f18ac1ef93073218c79d5cdb2ee75ab \ + --hash=sha256:1f79c89b5eb034d1722938a891916582f8f7f503f58ca22518a63c3f2cd18499 \ + --hash=sha256:23253dd5bcae3f9aaee0a1d30967a8dbd52e5d3cff93a2e5b84df57b77d4750d \ + --hash=sha256:249116b4a76063d930a46391ad56e135c286e4562a18309029fc2c73f4ed4c62 \ + --hash=sha256:29dfa8114c4a620c69591c5973860f768eac29d3fd6904f37f34266cb219c512 \ + --hash=sha256:2a606d9c24035242a3e256e9d5b77ed9cd6bccfcb7cf993e5ca3c0f6f68fb6a7 \ + --hash=sha256:2a636ff1e7a5c4edf71ef0e79adae7f25dba93b4fcbe3dc958733477ffeb0eaf \ + --hash=sha256:2bb5d041a8307d2e18782e7ce777f6fdb1e8c2f5d09291484b18c294b789d9aa \ + --hash=sha256:2e28e602bb13da44fbe518c1781a88e3b9d4c3d48d02c9bad83e546164336f57 \ + --hash=sha256:30bbe120437b5648a77d3519b7024ea09530e0b5b18d3698c5a0ae536fe0cc2e \ + --hash=sha256:34420aaa64440ebd51ac72ca8a45ef4626429438c9b02e633ae412ed43f925d3 \ + --hash=sha256:38565aca3e01ea8734e578fb2118dade0ecb0250533f29e22b8d1a7a196cf4d0 \ + --hash=sha256:387e8e4aa5df2f90b198fa3cad3478822a89cf905b6a6d6c97dc3664689640cc \ + --hash=sha256:39f2a024af5c345ffe8fcf1ee18c049c024c94df393bb09b044a6917c77bde43 \ + --hash=sha256:3df13f73af9b3b38ab1195eb299ecb67a4330c911c97ae04043ff74085728abe \ + --hash=sha256:414e596c75f74e0994084694189d7dc9229fb278e33064d6784b73ffbba3ca31 \ + --hash=sha256:41c8e77f17294c0ac18008a7309b99b34ee72247ef10b6dff4c3f8b5ac29896b \ + --hash=sha256:42290eb6db4ccaca7012656738214f8514082fb6fa40cdeb61bb9a471b52e383 \ + --hash=sha256:42f599f4d48c7e1a3338fdaac3acd075be3b3cf02d4b274f3bf2767aedd3d217 \ + --hash=sha256:43e3a9fdd7cbf7ba6040c31fae0faf84ca1474fef777c4e37912f1540f854499 \ + --hash=sha256:443aefe96b7fdb132e2a70806cca1f2af49bb3f28e47abcd7c2e9dcf4d8fa1b8 \ + --hash=sha256:46dcaa042cd1de6c59e7d9269fa63ff7572b6df40510600b678f0826b3c7af51 \ + --hash=sha256:496af849a472b531f758dbd4d61338f5000538cb1a7b3d20d9d32a264517f509 \ + --hash=sha256:49ae99bdfcae803a885c926bf14f886196e84925395bb3f568fef5c0f0979d7d \ + --hash=sha256:4b57693728576d84ede0a77987ab16881b783d2cd9f1dc180a8fbbc3f79c4428 \ + --hash=sha256:4e3b680b1e0a27457e727a0d572fd81dffa87b6dbf8b228ab57da64f7d85aead \ + --hash=sha256:4e8d01cc3bcae7bbf8167f944aeafefed590fae5693552bba9794a9df68371cc \ + --hash=sha256:5283810d2646741a0d8da2aa733d6aefa0545809afccb2a5d105a26bc45125f1 \ + --hash=sha256:53260c8930da5771cec89439bff99c20c8cb03ddb9588b980697355a83cd4bd3 \ + --hash=sha256:536676848fc5961aca9d20389951f59169508f765637a172403dc5434d722fa0 \ + --hash=sha256:54509b8e92fee4453e152b7558ddef37ce9705a044922f2095a6105e3f80c96f \ + --hash=sha256:56cd5fc4f10a9ea8aa0804bddb7b42506cf9e136046f3b4c27de8fec9e2ecba5 \ + --hash=sha256:5bfd1ac19b1b9986a9c95a82d5e23a391ebb09e12c34d7be6094b86efcc35731 \ + --hash=sha256:5c31aa7e39ee3e8a358573257f1c0bb5c52430d1b637030dd9c8cc2c282926be \ + --hash=sha256:5e3b7d601f6f84156b08cc4a5e541c2b50ad7b36cfc302b657a12477c904a5df \ + --hash=sha256:61922544a0587a13fd3f53e4c0e5e606510c7b0d9d22c8444e5fae22a06b38cb \ + --hash=sha256:6456ff333092d509127d75a638cb411afae8ff17f092635015d1902efec8a293 \ + --hash=sha256:69159730a823dde3ea8d08783e8d47ef135a6d7e8d44eb127e32b321c9db8e3e \ + --hash=sha256:69e52d175a0a7d1e13b4b67ad41c560b7d98e8c6f6126eb0bda496c784faf8c7 \ + --hash=sha256:6aaface73b9c71974c6497366d8b9628357f6c9749e09c4ea3610176c63f2ae3 \ + --hash=sha256:6abbd3e82c731c8e531714466acd5d87b5e88ac3243465337ba71d68e23ae7e3 \ + --hash=sha256:6ff9417c0ada4d0f7d212f928303e5579bdf3ace4c802fa4afabb30995da58c3 \ + --hash=sha256:7421fad442de870a8cbf2287d1cad7e706ece0dbfeba5e911df132cbdc1cb56a \ + --hash=sha256:7883388947767080f094950b342b30d35a2a06b849cd967c422fa0db72b40ea9 \ + --hash=sha256:79eace538c6a97e96d0d03d4f9d314f9677f5ed85a8a984992ffd90b13cb8a56 \ + --hash=sha256:7b1b19636af86a3c7995d4d028dbe376f39b4bf31541146f9c123582a6c94562 \ + --hash=sha256:7dfcad78ea1492ee3a9ec765cb7f51bbc17d477107aaf6b22abf7b2558d1c5a0 \ + --hash=sha256:8087e82f842609734c9b5a1330464f8e94e346ba0e18c832c08bafa4b0d63c15 \ + --hash=sha256:820fb8450edddae3812fd58cbc08e2bf22812cb248ecb5f06dbb82119a56e869 \ + --hash=sha256:8483c2096363120eea8b07c06ae7304d520f686665fffd4811fad423930a65d7 \ + --hash=sha256:84a2cef8deffbd9ab8ee0ea546a2a6a7030c28f44e6cdd4547dbfeb489eb8999 \ + --hash=sha256:86d7f0f8bdb25d2c632b72527325e4776430fd5bc61b9118de4e2b8ddb5f5b01 \ + --hash=sha256:8fe0b50da2d84535fb4f7b4bfa951280f97ce3d558a0443b541166d609e67b57 \ + --hash=sha256:90001d893bc368e302ef168d82130b4e4fdd27b85fa094682df9b667c2d48838 \ + --hash=sha256:9246a0d063cfcbcc85f2359dd6876d681213f4790832272aa16641b4ed5d64d4 \ + --hash=sha256:92b820d345f7a3fc7b8163949ee92df910f290c3fc517b3d5301c78065adafe1 \ + --hash=sha256:952303a7318d4cbe1011400839bb2051c9f84fa0a35923267f5daba34b15d458 \ + --hash=sha256:97fd3a0e8b53efa41970ac1dff3d8cf0d2884cadeb4caaf95db7ad1526926ee3 \ + --hash=sha256:9c1c5705e314449e3308872fe084b8571ce078ee4fc55a98a769bdefe5917392 \ + --hash=sha256:9c9f23004a3d40e89c01a7955d186a6cc83418d93b749701944ce2de3e95a1f3 \ + --hash=sha256:9f63bcef7f4b02b06b35fc01c93b96c43b5e88e1e8868676caacf493d5a31f3a \ + --hash=sha256:a0eadbbf2c30f01efa58e1f110eb6fa293261f6b0b1aa38f7f48707107690af9 \ + --hash=sha256:a28fcbc9b6baf54a2e23f8655f308e4ccc6afdd7266f8fe7954f320dcda0f785 \ + --hash=sha256:a6a61aff018180c9c50b7b0da33bfd29d378af3497429c95006c589a23a11648 \ + --hash=sha256:aabe464bfd13bd25f4821faf111da6fefdc389f870265a53105580e45b0a2e49 \ + --hash=sha256:ab59169ace05dcb49a1d4118f0bde139557adf45091bd85747e36bf5de984dd1 \ + --hash=sha256:b436f6ec4fc3a6b4237c84d3f83170ed2b40bb584222f0ac47a0c8a5921980c7 \ + --hash=sha256:b6b9dadbef0cccd9f4c4ee96b08898afa73e26803bbe0f6aeb5bb12b0074206d \ + --hash=sha256:b852788aa51764e2d8e4cf5493d559326bcae5e38d16ba25ffa322b034df272a \ + --hash=sha256:bae954c382e013d5ea5b190d2830526bfa45ad121c326da0049b8c769f185db6 \ + --hash=sha256:bcce07e23e5769375158f5efdcdafa8d5cd014b93c6683865b840ed65b96f231 \ + --hash=sha256:cc97814dfb786a83b6e2dc2e79351e1b83e6d715647d6887fcabd83026417a00 \ + --hash=sha256:cd2ca96a082a36964aca83e992f72abeb61b7306c1a6cba4c7d06a7b93750cac \ + --hash=sha256:cfb70b4eb56cac4da0a83588f3ad50d46beb0690391082f3d4e2d488c70b68ea \ + --hash=sha256:d0fcf657e9f13ff4b177960ab2200237b12994232dfb6df16f1cfe1d4339f93c \ + --hash=sha256:d14bfb217eb4701e850f1525c9d29d79c44794cdf1c299ead25f39f8c78dea81 \ + --hash=sha256:d57685547e0060cc6fd90ee6a28405d6bd395e525545f13c8d7cd99c78afd79f \ + --hash=sha256:d6bec75c290fe484a8ba4cacdf838501e17c06ecfbbf31eede81a9e431bd7751 \ + --hash=sha256:d9531d9cbeac99af6f038fb1bc351403531f7d634a2c2e10e2f7c854c6ed5b68 \ + --hash=sha256:da4ca1a9d72f9030b3146b8d7022719a9f3d478f61efe6f7dd51d243f61c51b2 \ + --hash=sha256:dab9eb87869da2d6ed3af3f3adf28414baae6ec9d4df355ffc18889132f3436c \ + --hash=sha256:db234eda965dcce15df96bb9709f587cd87d4d52aaf0e80e2f34ec04c7670c57 \ + --hash=sha256:dc0fad4933f427acd5b1cec210f3ea6dce7089e1724e4b9ec6ef47c6c04d1b3b \ + --hash=sha256:dc385593a42e31cd6fb60c19f0ecb015b386603818fc2c6c274fb42bd2bb4165 \ + --hash=sha256:dcc04fedf83effaeb9cce98abc9469bb1b42ef85f03e01c8c1f4438ef7555737 \ + --hash=sha256:e047dc87ef7ca50f4d309bf775ad4a71711c58556d75d7bd0604b2317f43e94b \ + --hash=sha256:e09f753a169951eb4f28c2c774f71069304f66e7277e0f5a2892423599cfa854 \ + --hash=sha256:ed5bb271084b46530ee2ddc0410537a9961152c5ccba2fc98c5276d992ccba87 \ + --hash=sha256:f0aa4aad3b1b69ad3fd85a0fd0952ec64331c762bd77ec51cc814170873890b2 \ + --hash=sha256:f17dbe07eb3ea7f99e4df9b7e0efefe80fbf30d37a8cc4d561a0aed310bc8847 \ + --hash=sha256:f2769a0344a09e9ccf5b3cce538bc75a51b53eff3275d3896310c8552049195d \ + --hash=sha256:f55f0b01956a094c8587146d9558c91937e78789c333860ffaf35931a6e5dbc4 \ + --hash=sha256:f5d497865f05bb222cab7016c6034542e84e5f29f49c6fd3f4939cda7197b5b8 \ + --hash=sha256:f70541f3104339f59f830522d94ebadb1bf47426287381623443d8bb1cdbf33d \ + --hash=sha256:fb9a0a6dc3d1b3986cb88091b6899f0396651e0f74e2c9766ab8d6ffc3842e29 \ + --hash=sha256:fce6c48559c86d1ac3632ccb1bebc7d5442fbe79bd9bb0e40379ee54be2a4051 \ + --hash=sha256:fd46fff7eb62c24804d234f0051c7a8ea81285ad63e0337d3dcf33ca82aee58a + # via -r .github/requirements/ci.in +websockets==17.1 ; python_full_version >= '3.11' \ + --hash=sha256:0014eaff8ad5b3b43feda2279f9d34bf2eaae040720b9fbbb55944b10f40b14d \ + --hash=sha256:00679b7468b4c2b12b0757118174e8eabac56bb2f579a928a104d9554a56e098 \ + --hash=sha256:00bf34b64501e3477e81fc281532ff3cbf4da26633c10b63979d5085d46602d3 \ + --hash=sha256:01dcb47deebc40b38fd4a493b9b9f4d0a704b7bec6f35e4d34085b329abce71a \ + --hash=sha256:020e271205f8ab3406d7a59cd00de6dec722315924411c421bd00642f18bad86 \ + --hash=sha256:0340bbef6bfbe16da888b3983d666a4db4954ac3253c38f13bc7aba0c7db5a2f \ + --hash=sha256:054c28db2dcec0e857e3b705d8c28012613e555b38c765d6a4f75340a4fc06a0 \ + --hash=sha256:073c5c3f7e127041fa9d34a9e29ceefee8c3cafbd267ed2927318f425144380d \ + --hash=sha256:07fa3e7c30e2c577928d359b56bf872a3e0cbcc15553eaa0907c1ee86344b56f \ + --hash=sha256:0c84bdef916556cbe1d5a43b423398be4dd3cba6522b463e53d848578b920695 \ + --hash=sha256:0c863507ada5805517ca6dff1c524dcd42942efe6304dacf06700878398d21a6 \ + --hash=sha256:0c9982938980e086da59f70d05f9418cd143401a601a0faac10fa48f7bb1cd3e \ + --hash=sha256:0de501b7f2db11e83739ac20e2d33d46da4604b829f506c24be80e7def069391 \ + --hash=sha256:0f4f50dfe2cc810fc4e2de979b35e83bf8bb4bccdc6fe472d93762ea7b1d5927 \ + --hash=sha256:1036189bd34b0bc1b10a4679321e2c7968af317efe6e8e4c1c5141c4254fb5bb \ + --hash=sha256:10ecb38ffc05e1841b619d99c725307a223ef9ad58e7b1ed33311d472dc43918 \ + --hash=sha256:12c8e2b25df59755954a04dfa09c990b96691025aaf7eafd19ed6da24b09c18d \ + --hash=sha256:12fe8984a32dbfd084e0603f1a8d740c0180cb85b3174585c54a80d2455a8394 \ + --hash=sha256:17aa424ab61620aad21b36b2240efc87b500cc496e7d0e999a5c2ae99395e886 \ + --hash=sha256:1a20136d61f9ca3a31493732762661fafc2c20e8861930214e21afc6a8a692a2 \ + --hash=sha256:1a5cf4e7bbe3ca499e6a289206cb4fcb7444b09919e129bd517f57d5fa192c13 \ + --hash=sha256:1c7ac77401227212dc6e849182feee50d57cf456ec6329ffd6979c94bb136c5c \ + --hash=sha256:1fce0f43e0d41422e0b2cad6561e1970df22f212f4c7e884967df7cf591b031c \ + --hash=sha256:259f45358c76d3b18489e3e80636cdbe807e05ecf1b10fdf1a779106d23d0c8e \ + --hash=sha256:2786cbd273ab69c22612db8a41229ddf2c158060b17b5928884bf388d07887f3 \ + --hash=sha256:29176d8b429cfa0fa443c473878d37a5c06cfd0cb36b71ba4314accc71e05906 \ + --hash=sha256:2a0162a6372110a5601cb5c9fd826635cedf69f3e110c545dd19774e040b970e \ + --hash=sha256:2a4dc6ef83f4559e0d05f313a375cb38f63c986096a9da99fe94fdd779d313e5 \ + --hash=sha256:2afb58c7ba48b329d56769f8dfd89f394efe587b65ef806bae810a484d6d3608 \ + --hash=sha256:2cce251f3e2469b99b6802b55435bcdd07123b41870f54c87b336183af9d7e68 \ + --hash=sha256:2d72879819f5145a342d0030c418702496c65a4b913ef81f5ae944dd91dd50f6 \ + --hash=sha256:32a2a68d989d6e5b74a9d5095415c51189ebae29fceb7cf2b64a1c0318a81256 \ + --hash=sha256:33e45c7ea38428e740a7f233555d71df0b875cef7fc080acebc9654475e35335 \ + --hash=sha256:34879e19bb0a3c44f9317679435aea5327fac993933a704cbf353bf1234b10c7 \ + --hash=sha256:3546ef55b3a074494106508bc6505c73825970d2d9505f7bf53882b3e88b0d1e \ + --hash=sha256:3709a1ab30b4b922027d22f68d2b61a0656a91680ac894a537624e6be7dd7f7c \ + --hash=sha256:4031152769179ab8dcdeafc7b0e58052a49117560a28671700b47b2c7b717aad \ + --hash=sha256:406b8107943a43ef4649b1e0cb0cdc052bbf08fe1c8905a623c4af9586e5cebb \ + --hash=sha256:416b4bc8789a1865a3ff643ec4ee073a5f52402d0dbeafd27b1798d5dd6b6a51 \ + --hash=sha256:43bd0c1ceb924d67f5c1a5254d8361dd9d94246e6331a726064dfa2917880780 \ + --hash=sha256:455ffeea0879d313205df1e745e5883e1feb7f31ecd26be882f5f0babd3db04f \ + --hash=sha256:469355ab1af100b9380f1afb09985019f4a4b94fa1dd0e9396db4361626d7ab8 \ + --hash=sha256:46c0331c9eaaf73a559f3a9e388466be0df96eb83d40f06f1ca6ab6613b35c82 \ + --hash=sha256:46e7a10bf04318c7b0c0273791925ae5e1cbe4a11e34aa934d2ef27862058a80 \ + --hash=sha256:47a62d6045c6eaa0d8f97bc2fb68b8cf90077a0cbfd4e83d6f2d2145611ee134 \ + --hash=sha256:4af784f3e436f65b355c117c6497320f2b5cf6a559295cb1c4c7338e335d45cc \ + --hash=sha256:4e8ffcb486c8490a34a4cef5e4409d8da5a1cb1681e5bf7d786ce5e84aa8540d \ + --hash=sha256:50903d335bfda026c2fa11dd9aed09d8cbee0c451e3a85122a9acb041b7dc69b \ + --hash=sha256:57b39dc8541cf7ed3f639da82bf7451060483967f9e733da1f8173e4095f0642 \ + --hash=sha256:581fa678ef46f4277cc8491312468e582f8ad609dbab907ba6096a08c6a0ff98 \ + --hash=sha256:5a74531ce81af587f906ab42f194032388fcff8fc7938402e5917c9147a39441 \ + --hash=sha256:5aefe78e6a3077fe22b5e64b04666a85a3eb8b934d40e8595a693adcbceb6f11 \ + --hash=sha256:5c27336b1a0ac56569493e858497870347854372395f50483725f8cdacc5a45c \ + --hash=sha256:5d4724255fb8398acd9e583b97eb2279cec20e0bd0f9a94bf75f6056ef9f13da \ + --hash=sha256:5f051f8030a51815dc00e24bd2e5f1435af095c1cc111d747ac6e2a3620d7641 \ + --hash=sha256:615746b12b26a3fd4077bc6fbeb277a1c192a45dd57b531d07ad9ed5c52a9a7a \ + --hash=sha256:617243e19a0992095956f406ee9cd3bc4ba92862d83cb1d83bb59ce574412bec \ + --hash=sha256:655a8e28010f09fd6fa317e857afab3af7647f33e41dee88fa421e92086d1090 \ + --hash=sha256:65be6bda2b537fefa4b3a5ccd6ab386533ce39dd8fe62433ec90901fdc81752d \ + --hash=sha256:67258b00302a5aaf0b267771c7014b13429abd7ea17eebc4c55bd935ff101555 \ + --hash=sha256:677014a073bcb1fbaa7e21144786864f16c08f856d66834f611eceb9006cbab8 \ + --hash=sha256:6a8e768a048c2220697477ce2e67e4345dc9f693d0ee6af53945b5e30227c6a7 \ + --hash=sha256:6da17fc94bd270f5987b10bee113461ac36a36a98b0481ddcc98056e5a90001a \ + --hash=sha256:6de9acef07e3a78e9567fcd26c29011a4da8f050b13004bbf880a0fd82a6eea5 \ + --hash=sha256:6e63c01803be425ff062b7f7fc201a74def1d49fc94a2410dd17375df75936e9 \ + --hash=sha256:6f42912fa9eb4cb7c7ec9fde9b3332ba339eb8a8811981043d4029599f3d950b \ + --hash=sha256:722ec21717eec6477bce582147a28acdfe034e604239466a6a95daedb863e774 \ + --hash=sha256:764cf7bfa149365f32b7a0fd9fed32debdac29dd06295d5635cde1745b446cd8 \ + --hash=sha256:76dd004f59115087c7b700474cb18f01325e37250032e19396c08ae41448e4b3 \ + --hash=sha256:77b37cceca17291897c3c73bd30a7c7c7909593554b5da574ec852af83c1742a \ + --hash=sha256:7a72efa3bf4fa3a6669a54420a472ad056da3973d827f10e3a536da463f926c2 \ + --hash=sha256:7e724f843fa6a0614aece65a7c73e51d0f4412ca41dccac13c3caf98e69536bb \ + --hash=sha256:8196d217eeca52b9235ee1f8a684a09885a5f953d5a31e80ef915bf2c5c94f9d \ + --hash=sha256:829dba1bc049779de9b332088c1a6a9858e96bd67e50b6b644a95e02b67836bc \ + --hash=sha256:84f25e099fdfe3b09f953d84698f729a1f7d1e99101b2787d7a28ed77b323750 \ + --hash=sha256:87f0d5e77548b0c40c8464cdb6108792e7e53f487c6400028a4ec28a8afbe5ab \ + --hash=sha256:880069d21cc33a558dcf180924a546d1ecf8ada5be3e4e70acee87019d706a24 \ + --hash=sha256:882af300d2c6a092b93767d5de03c7bb56dfb06314140c8e872d3f48e09f7b74 \ + --hash=sha256:88381602e379165b66244b2ebc29f9b23ea0851fbe63ae157f91ca324f072d6f \ + --hash=sha256:88b882764ef65147a7a5ae13168dedbe225a04e2ff4858fe543f2c402f093e9c \ + --hash=sha256:88bc5138e53903a85c354e59df7ba73ce306f7b09724cef74dba121e60a88ce2 \ + --hash=sha256:8d1b108bd8f5f6a8b90801f6db3b3858d5deca889acfdb8ac497bbb24e4b0edf \ + --hash=sha256:8f6c38cdcaf98a911d7acc25577f2f9e710f3a2fc2bde1563556784320196b51 \ + --hash=sha256:8fbf28e639544503b7d1c96452a5e5e043e4108d89b1f3fa02910603622d19db \ + --hash=sha256:96abdecbaae746851b87c3a36cb4a661df93ca3d92f114270f79228bf1d00de6 \ + --hash=sha256:98a5b2589a56a4b4f098b0a958099a4356ab904a7844f1da3841efca469af7e9 \ + --hash=sha256:9ae55d24241fc055f22aea3ac924559069848bd0ad4ea065fdd72d2194685fe8 \ + --hash=sha256:9f4a08ff7cb68c27b18e09223cc6304e01d0f82d5a240d251266dfd2e6e44729 \ + --hash=sha256:9f4c0377a83e163a303514fdfab501dbe379bdc13e5b9312a91d112658b29dce \ + --hash=sha256:9fa1ffa08c81a4f809cdab6129f8e55bee4650b9d6d3461019dda73aacd146b6 \ + --hash=sha256:a06f3b5085176763182449559e20391d7ce616a8972a9f7a33deda87ea6d4f3c \ + --hash=sha256:a62d8c424383c9dc769ff3672018df822603117e32686e567d452ed035b6fb2e \ + --hash=sha256:a7b35181a14cbfcae163b4de545d22abfd07d06c2c41ca69cfcd99251d6888ab \ + --hash=sha256:a9fe648abd1d9b89aebfa30407bfdd08a0271ec5dc7d44a4c6ccd1ce22cf562a \ + --hash=sha256:acfea4c20bf54384883ea33b1240fc1db4f52e190823a4e2b334bc3e8bfca96a \ + --hash=sha256:ad68c28a27246fed109a4409393d677b7e1388345cbbd2f5aee5c182d8506110 \ + --hash=sha256:aec00f018d34c67500ff0438dc314b40277be4a1b983cbacbf53ccf7db63e257 \ + --hash=sha256:b1bc819c6db90e8f91a38250a1ab4c058261871aa52d2fe36382eddedf146dee \ + --hash=sha256:b1c323fc3be1dc3f87f6c59458cb7d9e13dcbbf971d6c3f3e2bbaf58d3bfcdfe \ + --hash=sha256:b3b451fd2723ad3191a209afe6f3f4bc86c83e9a85bdc255353b91803ee6aa66 \ + --hash=sha256:b65d5fe48219dc2d5e158de9e6514e75600f379cc7e37108d35f31764c155566 \ + --hash=sha256:bb0efe019480a1c93e168ce96479273aaebd672fc8c350d5eed1e507ababb1b8 \ + --hash=sha256:be3f0129c5654517b2abf07dcb75bb1d9479759a4ccfb569e8293579e9fc029a \ + --hash=sha256:c3241d684a76eaaef8b2dc789afde4343cd3aad55ea81e4e8ab3605b529bae51 \ + --hash=sha256:c65da239a5ad553619804c1f9d65c1a0b3005381c6158ee14da2c7444cbd0c78 \ + --hash=sha256:c89406fa3dcd4aa8662c6406cc5c0de1790e9614d2c3aaf03ca53a8a8ccf3405 \ + --hash=sha256:ccbf3f4a9890d50b3a08ee04029fde30a03bfdeffaa19977628bf17251764e60 \ + --hash=sha256:ce0305b702b20d1e1d60a9aaace6bc89970e1753565543f310d549eab22c2435 \ + --hash=sha256:ce13c7d233239e739600a57d4a73c1192ad8259e655a4d55aa1a454242bc809d \ + --hash=sha256:cec1bb8f22abccc8d20f8ca63df9be41600c26c190f4b97ee86c675fd4a863a6 \ + --hash=sha256:d1e2f5fa2b6d01f0d85b4f223fea7ed1d504be282a02a81bd2be4817ef7a2f03 \ + --hash=sha256:d334d11398086bb5559606cb42d51c013ea7c061c7db701521392373d3c087f5 \ + --hash=sha256:d411ea5ca18ac1b12c0c94be88b60c18ca641ac43bcdfdf1c9f79d46cdbe1603 \ + --hash=sha256:d41ef69d5416fbc1d98cf96c37be6192d10fd101c3e0f8b3ddc36e09432b3c08 \ + --hash=sha256:d58159af7835fde09c462394293c0d7aaf8fb4557d8f8e5699f5e722ccae013d \ + --hash=sha256:d7b349265fad6244013eecd99df8d83c12bf3013943e431f4fadd5bffc37db42 \ + --hash=sha256:d8e83333385cac6030a5167fd18bf96cc6c58b914c308e683f05b0cf94bc8dd0 \ + --hash=sha256:d9d01e8ede41fea4f5a847dad9d628355f74905f437a5b6856d67aa66d193800 \ + --hash=sha256:d9fc873e239c5abeb150bc24dbd1a7af23a9254526383ce0a077f5e20adbeb19 \ + --hash=sha256:db9d7ee47f3ba531e278be539af39e2c7c7d28fb94897b6cd1120d63b0ef5922 \ + --hash=sha256:dc053f9e95a76213c5eb7ed95779f7daf0d2bf0e4e03073629ebfa43a033f151 \ + --hash=sha256:dc2b79afc074d2f3e64b26539350f697fe1b85ea1c49ea24eb588f247b053ce1 \ + --hash=sha256:dc5789e5ea182b77a38881383ada5347202a6c66f4857d054e075290e80b604b \ + --hash=sha256:dcb8d5f7edef7a399d322cf28d4c4e6f98dab64d301c8f50581a1080e5198142 \ + --hash=sha256:e4bd7eacb87d8cf3ed70d6392c770a0d92441f05d7d2a3efafb5bc171d5e3067 \ + --hash=sha256:e552e0037230ac16e5f568de7012041344d1b18c9feed30ec2891b8eba55af81 \ + --hash=sha256:e5f5c7a893507d0e83a80b88aefd6522f7e882cd53f9722c6f23f5a020c9557c \ + --hash=sha256:e74e41f0ad12ff1e8983e349daef79d37cc8280c743ce9d134d6c74c18dab5d6 \ + --hash=sha256:e78fd4b7b2c5086a38671c9c882c1e643385eccea360b5b1fda4a105e590087e \ + --hash=sha256:e8dc3fa6d6b7ead3f9de57895f41b116a28787548e066365d9d90f7356bcaad2 \ + --hash=sha256:ea0ed9373b880115911d9d39634bccc95b8ce590c9c42e8589f5cacc3ef3cee2 \ + --hash=sha256:edadce7a22052056fd4384543019856b34850363c9d387929f677ae01d79709c \ + --hash=sha256:eec113a5b41d124ef42ff56b0d74a6da3fd986400038eab9e58ee42a4024e837 \ + --hash=sha256:f221081107b8c48184d99f7019604486376e7ef826037e70aad6b02540732c23 \ + --hash=sha256:f3a1d577e081667dda7f8e5b4796e6e32f9713c93e2a3d930669519840a3c623 \ + --hash=sha256:f47aafd92aa28b941180e6da8a42b0f711851b14b81a5b6bb28dbbb1fa35152c \ + --hash=sha256:f4c45ee2512d3757b5e6c67c5a34e435143f2ecb7df3324f9fd888688c45c0f4 \ + --hash=sha256:f58f58b4b29bbea2a3635e2c56eff4a3adab011fe383802a9e542e31b97085fc \ + --hash=sha256:f612dc57f00c07cf4aa2673f7cbceabd654ad2457b7e639f061b794d6e11f9fd \ + --hash=sha256:f62114a54117e4948a1e414e89521f7fe1e3c2f83f2a571a06a4fc6718b0900a \ + --hash=sha256:f64e001bb7fa89b9f32cfa600bf8e9ac8ca26759d9b92ae01453ee303d9cd7b4 \ + --hash=sha256:f7233eaf441a345a5943a929fd4b5ea3278f11aed35a9ed0f3106b8cb3ca846a \ + --hash=sha256:f78a3ffb1994304db2c0c4588e4d1a518079b557054fa3bb985a6f5e50ff49a3 \ + --hash=sha256:f8e822efd54137d8cc8310eb64635ab827a4a6c72ff08691f38aa624776d8ecb \ + --hash=sha256:f98bf378d7a5be047a044a1a27c987a8f355e10e3b5754617dbe756248cbc5ce \ + --hash=sha256:fb88076df585b69c5761c387c0081aa87d7b9eb1b205a6535ca4777e25650d81 \ + --hash=sha256:fd8f47dbf2e8adb15c847215f83436de3fdb120b51fdae0fbbdf69fd97a3ad80 \ + --hash=sha256:ff3e2ba7a9f0a110b0555452e9b5a03a34e11662544e01beea15f144b48ba7b7 + # via -r .github/requirements/ci.in werkzeug==3.1.8 \ --hash=sha256:63a77fb8892bf28ebc3178683445222aa500e48ebad5ec77b0ad80f8726b1f50 \ --hash=sha256:9bad61a4268dac112f1c5cd4630a56ede601b6ed420300677a869083d70a4c44 diff --git a/README.md b/README.md index cf9fb8e..4001f64 100644 --- a/README.md +++ b/README.md @@ -234,8 +234,49 @@ The bundled `FlaskMockServer` now supports several layered features: ```bash apitestka mock --host 0.0.0.0 --port 9000 +apitestka mock --config mock.json # HTTP routes plus WebSocket and gRPC endpoints ``` +**WebSocket and gRPC endpoints.** `WebSocketMockServer` (`websocket` extra) serves scripted +routes: a greeting on connect, a reply map for known messages, and a fallback for the rest +(`{{message}}` echoes, `null` stays silent). `GrpcStubServer` (`grpc` extra) serves methods by +full path without compiled stubs: fixed unary responses, server streams, or a status-code +error. Payloads are raw bytes, so serialized protobuf works with generated clients; `str` is +sent as UTF-8 and other JSON values as compact JSON. Both keep what they received for +assertions, and an unknown WebSocket path is refused with HTTP 404. + +```python +from je_api_testka.utils.mock_server.websocket_mock import WebSocketMockServer, WebSocketRoute +from je_api_testka.utils.mock_server.grpc_stub import GrpcStubServer + +with WebSocketMockServer(port=0) as ws: # port 0 picks a free port + ws.add_route("/chat", WebSocketRoute(replies={"ping": "pong"})) + ... # connect to f"{ws.url}/chat" + ws.received("/chat") # frames the route got + +with GrpcStubServer(port=0) as grpc_mock: # needs the grpc extra + grpc_mock.add_unary_response("/shop.Catalog/Get", {"id": 1}) + ... # grpc.insecure_channel(grpc_mock.address) +``` + +A `--config` file describes all three kinds; every section is optional: + +```json +{ + "http": {"routes": [{"rule": "/health", "body": {"ok": true}}], "openapi": "spec.json"}, + "websocket": {"port": 8765, "routes": {"/chat": {"greeting": "hi", "replies": {"ping": "pong"}}}}, + "grpc": {"port": 50051, "methods": { + "/shop.Catalog/Get": {"response": {"id": 1}}, + "/shop.Catalog/List": {"stream": [{"id": 1}, {"id": 2}]}, + "/shop.Catalog/Delete": {"error": {"code": "PERMISSION_DENIED", "details": "read only"}} + }} +} +``` + +JSON actions reach the same mocks with `AT_mock_start_websocket_server` / `AT_mock_start_grpc_server` +(`routes` / `methods` in the config form above), `AT_mock_websocket_received` / +`AT_mock_grpc_received`, and `AT_mock_stop_websocket_server` / `AT_mock_stop_grpc_server`. + ### Runner ```python @@ -452,6 +493,7 @@ je_api_testka/ ```bash pip install 'je_api_testka[websocket]' +pip install 'je_api_testka[grpc]' pip install 'je_api_testka[schema]' pip install 'je_api_testka[security]' pip install 'je_api_testka[otel]' diff --git a/README/README_zh-CN.md b/README/README_zh-CN.md index 5f15c25..576761a 100644 --- a/README/README_zh-CN.md +++ b/README/README_zh-CN.md @@ -231,8 +231,48 @@ cassette = Cassette("tape.json") # 离线 replay-or-record ```bash apitestka mock --host 0.0.0.0 --port 9000 +apitestka mock --config mock.json # HTTP routes 再加上 WebSocket 与 gRPC 端点 ``` +**WebSocket 与 gRPC 端点。** `WebSocketMockServer`(`websocket` extra)提供按脚本响应的路由: +连接时发送问候、已知消息查回复表、其他消息走 fallback(`{{message}}` 会回声,`null` 不响应)。 +`GrpcStubServer`(`grpc` extra)不需要编译 stub,直接用完整路径提供方法:固定的 unary 响应、 +server stream,或返回某个状态码的错误。payload 是原始 bytes,所以序列化好的 protobuf 可以给 +生成出来的 client 使用;`str` 以 UTF-8 发送,其他 JSON 值以紧凑 JSON 发送。两者都会保留收到的内容 +供断言使用,未知的 WebSocket 路径会以 HTTP 404 拒绝。 + +```python +from je_api_testka.utils.mock_server.websocket_mock import WebSocketMockServer, WebSocketRoute +from je_api_testka.utils.mock_server.grpc_stub import GrpcStubServer + +with WebSocketMockServer(port=0) as ws: # port 0 picks a free port + ws.add_route("/chat", WebSocketRoute(replies={"ping": "pong"})) + ... # connect to f"{ws.url}/chat" + ws.received("/chat") # frames the route got + +with GrpcStubServer(port=0) as grpc_mock: # needs the grpc extra + grpc_mock.add_unary_response("/shop.Catalog/Get", {"id": 1}) + ... # grpc.insecure_channel(grpc_mock.address) +``` + +`--config` 文件可以描述三种端点,每一段都可以省略: + +```json +{ + "http": {"routes": [{"rule": "/health", "body": {"ok": true}}], "openapi": "spec.json"}, + "websocket": {"port": 8765, "routes": {"/chat": {"greeting": "hi", "replies": {"ping": "pong"}}}}, + "grpc": {"port": 50051, "methods": { + "/shop.Catalog/Get": {"response": {"id": 1}}, + "/shop.Catalog/List": {"stream": [{"id": 1}, {"id": 2}]}, + "/shop.Catalog/Delete": {"error": {"code": "PERMISSION_DENIED", "details": "read only"}} + }} +} +``` + +JSON action 也能操作同样的 mock:`AT_mock_start_websocket_server` / `AT_mock_start_grpc_server` +(`routes` / `methods` 使用上面的配置格式)、`AT_mock_websocket_received` / `AT_mock_grpc_received`, +以及 `AT_mock_stop_websocket_server` / `AT_mock_stop_grpc_server`。 + ### Runner ```python @@ -446,6 +486,7 @@ je_api_testka/ ```bash pip install 'je_api_testka[websocket]' +pip install 'je_api_testka[grpc]' pip install 'je_api_testka[schema]' pip install 'je_api_testka[security]' pip install 'je_api_testka[otel]' diff --git a/README/README_zh-TW.md b/README/README_zh-TW.md index e14f612..b1c5a71 100644 --- a/README/README_zh-TW.md +++ b/README/README_zh-TW.md @@ -231,8 +231,48 @@ cassette = Cassette("tape.json") # 離線 replay-or-record ```bash apitestka mock --host 0.0.0.0 --port 9000 +apitestka mock --config mock.json # HTTP routes 再加上 WebSocket 與 gRPC 端點 ``` +**WebSocket 與 gRPC 端點。** `WebSocketMockServer`(`websocket` extra)提供照腳本回應的路由: +連線時送出問候、已知訊息查回覆表、其他訊息走 fallback(`{{message}}` 會回聲,`null` 不回應)。 +`GrpcStubServer`(`grpc` extra)不需要編譯 stub,直接用完整路徑提供方法:固定的 unary 回應、 +server stream,或回傳某個狀態碼的錯誤。payload 是原始 bytes,所以序列化好的 protobuf 可以給 +產生出來的 client 用;`str` 以 UTF-8 送出,其他 JSON 值以精簡 JSON 送出。兩者都會保留收到的內容 +供斷言使用,未知的 WebSocket 路徑會以 HTTP 404 拒絕。 + +```python +from je_api_testka.utils.mock_server.websocket_mock import WebSocketMockServer, WebSocketRoute +from je_api_testka.utils.mock_server.grpc_stub import GrpcStubServer + +with WebSocketMockServer(port=0) as ws: # port 0 picks a free port + ws.add_route("/chat", WebSocketRoute(replies={"ping": "pong"})) + ... # connect to f"{ws.url}/chat" + ws.received("/chat") # frames the route got + +with GrpcStubServer(port=0) as grpc_mock: # needs the grpc extra + grpc_mock.add_unary_response("/shop.Catalog/Get", {"id": 1}) + ... # grpc.insecure_channel(grpc_mock.address) +``` + +`--config` 檔可以描述三種端點,每一段都可以省略: + +```json +{ + "http": {"routes": [{"rule": "/health", "body": {"ok": true}}], "openapi": "spec.json"}, + "websocket": {"port": 8765, "routes": {"/chat": {"greeting": "hi", "replies": {"ping": "pong"}}}}, + "grpc": {"port": 50051, "methods": { + "/shop.Catalog/Get": {"response": {"id": 1}}, + "/shop.Catalog/List": {"stream": [{"id": 1}, {"id": 2}]}, + "/shop.Catalog/Delete": {"error": {"code": "PERMISSION_DENIED", "details": "read only"}} + }} +} +``` + +JSON action 也能操作同樣的 mock:`AT_mock_start_websocket_server` / `AT_mock_start_grpc_server` +(`routes` / `methods` 用上面的設定格式)、`AT_mock_websocket_received` / `AT_mock_grpc_received`, +以及 `AT_mock_stop_websocket_server` / `AT_mock_stop_grpc_server`。 + ### Runner ```python @@ -446,6 +486,7 @@ je_api_testka/ ```bash pip install 'je_api_testka[websocket]' +pip install 'je_api_testka[grpc]' pip install 'je_api_testka[schema]' pip install 'je_api_testka[security]' pip install 'je_api_testka[otel]' diff --git a/architecture.md b/architecture.md index c82789d..f262f6d 100644 --- a/architecture.md +++ b/architecture.md @@ -25,7 +25,7 @@ CLI, a TCP socket server, an MCP server, a pytest plugin and an optional PySide6 | `je_api_testka/utils/assert_result/` | `check_result`, JSON Schema and JSONPath checks, snapshots | | `je_api_testka/utils/generate_report/` | HTML, JSON, XML, JUnit, Allure and Markdown reports; badge, run diff, trend store | | `je_api_testka/utils/callback/` | `callback_executor`: run a trigger command, then a callback | -| `je_api_testka/utils/mock_server/` | `flask_mock_server_instance` with dynamic, template, proxy, webhook and OpenAPI routes, plus fault injection | +| `je_api_testka/utils/mock_server/` | `flask_mock_server_instance` with dynamic, template, proxy, webhook and OpenAPI routes, plus fault injection; `WebSocketMockServer` (`websocket_mock.py`) and `GrpcStubServer` (`grpc_stub.py`) run in background threads; `protocol_mocks.py` keeps one of each for JSON actions; `mock_config.py` reads `apitestka mock --config` | | `je_api_testka/utils/socket_server/` | `start_apitestka_socket_server` (TCP command server) | | `je_api_testka/utils/package_manager/` | `package_manager`: loads an installed package's members into the executor | | `je_api_testka/utils/project/` | `create_project_dir` scaffolding (keyword and executor templates) | @@ -73,8 +73,8 @@ CLI, a TCP socket server, an MCP server, a pytest plugin and an optional PySide6 `apitestka_record`, `apitestka_clean_record` and `apitestka_mock_server`. - **GUI**: `je_api_testka.gui.main_window.APITestkaUI` (`python -m je_api_testka.gui.main_window`). The embeddable widget is `je_api_testka.gui.main_widget.APITestkaWidget`. -- **Packaging gap**: only `pyproject.toml` declares the console scripts, the `pytest11` entry point - and the extras other than `gui`. `dev.toml` (`je_api_testka_dev`) currently declares none of them. +- **Packaging**: `pyproject.toml` and `dev.toml` (`je_api_testka_dev`) declare the same console scripts, + `pytest11` entry point and extras; `test/test_utils/test_dev_toml_parity.py` keeps them equal. ## 4. Main flows @@ -107,13 +107,16 @@ MCP host → apitestka-mcp → build_server → dispatch_tool(name, args) → AP - **New report format**: module in `utils/generate_report/` that reads `test_record_instance` → `AT_generate_*` command → export from `__init__.py` → tests. - **New protocol backend**: `_wrapper/` package that records into `test_record_instance`, with - a lazy import → extra under `[project.optional-dependencies]` in `pyproject.toml` (`dev.toml` mirrors - only `gui`) → executor registration → tests with `pytest.importorskip` plus the missing-dependency path. + a lazy import → extra under `[project.optional-dependencies]` in `pyproject.toml` and `dev.toml`, and the + package in `.github/requirements/ci.in` (regenerate `ci.txt`) so CI runs its tests → executor registration → + tests with `pytest.importorskip` plus the missing-dependency path. - **MCP tool**: add an `MCPToolSpec` and handler to `APITESTKA_TOOLS`, then tests in `test/test_mcp_server/`. - **CLI subcommand**: add `_cmd_` and a subparser in `build_parser()` (`cli/cli_main.py`), then tests in `test/test_cli/`. - **Mock routes / runtime plugins**: `flask_mock_server_instance.add_router()` (or `add_dynamic_route()`, - `add_template_route()`); `AT_add_package_to_executor` registers an installed package's members. + `add_template_route()`); `WebSocketRoute` (reply map, fallback or `handler`) and `GrpcStubServer.register()` / + `register_server_stream()` for scripted protocol endpoints; `AT_add_package_to_executor` registers an installed + package's members. ## 6. Cross-project boundaries @@ -142,7 +145,7 @@ MCP host → apitestka-mcp → build_server → dispatch_tool(name, args) → AP justification comment. No `shell=True` (§ Security (Mandatory); § Static Analysis Compliance › Security). - Extend through `add_command_to_executor` or new `AT_` commands rather than reshaping the core map (§ Software Engineering Practices; § Common Development Workflows › Adding a New Executor Command). -- Import heavy optional dependencies lazily (PySide6, websockets, jsonschema, mcp). +- Import heavy optional dependencies lazily (PySide6, websockets, grpcio, jsonschema, mcp). `test_record_instance` must stay thread-safe (§ Performance Best Practices). - Limits: cognitive complexity ≤ 15, cyclomatic complexity ≤ 10, ≤ 7 parameters, functions ≤ 50 lines, files ≤ 500 lines, lines ≤ 120 characters (§ Static Analysis Compliance › Maintainability & Complexity). diff --git a/dev.toml b/dev.toml index 6b0a7d4..4842962 100644 --- a/dev.toml +++ b/dev.toml @@ -51,6 +51,7 @@ apitestka = "je_api_testka.pytest_plugin.plugin" [project.optional-dependencies] gui = ["PySide6==6.11.2", "qt-material"] websocket = ["websockets>=12.0"] +grpc = ["grpcio>=1.60"] schema = ["jsonschema>=4.0", "jsonpath-ng>=1.6"] security = ["pyjwt>=2.8", "botocore>=1.34"] otel = ["opentelemetry-api>=1.25", "opentelemetry-sdk>=1.25"] diff --git a/docs/source/Eng/doc/cli/cli_doc.rst b/docs/source/Eng/doc/cli/cli_doc.rst index a67bfe5..7c5789a 100644 --- a/docs/source/Eng/doc/cli/cli_doc.rst +++ b/docs/source/Eng/doc/cli/cli_doc.rst @@ -64,8 +64,8 @@ The ``apitestka`` command (installed with the package) groups the everyday tasks - Execute an action JSON file or every JSON file in a directory * - ``create PATH`` - Scaffold a project directory - * - ``mock [--host] [--port]`` - - Start the Flask mock server + * - ``mock [--host] [--port] [--config]`` + - Start the Flask mock server; ``--config`` adds WebSocket and gRPC endpoints (see Mock Server) * - ``import INPUT OUTPUT [--format openapi|postman]`` - Convert an OpenAPI document or Postman collection into action JSON * - ``repl`` diff --git a/docs/source/Eng/doc/mock_server/mock_server_doc.rst b/docs/source/Eng/doc/mock_server/mock_server_doc.rst index c9d92a3..e505d62 100644 --- a/docs/source/Eng/doc/mock_server/mock_server_doc.rst +++ b/docs/source/Eng/doc/mock_server/mock_server_doc.rst @@ -53,3 +53,84 @@ Custom Host and Port server = FlaskMockServer("0.0.0.0", 5000) server.add_router({"/health": lambda: "OK"}, methods=["GET"]) server.start_mock_server() + +WebSocket endpoints +------------------- + +``WebSocketMockServer`` needs the ``websocket`` extra. Each route can greet a new +connection, answer known messages from ``replies`` and handle the rest with ``fallback``: +``{{message}}`` in a reply is replaced by the incoming message, so the default fallback +echoes, and ``None`` sends nothing. A ``handler`` callable can replace both. The server runs +in a background thread; port 0 picks a free port. ``received(path)`` returns the text frames +a route got, and an unknown path is refused with HTTP 404 before the handshake. + +.. code-block:: python + + from je_api_testka.utils.mock_server.websocket_mock import WebSocketMockServer, WebSocketRoute + + with WebSocketMockServer(port=0) as ws: + ws.add_route("/chat", WebSocketRoute(greeting="hi", replies={"ping": "pong"})) + url = f"{ws.url}/chat" + ... + ws.received("/chat") + +gRPC endpoints +-------------- + +``GrpcStubServer`` needs the ``grpc`` extra and serves methods by full path +(``/package.Service/Method``) without compiled stubs: a fixed unary response, a +``bytes -> bytes`` handler (``register``), a server stream, or an error with a gRPC status +code. Payloads are raw bytes: ``bytes`` pass through (serialized protobuf works with +generated clients), ``str`` is UTF-8 and other JSON values are compact JSON. + +.. code-block:: python + + import grpc + from je_api_testka.utils.mock_server.grpc_stub import GrpcStubServer + + with GrpcStubServer(port=0) as mock: + mock.add_unary_response("/shop.Catalog/Get", {"id": 1}) + mock.add_stream_responses("/shop.Catalog/List", [{"id": 1}, {"id": 2}]) + mock.add_error("/shop.Catalog/Delete", "PERMISSION_DENIED", "read only") + with grpc.insecure_channel(mock.address) as channel: + channel.unary_unary("/shop.Catalog/Get")(b"{}") # b'{"id":1}' + mock.received("/shop.Catalog/Get") # [b'{}'] + +From JSON actions +----------------- + +One WebSocket mock and one gRPC mock can run at a time: +``AT_mock_start_websocket_server`` (``routes``, ``host``, ``port``; returns the base URL), +``AT_mock_start_grpc_server`` (``methods``, ``host``, ``port``; returns ``host:port``), +``AT_mock_websocket_received`` / ``AT_mock_grpc_received`` and the matching +``AT_mock_stop_*_server`` commands. Routes and methods use the config form below. + +.. code-block:: json + + [ + ["AT_mock_start_websocket_server", {"port": 8765, "routes": {"/chat": {"replies": {"ping": "pong"}}}}], + ["AT_test_api_method_websocket", {"url": "ws://127.0.0.1:8765/chat", "messages": ["ping"]}], + ["AT_mock_websocket_received", {"path": "/chat"}], + ["AT_mock_stop_websocket_server"] + ] + +Config file +----------- + +``apitestka mock --config mock.json`` reads one JSON file with optional ``http``, +``websocket`` and ``grpc`` sections, starts the WebSocket and gRPC mocks, then serves HTTP +until stopped. A relative ``openapi`` path is resolved against the config file. A gRPC +method takes exactly one of ``response``, ``response_base64`` (raw bytes such as a +serialized protobuf), ``stream`` or ``error``. + +.. code-block:: json + + { + "http": {"routes": [{"rule": "/health", "body": {"ok": true}}], "openapi": "spec.json"}, + "websocket": {"port": 8765, "routes": {"/chat": {"greeting": "hi", "replies": {"ping": "pong"}}}}, + "grpc": {"port": 50051, "methods": { + "/shop.Catalog/Get": {"response": {"id": 1}}, + "/shop.Catalog/List": {"stream": [{"id": 1}, {"id": 2}]}, + "/shop.Catalog/Delete": {"error": {"code": "PERMISSION_DENIED", "details": "read only"}} + }} + } diff --git a/docs/source/Zh/doc/cli/cli_doc.rst b/docs/source/Zh/doc/cli/cli_doc.rst index 6917149..ff897bb 100644 --- a/docs/source/Zh/doc/cli/cli_doc.rst +++ b/docs/source/Zh/doc/cli/cli_doc.rst @@ -64,8 +64,8 @@ CLI 參數 - 執行一個動作 JSON 檔，或目錄裡每個 JSON 檔 * - ``create PATH`` - 建立專案目錄 - * - ``mock [--host] [--port]`` - - 啟動 Flask 模擬伺服器 + * - ``mock [--host] [--port] [--config]`` + - 啟動 Flask 模擬伺服器；``--config`` 另外加上 WebSocket 與 gRPC 端點（見模擬伺服器一章） * - ``import INPUT OUTPUT [--format openapi|postman]`` - 把 OpenAPI 文件或 Postman collection 轉成動作 JSON * - ``repl`` diff --git a/docs/source/Zh/doc/mock_server/mock_server_doc.rst b/docs/source/Zh/doc/mock_server/mock_server_doc.rst index 7eddd2f..430905a 100644 --- a/docs/source/Zh/doc/mock_server/mock_server_doc.rst +++ b/docs/source/Zh/doc/mock_server/mock_server_doc.rst @@ -53,3 +53,80 @@ APITestka 內建基於 Flask 的模擬伺服器，用於本地測試。 server = FlaskMockServer("0.0.0.0", 5000) server.add_router({"/health": lambda: "OK"}, methods=["GET"]) server.start_mock_server() + +WebSocket 端點 +-------------- + +``WebSocketMockServer`` 需要 ``websocket`` extra。每條路由可以在連線時送出問候、用 ``replies`` +回覆已知訊息，其他訊息交給 ``fallback``：回覆中的 ``{{message}}`` 會換成收到的訊息，所以預設 +fallback 會回聲，``None`` 則不回應。``handler`` 函式可以取代兩者。伺服器在背景執行緒執行， +port 0 會挑一個空的埠。``received(path)`` 回傳該路由收到的文字 frame，未知的路徑會在握手前以 +HTTP 404 拒絕。 + +.. code-block:: python + + from je_api_testka.utils.mock_server.websocket_mock import WebSocketMockServer, WebSocketRoute + + with WebSocketMockServer(port=0) as ws: + ws.add_route("/chat", WebSocketRoute(greeting="hi", replies={"ping": "pong"})) + url = f"{ws.url}/chat" + ... + ws.received("/chat") + +gRPC 端點 +--------- + +``GrpcStubServer`` 需要 ``grpc`` extra，不用編譯 stub，直接以完整路徑（``/package.Service/Method``） +提供方法：固定的 unary 回應、``bytes -> bytes`` 的處理函式（``register``）、server stream， +或回傳 gRPC 狀態碼的錯誤。payload 是原始 bytes：``bytes`` 原樣送出（序列化好的 protobuf 可給 +產生出來的 client 用），``str`` 以 UTF-8 送出，其他 JSON 值以精簡 JSON 送出。 + +.. code-block:: python + + import grpc + from je_api_testka.utils.mock_server.grpc_stub import GrpcStubServer + + with GrpcStubServer(port=0) as mock: + mock.add_unary_response("/shop.Catalog/Get", {"id": 1}) + mock.add_stream_responses("/shop.Catalog/List", [{"id": 1}, {"id": 2}]) + mock.add_error("/shop.Catalog/Delete", "PERMISSION_DENIED", "read only") + with grpc.insecure_channel(mock.address) as channel: + channel.unary_unary("/shop.Catalog/Get")(b"{}") # b'{"id":1}' + mock.received("/shop.Catalog/Get") # [b'{}'] + +從 JSON action 使用 +------------------- + +同時最多各跑一個 WebSocket mock 與 gRPC mock：``AT_mock_start_websocket_server``（``routes``、 +``host``、``port``；回傳 base URL）、``AT_mock_start_grpc_server``（``methods``、``host``、 +``port``；回傳 ``host:port``）、``AT_mock_websocket_received`` / ``AT_mock_grpc_received``， +以及對應的 ``AT_mock_stop_*_server``。路由與方法使用下面設定檔的格式。 + +.. code-block:: json + + [ + ["AT_mock_start_websocket_server", {"port": 8765, "routes": {"/chat": {"replies": {"ping": "pong"}}}}], + ["AT_test_api_method_websocket", {"url": "ws://127.0.0.1:8765/chat", "messages": ["ping"]}], + ["AT_mock_websocket_received", {"path": "/chat"}], + ["AT_mock_stop_websocket_server"] + ] + +設定檔 +------ + +``apitestka mock --config mock.json`` 讀一個 JSON 檔，其中 ``http``、``websocket``、``grpc`` +三段都可省略；先啟動 WebSocket 與 gRPC mock，再持續提供 HTTP 直到停止。相對路徑的 ``openapi`` +以設定檔所在目錄為準。一個 gRPC 方法必須剛好給 ``response``、``response_base64``（原始 bytes， +例如序列化好的 protobuf）、``stream``、``error`` 其中之一。 + +.. code-block:: json + + { + "http": {"routes": [{"rule": "/health", "body": {"ok": true}}], "openapi": "spec.json"}, + "websocket": {"port": 8765, "routes": {"/chat": {"greeting": "hi", "replies": {"ping": "pong"}}}}, + "grpc": {"port": 50051, "methods": { + "/shop.Catalog/Get": {"response": {"id": 1}}, + "/shop.Catalog/List": {"stream": [{"id": 1}, {"id": 2}]}, + "/shop.Catalog/Delete": {"error": {"code": "PERMISSION_DENIED", "details": "read only"}} + }} + } diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 2df32c7..9c6fe61 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -23,3 +23,17 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Tests**: new `test/test_utils/test_generated_actions_run.py` runs each converter's output against the local mock and checks for success records and no errors; 3 new cases in `test/test_diff/test_sla_check.py`. Checked in a worktree of `4810743`: 11 of the new tests fail on the old code. The existing tests that pinned the dict shape now assert the list form. - **Result / numbers**: `pytest` 460 passed, 2 skipped (447 before); `sphinx -W` clean. - **Files**: `je_api_testka/utils/executor/{request_action,action_executor}.py`, `je_api_testka/{cli/import_specs,cli/scaffold_test,integrations/curl_import,integrations/har_import,ai/test_generator,mcp_server/tool_definitions,diff/sla_check}.py`; README ×3, `docs/source/{Eng,Zh}/doc/{integrations,diff,runner}`, `architecture.md` §3. + +## U-20261001-03 · 2026-10-01 · Mock server serves WebSocket and gRPC endpoints · #done #mock #grpc #websocket + +- **What** (`progress.md` #8): the mock server can now serve WebSocket and gRPC endpoints next to HTTP. + - `utils/mock_server/websocket_mock.py`: `WebSocketMockServer` runs a `websockets` sync server in a daemon thread (port 0 picks a free port). Each `WebSocketRoute` can send a greeting, answer from a reply map (a string or a list of frames) and fall back to `{{message}}`, which echoes, or `None`, which stays silent. A `handler` callable can replace both. It keeps every received frame per path, and an unknown path gets HTTP 404 before the handshake. + - `grpc_stub.py` (unary `bytes -> bytes` only before, and unreachable from JSON or the CLI): fixed unary responses, server streams, `add_error` with any gRPC status code, request capture, port 0, context manager. Payloads: `bytes` as-is (serialized protobuf), `str` as UTF-8, other JSON values as compact JSON. `configure_grpc_method` reads the JSON form (`response` / `response_base64` / `stream` / `error`). + - `protocol_mocks.py` keeps one of each for JSON actions: `AT_mock_start_websocket_server`, `AT_mock_stop_websocket_server`, `AT_mock_websocket_received`, `AT_mock_start_grpc_server`, `AT_mock_stop_grpc_server`, `AT_mock_grpc_received`. + - `mock_config.py` plus `apitestka mock --config FILE`: one JSON file with optional `http` (template routes, an `openapi` file relative to the config), `websocket` and `grpc` sections. The protocol mocks stop when the HTTP server exits. +- **Packaging**: new `grpc` extra (`grpcio>=1.60`) in `pyproject.toml` and `dev.toml`. `.github/requirements/ci.in` adds `websockets>=12.0` and `grpcio>=1.60`, and `ci.txt` is regenerated (only additions: grpcio 1.84.0, which has wheels for cp310–cp314 on Linux, macOS and Windows, and websockets 16.1.1 / 17.1), so CI runs these tests instead of skipping them. `uv.lock` adds grpcio. +- **Compatibility**: the WebSocket mock rejects unknown paths with `connection.protocol.reject`, because `ServerConnection.respond` does not exist in websockets 12.0, which the extra still allows. On 12.0 the handler failed and the client saw a 500, so the test now asserts the 404. Checked on Python 3.10 with websockets 12.0 and on 3.11 with 17.1. `GrpcStubServer` raises `MockServerException`, a subclass of the old `APITesterException`, when grpcio is missing. +- **Also**: `architecture.md` said `dev.toml` declares no scripts or extras; it has carried them since U-20260923-04. That line and the protocol-backend recipe are corrected. +- **Tests**: `test/test_mock_server/test_websocket_mock.py` (new; includes a round trip through APITestka's own `test_api_method_websocket`), `test_grpc_stub.py` (extended), `test_protocol_mocks.py` and `test_mock_config.py` (new, including `apitestka mock --config` with the blocking Flask run patched out). +- **Result / numbers**: `pytest` 522 passed, 2 skipped (460 before); `sphinx -W` clean. +- **Files**: `je_api_testka/utils/mock_server/{websocket_mock,grpc_stub,protocol_mocks,mock_config}.py`, `je_api_testka/utils/executor/action_executor.py`, `je_api_testka/cli/cli_main.py`, `pyproject.toml`, `dev.toml`, `.github/requirements/ci.{in,txt}`, `uv.lock`; README ×3, `docs/source/{Eng,Zh}/doc/{mock_server,cli}`, `architecture.md` §2, §3, §5, §7. diff --git a/docs/updates/README.md b/docs/updates/README.md index 6958192..065330c 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261001-03 | 2026-10-01 | Mock server serves WebSocket and gRPC endpoints | #done #mock #grpc #websocket | [2026-10](2026-10.md) | | U-20261001-02 | 2026-10-01 | Imported, scaffolded and AI-generated actions run as they are | #bugfix #executor | [2026-10](2026-10.md) | | U-20261001-01 | 2026-10-01 | records_to_openapi reachable from the CLI and MCP | #done #openapi #cli #mcp | [2026-10](2026-10.md) | | U-20260925-03 | 2026-09-25 | Python classifiers list every version CI tests | #packaging #tests | [2026-09](2026-09.md) | @@ -88,5 +89,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 2 | +| [2026-10.md](2026-10.md) | 2026-10 | 3 | | [2026-09.md](2026-09.md) | 2026-09 | 23 | diff --git a/je_api_testka/cli/cli_main.py b/je_api_testka/cli/cli_main.py index 82945bd..7d29885 100644 --- a/je_api_testka/cli/cli_main.py +++ b/je_api_testka/cli/cli_main.py @@ -55,9 +55,18 @@ def _cmd_create(args: argparse.Namespace) -> int: def _cmd_mock(args: argparse.Namespace) -> int: from je_api_testka.utils.mock_server.flask_mock_server import FlaskMockServer + from je_api_testka.utils.mock_server.mock_config import apply_mock_config, read_mock_config, stop_protocol_mocks server = FlaskMockServer(args.host, args.port) - server.start_mock_server() + if args.config: + endpoints = apply_mock_config(read_mock_config(args.config), server, + base_dir=str(Path(args.config).resolve().parent)) + for kind, address in endpoints.items(): + apitestka_logger.info(f"cli mock: {kind} mock at {address}") + try: + server.start_mock_server() + finally: + stop_protocol_mocks() return 0 @@ -150,6 +159,7 @@ def build_parser() -> argparse.ArgumentParser: mock_parser = sub.add_parser("mock", help="Start the Flask mock server") mock_parser.add_argument("--host", default=DEFAULT_MOCK_HOST) mock_parser.add_argument("--port", type=int, default=DEFAULT_MOCK_PORT) + mock_parser.add_argument("--config", help="JSON file with http, websocket and grpc mock endpoints") mock_parser.set_defaults(func=_cmd_mock) import_parser = sub.add_parser("import", help="Convert specs to action JSON") diff --git a/je_api_testka/utils/executor/action_executor.py b/je_api_testka/utils/executor/action_executor.py index f21365d..e0f6ac5 100644 --- a/je_api_testka/utils/executor/action_executor.py +++ b/je_api_testka/utils/executor/action_executor.py @@ -58,6 +58,14 @@ from je_api_testka.utils.json.json_file.json_file import read_action_json from je_api_testka.utils.logging.loggin_instance import apitestka_logger from je_api_testka.utils.mock_server.flask_mock_server import flask_mock_server_instance +from je_api_testka.utils.mock_server.protocol_mocks import ( + grpc_mock_received, + start_grpc_mock, + start_websocket_mock, + stop_grpc_mock, + stop_websocket_mock, + websocket_mock_received, +) from je_api_testka.utils.package_manager.package_manager_class import package_manager from je_api_testka.websocket_wrapper.websocket_method import test_api_method_websocket @@ -174,6 +182,13 @@ def __init__(self): "AT_mock_add_webhook": flask_mock_server_instance.add_webhook, "AT_mock_add_proxy": flask_mock_server_instance.add_proxy, "AT_mock_load_openapi": flask_mock_server_instance.load_openapi, + # WebSocket and gRPC mocks (one of each, running in background threads) + "AT_mock_start_websocket_server": start_websocket_mock, + "AT_mock_stop_websocket_server": stop_websocket_mock, + "AT_mock_websocket_received": websocket_mock_received, + "AT_mock_start_grpc_server": start_grpc_mock, + "AT_mock_stop_grpc_server": stop_grpc_mock, + "AT_mock_grpc_received": grpc_mock_received, # Runner "AT_run_actions_parallel": run_actions_parallel, "AT_filter_actions_by_tag": filter_actions_by_tag, diff --git a/je_api_testka/utils/mock_server/grpc_stub.py b/je_api_testka/utils/mock_server/grpc_stub.py index 9b032ae..45bc325 100644 --- a/je_api_testka/utils/mock_server/grpc_stub.py +++ b/je_api_testka/utils/mock_server/grpc_stub.py @@ -1,26 +1,47 @@ """ -gRPC mock stub. - -We do not pull in the heavyweight ``grpcio`` package by default. This module -provides: - -* :func:`is_grpc_available` checks whether the optional dependency exists. -* :class:`GrpcStubServer` lazily imports grpcio and exposes ``register`` / - ``start`` / ``stop`` for unary RPCs only. - -For full gRPC testing, install ``grpcio`` separately. +gRPC mock endpoints. + +``grpcio`` is optional (the ``grpc`` extra) and imported lazily. +:class:`GrpcStubServer` serves methods by their full path +(``/package.Service/Method``) without compiled stubs: + +* unary methods answer with a fixed payload (:meth:`GrpcStubServer.add_unary_response`) + or a ``bytes -> bytes`` callable (:meth:`GrpcStubServer.register`); +* server-streaming methods answer with several messages + (:meth:`GrpcStubServer.add_stream_responses`, :meth:`GrpcStubServer.register_server_stream`); +* :meth:`GrpcStubServer.add_error` makes a method fail with a gRPC status code. + +Payloads are raw bytes on the wire: ``bytes`` pass through, ``str`` is UTF-8 and +any other JSON value (object, array, number, boolean, null) is compact JSON. Serialized protobuf messages +(``message.SerializeToString()``) therefore work with generated clients, and JSON +payloads suit clients that use JSON (de)serializers. Every request is kept per +method for later assertions. """ from __future__ import annotations -from typing import Callable, Dict +import base64 +import binascii +import json +import threading +from concurrent.futures import ThreadPoolExecutor +from typing import Callable, Dict, Iterable, List, Mapping, Sequence, Tuple, Union -from je_api_testka.utils.exception.exceptions import APITesterException +from je_api_testka.utils.exception.exceptions import MockServerException from je_api_testka.utils.logging.loggin_instance import apitestka_logger -GRPCIO_NOT_INSTALLED: str = "grpcio not installed. Install with `pip install grpcio`." +GRPCIO_NOT_INSTALLED: str = "grpcio is not installed. Install with `pip install je_api_testka[grpc]`." +DEFAULT_GRPC_MOCK_HOST: str = "127.0.0.1" +DEFAULT_GRPC_MOCK_PORT: int = 50051 +_MAX_WORKERS: int = 4 +_STOP_WAIT_SECONDS: float = 5.0 +_UNARY: str = "unary" +_STREAM: str = "stream" + +Payload = Union[bytes, str, dict, list, int, float, bool, None] def is_grpc_available() -> bool: + """Return True when ``grpcio`` can be imported.""" try: import grpc # type: ignore # noqa: F401 return True @@ -28,43 +49,111 @@ def is_grpc_available() -> bool: return False +def encode_payload(value: Payload) -> bytes: + """Return the wire bytes for ``value``: bytes as-is, str as UTF-8, any other JSON value as compact JSON.""" + if isinstance(value, (bytes, bytearray)): + return bytes(value) + if isinstance(value, str): + return value.encode("utf-8") + if value is None or isinstance(value, (dict, list, int, float, bool)): + return json.dumps(value, separators=(",", ":"), ensure_ascii=False).encode("utf-8") + raise MockServerException(f"unsupported gRPC payload type: {type(value).__name__}") + + class GrpcStubServer: - """Wrapper around a generic-handler-only gRPC server. Unary calls only.""" + """Generic-handler gRPC server for tests. Port 0 picks a free port, readable from :attr:`port`.""" - def __init__(self, host: str = "127.0.0.1", port: int = 50051) -> None: + def __init__(self, host: str = DEFAULT_GRPC_MOCK_HOST, port: int = DEFAULT_GRPC_MOCK_PORT) -> None: if not is_grpc_available(): - raise APITesterException(GRPCIO_NOT_INSTALLED) + raise MockServerException(GRPCIO_NOT_INSTALLED) import grpc # type: ignore - from concurrent.futures import ThreadPoolExecutor self._grpc = grpc - self._server = grpc.server(ThreadPoolExecutor(max_workers=4)) - self._handlers: Dict[str, Callable[[bytes], bytes]] = {} - self._address = f"{host}:{port}" - self._server.add_insecure_port(self._address) - apitestka_logger.info(f"GrpcStubServer listening on {self._address}") + self._server = grpc.server(ThreadPoolExecutor(max_workers=_MAX_WORKERS)) + self._methods: Dict[str, Tuple[str, Callable]] = {} + self._received: Dict[str, List[bytes]] = {} + self._lock = threading.Lock() + self._started = False + self.host = host + self.port = self._server.add_insecure_port(f"{host}:{port}") + + @property + def address(self) -> str: + """``host:port`` to pass to ``grpc.insecure_channel``.""" + return f"{self.host}:{self.port}" def register(self, method_path: str, handler: Callable[[bytes], bytes]) -> None: - """Register a unary handler: bytes-in, bytes-out.""" - self._handlers[method_path] = handler - - def start(self) -> None: - grpc = self._grpc - - def _generic(handler_call_details): - handler = self._handlers.get(handler_call_details.method) - if handler is None: - return None - return grpc.unary_unary_rpc_method_handler( - lambda request, _ctx, fn=handler: fn(request), - request_deserializer=lambda blob: blob, - response_serializer=lambda blob: blob, - ) - - self._server.add_generic_rpc_handlers((_GenericService(_generic),)) + """Register a unary handler: request bytes in, response bytes out.""" + self._methods[method_path] = (_UNARY, lambda request, _context: handler(request)) + + def register_server_stream(self, method_path: str, handler: Callable[[bytes], Iterable[bytes]]) -> None: + """Register a server-streaming handler: request bytes in, an iterable of response bytes out.""" + self._methods[method_path] = (_STREAM, lambda request, _context: iter(handler(request))) + + def add_unary_response(self, method_path: str, response: Payload) -> None: + """Answer every call of a unary method with ``response``.""" + wire = encode_payload(response) + self.register(method_path, lambda _request: wire) + + def add_stream_responses(self, method_path: str, responses: Sequence[Payload]) -> None: + """Answer every call of a server-streaming method with ``responses``, in order.""" + wires = [encode_payload(response) for response in responses] + self.register_server_stream(method_path, lambda _request: wires) + + def add_error(self, method_path: str, code: str, details: str = "") -> None: + """Fail every unary call of ``method_path`` with gRPC status ``code`` (e.g. ``"NOT_FOUND"``).""" + try: + status = self._grpc.StatusCode[code.upper()] + except KeyError as error: + raise MockServerException(f"unknown gRPC status code: {code!r}") from error + + def _abort(_request: bytes, context) -> bytes: + context.abort(status, details) + return b"" # not reached: abort raises + + self._methods[method_path] = (_UNARY, _abort) + + def received(self, method_path: str) -> List[bytes]: + """Return a copy of the request payloads received by ``method_path`` so far.""" + with self._lock: + return list(self._received.get(method_path, [])) + + def start(self) -> str: + """Start serving and return :attr:`address`.""" + if self._started: + raise MockServerException("gRPC mock server is already running") + self._server.add_generic_rpc_handlers((_GenericService(self._lookup),)) self._server.start() + self._started = True + apitestka_logger.info(f"GrpcStubServer listening on {self.address}") + return self.address def stop(self, grace: float = 0.0) -> None: - self._server.stop(grace) + """Stop serving; in-flight calls get ``grace`` seconds to finish.""" + self._server.stop(grace).wait(_STOP_WAIT_SECONDS) + self._started = False + + def __enter__(self) -> "GrpcStubServer": + self.start() + return self + + def __exit__(self, *_exc_info: object) -> None: + self.stop() + + def _lookup(self, handler_call_details): + method_path = handler_call_details.method + entry = self._methods.get(method_path) + if entry is None: + return None + kind, behavior = entry + + def _recorded(request: bytes, context): + with self._lock: + self._received.setdefault(method_path, []).append(request) + return behavior(request, context) + + if kind == _STREAM: + return self._grpc.unary_stream_rpc_method_handler(_recorded) + return self._grpc.unary_unary_rpc_method_handler(_recorded) class _GenericService: @@ -75,3 +164,40 @@ def __init__(self, callback) -> None: def service(self, handler_call_details): return self._callback(handler_call_details) + + +def _config_payload(config: Mapping[str, object]) -> Payload: + if "response_base64" in config: + try: + return base64.b64decode(str(config["response_base64"]), validate=True) + except (binascii.Error, ValueError) as error: + raise MockServerException("gRPC 'response_base64' is not valid base64") from error + return config["response"] # type: ignore[return-value] + + +def configure_grpc_method(server: GrpcStubServer, method_path: str, config: Mapping[str, object]) -> None: + """ + Register one method from its JSON form. + + Exactly one of ``response`` (JSON value or string), ``response_base64`` (raw bytes, + e.g. a serialized protobuf), ``stream`` (list of payloads) or + ``error`` (``{"code": "NOT_FOUND", "details": "..."}``) must be given. + + :raises MockServerException: when none or several are given, or a value has the wrong type. + """ + kinds = [key for key in ("response", "response_base64", "stream", "error") if key in config] + if len(kinds) != 1 or set(config) - set(kinds): + raise MockServerException( + f"gRPC method {method_path} needs exactly one of response, response_base64, stream, error" + ) + if "stream" in config: + if not isinstance(config["stream"], list): + raise MockServerException("gRPC 'stream' must be a list") + server.add_stream_responses(method_path, config["stream"]) + elif "error" in config: + error = config["error"] + if not isinstance(error, dict) or not isinstance(error.get("code"), str): + raise MockServerException("gRPC 'error' must be an object with a string 'code'") + server.add_error(method_path, error["code"], str(error.get("details", ""))) + else: + server.add_unary_response(method_path, _config_payload(config)) diff --git a/je_api_testka/utils/mock_server/mock_config.py b/je_api_testka/utils/mock_server/mock_config.py new file mode 100644 index 0000000..51e904f --- /dev/null +++ b/je_api_testka/utils/mock_server/mock_config.py @@ -0,0 +1,113 @@ +""" +Declarative mock configuration for ``apitestka mock --config``. + +One JSON file describes HTTP, WebSocket and gRPC endpoints:: + + { + "http": {"routes": [{"rule": "/health", "body": {"ok": true}}], "openapi": "spec.json"}, + "websocket": {"port": 8765, "routes": {"/chat": {"greeting": "hi", "replies": {"ping": "pong"}}}}, + "grpc": {"port": 50051, "methods": {"/shop.Catalog/Get": {"response": {"id": 1}}}} + } + +Every section is optional. HTTP routes go onto the given :class:`FlaskMockServer`; +the WebSocket and gRPC mocks start in background threads through +:mod:`protocol_mocks`. A relative ``openapi`` path is resolved against the +config file's directory. +""" +from __future__ import annotations + +import json +from pathlib import Path +from typing import Dict, Mapping, Optional + +from je_api_testka.utils.exception.exceptions import MockServerException +from je_api_testka.utils.mock_server.flask_mock_server import FlaskMockServer +from je_api_testka.utils.mock_server.protocol_mocks import ( + start_grpc_mock, + start_websocket_mock, + stop_grpc_mock, + stop_websocket_mock, +) + +_SECTIONS: frozenset = frozenset({"http", "websocket", "grpc"}) +_ROUTE_KEYS: frozenset = frozenset({"rule", "body", "status", "methods"}) + + +def read_mock_config(path: str) -> dict: + """Read a mock config file; raise :class:`MockServerException` when it is not a JSON object.""" + try: + config = json.loads(Path(path).read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as error: + raise MockServerException(f"{path}: cannot read mock config: {error!r}") from error + if not isinstance(config, dict): + raise MockServerException(f"{path}: a mock config must be a JSON object") + return config + + +def _section(config: Mapping[str, object], name: str) -> Optional[Mapping[str, object]]: + value = config.get(name) + if value is not None and not isinstance(value, dict): + raise MockServerException(f"mock config '{name}' must be an object") + return value + + +def _apply_http(server: FlaskMockServer, http: Mapping[str, object], base_dir: Path) -> None: + routes = http.get("routes", []) + if not isinstance(routes, list): + raise MockServerException("mock config 'http.routes' must be a list") + for route in routes: + if not isinstance(route, dict) or "rule" not in route or set(route) - _ROUTE_KEYS: + raise MockServerException(f"bad http route {route!r}; keys: rule, body, status, methods") + server.add_template_route(route["rule"], route.get("body", ""), status=route.get("status", 200), + methods=route.get("methods")) + if "openapi" in http: + spec_path = base_dir / str(http["openapi"]) + try: + spec = json.loads(spec_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as error: + raise MockServerException(f"{spec_path}: cannot read OpenAPI document: {error!r}") from error + server.load_openapi(spec) + + +def _address(section: Mapping[str, object]) -> dict: + return {key: section[key] for key in ("host", "port") if key in section} + + +def _start_protocol_mocks(config: Mapping[str, object]) -> Dict[str, str]: + websocket = _section(config, "websocket") + grpc = _section(config, "grpc") + endpoints: Dict[str, str] = {} + if websocket is not None: + endpoints["websocket"] = start_websocket_mock(websocket.get("routes"), **_address(websocket)) + if grpc is not None: + try: + endpoints["grpc"] = start_grpc_mock(grpc.get("methods") or {}, **_address(grpc)) + except Exception: + if "websocket" in endpoints: + stop_websocket_mock() + raise + return endpoints + + +def apply_mock_config(config: Mapping[str, object], http_server: FlaskMockServer, + base_dir: str = ".") -> Dict[str, str]: + """ + Register the config's HTTP routes on ``http_server`` and start its WebSocket and gRPC mocks. + + :return: the started endpoints, e.g. ``{"websocket": "ws://127.0.0.1:8765", "grpc": "127.0.0.1:50051"}``. + :raises MockServerException: on an unknown section or a malformed entry. A WebSocket mock + started by this call is stopped again when the gRPC mock then fails. + """ + unknown = set(config) - _SECTIONS + if unknown: + raise MockServerException(f"unknown mock config sections: {sorted(unknown)}") + http = _section(config, "http") + if http is not None: + _apply_http(http_server, http, Path(base_dir)) + return _start_protocol_mocks(config) + + +def stop_protocol_mocks() -> None: + """Stop the WebSocket and gRPC mocks started from a config, if any.""" + stop_websocket_mock() + stop_grpc_mock() diff --git a/je_api_testka/utils/mock_server/protocol_mocks.py b/je_api_testka/utils/mock_server/protocol_mocks.py new file mode 100644 index 0000000..ae5f05d --- /dev/null +++ b/je_api_testka/utils/mock_server/protocol_mocks.py @@ -0,0 +1,114 @@ +""" +Start and stop the WebSocket and gRPC mocks by name, for JSON actions. + +JSON actions cannot hold a server object between steps, so this module keeps at +most one running WebSocket mock and one running gRPC mock. Route and method +definitions use the JSON forms of :func:`route_from_config` and +:func:`configure_grpc_method`. +""" +from __future__ import annotations + +import threading +from typing import Dict, List, Mapping, Optional, Union + +from je_api_testka.utils.exception.exceptions import MockServerException +from je_api_testka.utils.mock_server.grpc_stub import ( + DEFAULT_GRPC_MOCK_HOST, + DEFAULT_GRPC_MOCK_PORT, + GrpcStubServer, + configure_grpc_method, +) +from je_api_testka.utils.mock_server.websocket_mock import ( + DEFAULT_WS_MOCK_HOST, + DEFAULT_WS_MOCK_PORT, + WebSocketMockServer, + route_from_config, +) + +_WEBSOCKET: str = "websocket" +_GRPC: str = "grpc" +_MockServer = Union[WebSocketMockServer, GrpcStubServer] +_running: Dict[str, _MockServer] = {} +_lock = threading.Lock() + + +def _claim(kind: str, server: _MockServer) -> None: + with _lock: + if kind in _running: + raise MockServerException(f"a {kind} mock is already running; stop it first") + _running[kind] = server + + +def _running_server(kind: str) -> _MockServer: + with _lock: + server = _running.get(kind) + if server is None: + raise MockServerException(f"no {kind} mock is running") + return server + + +def start_websocket_mock(routes: Optional[Mapping[str, Mapping[str, object]]] = None, + host: str = DEFAULT_WS_MOCK_HOST, port: int = DEFAULT_WS_MOCK_PORT) -> str: + """ + Start the WebSocket mock and return its base URL. + + ``routes`` maps a path to its JSON route form; without routes, ``/`` echoes. + """ + server = WebSocketMockServer(host, port) + for path, config in (routes or {"/": {}}).items(): + server.add_route(path, route_from_config(config)) + _claim(_WEBSOCKET, server) + try: + return server.start() + except Exception: + with _lock: + _running.pop(_WEBSOCKET, None) + raise + + +def stop_websocket_mock() -> None: + """Stop the WebSocket mock if one is running.""" + with _lock: + server = _running.pop(_WEBSOCKET, None) + if server is not None: + server.stop() + + +def websocket_mock_received(path: str) -> List[str]: + """Return the text frames the running WebSocket mock received on ``path``.""" + return _running_server(_WEBSOCKET).received(path) + + +def start_grpc_mock(methods: Mapping[str, Mapping[str, object]], host: str = DEFAULT_GRPC_MOCK_HOST, + port: int = DEFAULT_GRPC_MOCK_PORT) -> str: + """ + Start the gRPC mock and return its ``host:port`` address. + + ``methods`` maps a full method path (``/package.Service/Method``) to its JSON form. + """ + if not methods: + raise MockServerException("a gRPC mock needs at least one method") + server = GrpcStubServer(host, port) + for method_path, config in methods.items(): + configure_grpc_method(server, method_path, config) + _claim(_GRPC, server) + try: + return server.start() + except Exception: + with _lock: + _running.pop(_GRPC, None) + raise + + +def stop_grpc_mock() -> None: + """Stop the gRPC mock if one is running.""" + with _lock: + server = _running.pop(_GRPC, None) + if server is not None: + server.stop() + + +def grpc_mock_received(method_path: str) -> List[str]: + """Return the requests the running gRPC mock received on ``method_path``, decoded as UTF-8.""" + return [request.decode("utf-8", errors="replace") + for request in _running_server(_GRPC).received(method_path)] diff --git a/je_api_testka/utils/mock_server/websocket_mock.py b/je_api_testka/utils/mock_server/websocket_mock.py new file mode 100644 index 0000000..8688590 --- /dev/null +++ b/je_api_testka/utils/mock_server/websocket_mock.py @@ -0,0 +1,165 @@ +""" +WebSocket mock endpoints. + +:class:`WebSocketMockServer` serves scripted WebSocket routes from a background +thread (the ``websocket`` extra's ``websockets`` package, imported lazily). +Each route may greet a new connection, answer known messages from a reply map, +and handle the rest with a fallback; ``{{message}}`` in a reply is replaced by +the incoming message, so the default fallback echoes. Every text frame a route +receives is kept for later assertions. An unknown path is refused with HTTP 404 +before the handshake. +""" +from __future__ import annotations + +import threading +from dataclasses import dataclass, field +from http import HTTPStatus +from typing import Callable, Dict, List, Mapping, Optional, Sequence, Union +from urllib.parse import urlsplit + +from je_api_testka.utils.exception.exceptions import MockServerException +from je_api_testka.utils.logging.loggin_instance import apitestka_logger + +MESSAGE_PLACEHOLDER: str = "{{message}}" +WEBSOCKETS_MISSING: str = "websockets is not installed. Install with `pip install je_api_testka[websocket]`." +DEFAULT_WS_MOCK_HOST: str = "127.0.0.1" +DEFAULT_WS_MOCK_PORT: int = 8765 +_JOIN_TIMEOUT_SECONDS: float = 5.0 + +Reply = Union[str, Sequence[str], None] +MessageHandler = Callable[[str], Reply] + + +def _import_sync_server(): + try: + from websockets.sync import server # type: ignore + except ImportError as error: + raise MockServerException(WEBSOCKETS_MISSING) from error + return server + + +@dataclass +class WebSocketRoute: + """ + Scripted behaviour of one WebSocket path. + + :param replies: exact incoming message → reply (a string or a list of frames). + :param fallback: reply for any other message; ``None`` sends nothing. + :param greeting: frame sent as soon as a client connects. + :param handler: optional callable overriding ``replies`` and ``fallback``. + """ + + replies: Dict[str, Reply] = field(default_factory=dict) + fallback: Optional[str] = MESSAGE_PLACEHOLDER + greeting: Optional[str] = None + handler: Optional[MessageHandler] = None + + def answer(self, message: str) -> List[str]: + """Return the frames to send back for ``message`` (possibly none).""" + reply = self.handler(message) if self.handler is not None else self.replies.get(message, self.fallback) + if reply is None: + return [] + frames = [reply] if isinstance(reply, str) else list(reply) + return [str(frame).replace(MESSAGE_PLACEHOLDER, message) for frame in frames] + + +class WebSocketMockServer: + """Scripted WebSocket server for tests. Port 0 picks a free port, readable from :attr:`port`.""" + + def __init__(self, host: str = DEFAULT_WS_MOCK_HOST, port: int = DEFAULT_WS_MOCK_PORT) -> None: + self.host = host + self.port = port + self._routes: Dict[str, WebSocketRoute] = {} + self._received: Dict[str, List[str]] = {} + self._lock = threading.Lock() + self._server = None + self._thread: Optional[threading.Thread] = None + + def add_route(self, path: str, route: Optional[WebSocketRoute] = None) -> WebSocketRoute: + """Serve ``path`` with ``route`` (an echo route by default) and return it.""" + if not path.startswith("/"): + raise MockServerException(f"websocket route must start with '/': {path!r}") + self._routes[path] = route or WebSocketRoute() + return self._routes[path] + + def received(self, path: str) -> List[str]: + """Return a copy of the text frames received on ``path`` so far.""" + with self._lock: + return list(self._received.get(path, [])) + + @property + def url(self) -> str: + """Base ``ws://host:port`` URL; append a route path to connect.""" + return f"ws://{self.host}:{self.port}" + + def start(self) -> str: + """Start serving in a daemon thread and return :attr:`url`.""" + if self._server is not None: + raise MockServerException("websocket mock server is already running") + server_module = _import_sync_server() + self._server = server_module.serve(self._handle, self.host, self.port, + process_request=self._reject_unknown_path) + self.port = self._server.socket.getsockname()[1] + self._thread = threading.Thread(target=self._server.serve_forever, daemon=True) + self._thread.start() + apitestka_logger.info(f"WebSocketMockServer listening on {self.url}") + return self.url + + def stop(self) -> None: + """Stop serving and close open connections; a no-op when not running.""" + if self._server is None: + return + self._server.shutdown() + if self._thread is not None: + self._thread.join(_JOIN_TIMEOUT_SECONDS) + self._server = None + self._thread = None + + def __enter__(self) -> "WebSocketMockServer": + self.start() + return self + + def __exit__(self, *_exc_info: object) -> None: + self.stop() + + def _reject_unknown_path(self, connection, request): + if urlsplit(request.path).path in self._routes: + return None + # protocol.reject also exists on websockets 12, where ServerConnection.respond does not. + return connection.protocol.reject(HTTPStatus.NOT_FOUND, f"no websocket route for {request.path}\n") + + def _handle(self, connection) -> None: + path = urlsplit(connection.request.path).path + route = self._routes[path] + if route.greeting is not None: + connection.send(route.greeting) + for message in connection: + text = message if isinstance(message, str) else message.decode("utf-8", errors="replace") + with self._lock: + self._received.setdefault(path, []).append(text) + for frame in route.answer(text): + connection.send(frame) + + +def route_from_config(config: Mapping[str, object]) -> WebSocketRoute: + """ + Build a :class:`WebSocketRoute` from its JSON form. + + Keys: ``replies`` (object), ``fallback`` (string or null), ``greeting`` (string). + A missing ``fallback`` echoes. + + :raises MockServerException: on an unknown key or a value of the wrong type. + """ + unknown = set(config) - {"replies", "fallback", "greeting"} + if unknown: + raise MockServerException(f"unknown websocket route keys: {sorted(unknown)}") + replies = config.get("replies", {}) + fallback = config.get("fallback", MESSAGE_PLACEHOLDER) + greeting = config.get("greeting") + if not isinstance(replies, dict): + raise MockServerException("websocket 'replies' must be an object") + if fallback is not None and not isinstance(fallback, str): + raise MockServerException("websocket 'fallback' must be a string or null") + if greeting is not None and not isinstance(greeting, str): + raise MockServerException("websocket 'greeting' must be a string") + return WebSocketRoute(replies=dict(replies), fallback=fallback, greeting=greeting) diff --git a/progress.md b/progress.md index 29fb5f3..23135c3 100644 --- a/progress.md +++ b/progress.md @@ -8,7 +8,6 @@ Cross-repo and workspace items live in `D:\Codes\progress.md` (relevant here: X- - **#3** [DECIDE] The PyPI classifier still says `Development Status :: 2 - Pre-Alpha`. - **#6** GUI redesign: restructure and restyle. `gui/main_widget.py` is one 605-line widget (over the 500-line limit) with six flat tabs. It should become a sidebar-navigated window with a request/response split view, and history and environment panels docked (`gui/history_panel.py`, `gui/env_manager_model.py` exist but are not wired in). It needs a refreshed theme (`gui/main_window.py:21` hard-codes `dark_amber.xml`), pages for the newer features (#8–#13), and all four language wrappers (English, Traditional Chinese, Simplified Chinese, Japanese). `APITestkaWidget` stays the embeddable entry point (PyBreeze contract, `architecture.md` §6). -- **#8** Mock server: gRPC and WebSocket endpoints. `utils/mock_server/grpc_stub.py` serves unary calls only and is not reachable from `apitestka mock`, the executor or `flask_mock_server_instance`. There is no WebSocket mock endpoint. - **#9** Pact-style bidirectional contract testing. `diff/contract_diff.py` only compares two OpenAPI documents. Missing: a consumer contract recorded from the consumer's tests (interactions plus matchers), provider verification replaying it against a running provider, and a cross-check of the consumer contract against the provider's OpenAPI. - **#10** Reference AI backend on the Anthropic API. `ai/backend.py:1` ships only `NoOpAIBackend` and `StaticAIBackend`. Missing: an optional `ai` extra, an `AnthropicAIBackend` (key from the environment, never stored), and a way to select it (CLI, executor). - **#11** Load-test bridge to LoadDensity. No way to turn APITestka actions into a LoadDensity run. It needs a converter to LoadDensity's action JSON (`LD_start_test` with `tasks`) and a runner that uses LoadDensity's `--execute_file` CLI. That is a new cross-project contract, so it also needs §6 in both repos and the root `AI_CONTEXT.md`. diff --git a/pyproject.toml b/pyproject.toml index c522cee..0993d27 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -51,6 +51,7 @@ apitestka = "je_api_testka.pytest_plugin.plugin" [project.optional-dependencies] gui = ["PySide6==6.11.2", "qt-material"] websocket = ["websockets>=12.0"] +grpc = ["grpcio>=1.60"] schema = ["jsonschema>=4.0", "jsonpath-ng>=1.6"] security = ["pyjwt>=2.8", "botocore>=1.34"] otel = ["opentelemetry-api>=1.25", "opentelemetry-sdk>=1.25"] diff --git a/test/test_executor_integration.py b/test/test_executor_integration.py index 70867e3..8ae97a1 100644 --- a/test/test_executor_integration.py +++ b/test/test_executor_integration.py @@ -53,6 +53,8 @@ # Mock server advanced "AT_mock_add_dynamic_route", "AT_mock_add_template_route", "AT_mock_add_webhook", "AT_mock_add_proxy", "AT_mock_load_openapi", + "AT_mock_start_websocket_server", "AT_mock_stop_websocket_server", "AT_mock_websocket_received", + "AT_mock_start_grpc_server", "AT_mock_stop_grpc_server", "AT_mock_grpc_received", # Runner "AT_run_actions_parallel", "AT_filter_actions_by_tag", "AT_order_actions", ) diff --git a/test/test_mock_server/test_grpc_stub.py b/test/test_mock_server/test_grpc_stub.py index e440891..80711bc 100644 --- a/test/test_mock_server/test_grpc_stub.py +++ b/test/test_mock_server/test_grpc_stub.py @@ -1,12 +1,15 @@ """Tests for the optional gRPC stub server.""" from __future__ import annotations +import base64 import builtins +import json import pytest -from je_api_testka.utils.exception.exceptions import APITesterException +from je_api_testka.utils.exception.exceptions import APITesterException, MockServerException from je_api_testka.utils.mock_server import grpc_stub +from je_api_testka.utils.mock_server.grpc_stub import GrpcStubServer, configure_grpc_method, encode_payload def test_is_grpc_available_reflects_import(monkeypatch): @@ -25,3 +28,97 @@ def test_grpc_stub_server_raises_when_dep_missing(monkeypatch): monkeypatch.setattr(grpc_stub, "is_grpc_available", lambda: False) with pytest.raises(APITesterException): grpc_stub.GrpcStubServer() + + +@pytest.mark.parametrize(("value", "wire"), [ + (b"\x01\x02", b"\x01\x02"), ("héllo", "héllo".encode("utf-8")), ({"a": 1}, b'{"a":1}'), ([1, 2], b"[1,2]"), + (3, b"3"), (True, b"true"), (None, b"null"), +]) +def test_encode_payload(value, wire): + assert encode_payload(value) == wire + + +def test_encode_payload_rejects_other_types(): + with pytest.raises(MockServerException): + encode_payload(object()) + + +@pytest.fixture +def grpc_module(): + return pytest.importorskip("grpc") + + +@pytest.fixture +def server(grpc_module): + stub = GrpcStubServer(port=0) + with stub: + yield stub + + +def _unary(grpc_module, server, method, request=b""): + with grpc_module.insecure_channel(server.address) as channel: + return channel.unary_unary(method)(request, timeout=5) + + +def test_unary_json_response_and_request_capture(grpc_module, server): + server.add_unary_response("/shop.Catalog/Get", {"id": 1}) + assert json.loads(_unary(grpc_module, server, "/shop.Catalog/Get", b'{"sku":"a"}')) == {"id": 1} + assert server.received("/shop.Catalog/Get") == [b'{"sku":"a"}'] + + +def test_callable_handler_sees_request(grpc_module, server): + server.register("/echo.Echo/Say", lambda request: request.upper()) + assert _unary(grpc_module, server, "/echo.Echo/Say", b"hi") == b"HI" + + +def test_server_stream(grpc_module, server): + server.add_stream_responses("/shop.Catalog/List", [{"id": 1}, "two", b"3"]) + with grpc_module.insecure_channel(server.address) as channel: + replies = list(channel.unary_stream("/shop.Catalog/List")(b"", timeout=5)) + assert replies == [b'{"id":1}', b"two", b"3"] + + +def test_error_status(grpc_module, server): + server.add_error("/shop.Catalog/Get", "not_found", "no such item") + with pytest.raises(grpc_module.RpcError) as caught: + _unary(grpc_module, server, "/shop.Catalog/Get") + assert caught.value.code() == grpc_module.StatusCode.NOT_FOUND + assert caught.value.details() == "no such item" + + +def test_unknown_method_is_unimplemented(grpc_module, server): + with pytest.raises(grpc_module.RpcError) as caught: + _unary(grpc_module, server, "/nope.Nope/Nope") + assert caught.value.code() == grpc_module.StatusCode.UNIMPLEMENTED + + +def test_unknown_status_code_raises(server): + with pytest.raises(MockServerException): + server.add_error("/a.B/C", "NOT_A_CODE") + + +def test_second_start_raises(server): + with pytest.raises(MockServerException): + server.start() + + +def test_configure_from_json_forms(grpc_module, server): + configure_grpc_method(server, "/a.S/Json", {"response": {"ok": True}}) + configure_grpc_method(server, "/a.S/Raw", {"response_base64": base64.b64encode(b"\x08\x01").decode()}) + configure_grpc_method(server, "/a.S/Err", {"error": {"code": "UNAVAILABLE"}}) + configure_grpc_method(server, "/a.S/List", {"stream": [1, 2]}) + with grpc_module.insecure_channel(server.address) as channel: + assert list(channel.unary_stream("/a.S/List")(b"", timeout=5)) == [b"1", b"2"] + assert _unary(grpc_module, server, "/a.S/Json") == b'{"ok":true}' + assert _unary(grpc_module, server, "/a.S/Raw") == b"\x08\x01" + with pytest.raises(grpc_module.RpcError): + _unary(grpc_module, server, "/a.S/Err") + + +@pytest.mark.parametrize("config", [ + {}, {"response": 1, "stream": []}, {"response": 1, "extra": 2}, {"stream": "x"}, + {"error": "NOT_FOUND"}, {"response_base64": "***"}, +]) +def test_configure_rejects_bad_forms(server, config): + with pytest.raises(MockServerException): + configure_grpc_method(server, "/a.S/M", config) diff --git a/test/test_mock_server/test_mock_config.py b/test/test_mock_server/test_mock_config.py new file mode 100644 index 0000000..5460728 --- /dev/null +++ b/test/test_mock_server/test_mock_config.py @@ -0,0 +1,101 @@ +"""Tests for the declarative mock config and ``apitestka mock --config``.""" +from __future__ import annotations + +import json + +import pytest + +from je_api_testka.cli.cli_main import main +from je_api_testka.utils.exception.exceptions import MockServerException +from je_api_testka.utils.mock_server import protocol_mocks +from je_api_testka.utils.mock_server.flask_mock_server import FlaskMockServer +from je_api_testka.utils.mock_server.mock_config import apply_mock_config, read_mock_config, stop_protocol_mocks + + +@pytest.fixture(autouse=True) +def _stop_mocks(): + yield + stop_protocol_mocks() + + +def _server() -> FlaskMockServer: + return FlaskMockServer("127.0.0.1", 0) + + +def test_http_routes_and_relative_openapi(tmp_path): + spec = {"paths": {"/items": {"get": {"responses": {"200": { + "content": {"application/json": {"example": [1, 2]}}}}}}}} + (tmp_path / "spec.json").write_text(json.dumps(spec), encoding="utf-8") + server = _server() + endpoints = apply_mock_config({"http": { + "routes": [{"rule": "/health", "body": {"ok": True}, "status": 201, "methods": ["GET"]}], + "openapi": "spec.json", + }}, server, base_dir=str(tmp_path)) + client = server.app.test_client() + health = client.get("/health") + assert endpoints == {} + assert (health.status_code, health.get_json()) == (201, {"ok": True}) + assert client.get("/items").get_json() == [1, 2] + + +def test_websocket_and_grpc_sections_start_mocks(): + pytest.importorskip("websockets") + pytest.importorskip("grpc") + endpoints = apply_mock_config({ + "websocket": {"port": 0, "routes": {"/chat": {}}}, + "grpc": {"port": 0, "methods": {"/a.S/M": {"response": "ok"}}}, + }, _server()) + assert endpoints["websocket"].startswith("ws://") + assert endpoints["grpc"].startswith("127.0.0.1:") + stop_protocol_mocks() + with pytest.raises(MockServerException): + protocol_mocks.websocket_mock_received("/chat") + + +def test_failed_grpc_section_stops_the_websocket_it_started(): + pytest.importorskip("websockets") + pytest.importorskip("grpc") + with pytest.raises(MockServerException): + apply_mock_config({ + "websocket": {"port": 0}, + "grpc": {"port": 0, "methods": {"/a.S/M": {"bogus": 1}}}, + }, _server()) + with pytest.raises(MockServerException): + protocol_mocks.websocket_mock_received("/") + + +@pytest.mark.parametrize("config", [ + {"ftp": {}}, + {"http": []}, + {"http": {"routes": {}}}, + {"http": {"routes": [{"body": "no rule"}]}}, + {"http": {"routes": [{"rule": "/x", "colour": "red"}]}}, + {"http": {"openapi": "missing.json"}}, +]) +def test_rejects_malformed_config(config, tmp_path): + with pytest.raises(MockServerException): + apply_mock_config(config, _server(), base_dir=str(tmp_path)) + + +def test_read_mock_config_errors(tmp_path): + (tmp_path / "list.json").write_text("[]", encoding="utf-8") + (tmp_path / "bad.json").write_text("{", encoding="utf-8") + for name in ("list.json", "bad.json", "missing.json"): + with pytest.raises(MockServerException): + read_mock_config(str(tmp_path / name)) + + +def test_cli_mock_config_starts_and_then_stops_protocol_mocks(tmp_path, monkeypatch): + pytest.importorskip("websockets") + config = tmp_path / "mock.json" + config.write_text(json.dumps({"websocket": {"port": 0}}), encoding="utf-8") + seen = {} + + def _fake_run(self): + seen["received"] = protocol_mocks.websocket_mock_received("/") + + monkeypatch.setattr(FlaskMockServer, "start_mock_server", _fake_run) + assert main(["mock", "--port", "0", "--config", str(config)]) == 0 + assert seen == {"received": []} + with pytest.raises(MockServerException): + protocol_mocks.websocket_mock_received("/") diff --git a/test/test_mock_server/test_protocol_mocks.py b/test/test_mock_server/test_protocol_mocks.py new file mode 100644 index 0000000..c95d9da --- /dev/null +++ b/test/test_mock_server/test_protocol_mocks.py @@ -0,0 +1,82 @@ +"""Tests for driving the WebSocket and gRPC mocks from JSON actions.""" +from __future__ import annotations + +import pytest + +from je_api_testka.utils.exception.exceptions import MockServerException +from je_api_testka.utils.executor.action_executor import execute_action +from je_api_testka.utils.mock_server.protocol_mocks import ( + grpc_mock_received, + start_grpc_mock, + start_websocket_mock, + stop_grpc_mock, + stop_websocket_mock, + websocket_mock_received, +) +from je_api_testka.websocket_wrapper.websocket_method import test_api_method_websocket as ws_round_trip + + +@pytest.fixture(autouse=True) +def _stop_mocks(): + yield + stop_websocket_mock() + stop_grpc_mock() + + +def _values(record: dict) -> list: + return list(record.values()) + + +def test_websocket_mock_through_json_actions(): + pytest.importorskip("websockets") + url = _values(execute_action([["AT_mock_start_websocket_server", { + "port": 0, "routes": {"/chat": {"replies": {"ping": "pong"}}}, + }]]))[0] + assert url.startswith("ws://127.0.0.1:") + assert ws_round_trip(f"{url}/chat", messages=["ping"], timeout=5)["messages_received"] == ["pong"] + received = _values(execute_action([ + ["AT_mock_websocket_received", {"path": "/chat"}], + ["AT_mock_stop_websocket_server"], + ])) + assert received[0] == ["ping"] + + +def test_websocket_mock_defaults_to_echo_on_root(): + pytest.importorskip("websockets") + url = start_websocket_mock(port=0) + assert ws_round_trip(f"{url}/", messages=["x"], timeout=5)["messages_received"] == ["x"] + + +def test_grpc_mock_through_json_actions(): + grpc = pytest.importorskip("grpc") + address = _values(execute_action([["AT_mock_start_grpc_server", { + "port": 0, "methods": {"/shop.Catalog/Get": {"response": {"id": 1}}}, + }]]))[0] + with grpc.insecure_channel(address) as channel: + assert channel.unary_unary("/shop.Catalog/Get")(b'{"sku":"a"}', timeout=5) == b'{"id":1}' + assert grpc_mock_received("/shop.Catalog/Get") == ['{"sku":"a"}'] + + +def test_second_start_raises_and_keeps_the_first(): + pytest.importorskip("websockets") + url = start_websocket_mock(port=0) + with pytest.raises(MockServerException): + start_websocket_mock(port=0) + assert ws_round_trip(f"{url}/", messages=["still up"], timeout=5)["messages_received"] == ["still up"] + + +def test_received_without_a_running_mock_raises(): + with pytest.raises(MockServerException): + websocket_mock_received("/chat") + with pytest.raises(MockServerException): + grpc_mock_received("/a.B/C") + + +def test_grpc_mock_needs_methods(): + with pytest.raises(MockServerException): + start_grpc_mock({}) + + +def test_stop_when_nothing_runs_is_a_no_op(): + stop_websocket_mock() + stop_grpc_mock() diff --git a/test/test_mock_server/test_websocket_mock.py b/test/test_mock_server/test_websocket_mock.py new file mode 100644 index 0000000..25d03f3 --- /dev/null +++ b/test/test_mock_server/test_websocket_mock.py @@ -0,0 +1,109 @@ +"""Tests for the WebSocket mock endpoints.""" +from __future__ import annotations + +import builtins + +import pytest + +from je_api_testka.utils.exception.exceptions import MockServerException +from je_api_testka.utils.mock_server.websocket_mock import ( + WebSocketMockServer, + WebSocketRoute, + route_from_config, +) +from je_api_testka.websocket_wrapper.websocket_method import test_api_method_websocket as ws_round_trip + + +def test_default_route_echoes(): + assert WebSocketRoute().answer("hello") == ["hello"] + + +def test_replies_take_precedence_and_expand_message(): + route = WebSocketRoute(replies={"ping": "pong", "hi": ["a", "you said {{message}}"]}) + assert route.answer("ping") == ["pong"] + assert route.answer("hi") == ["a", "you said hi"] + assert route.answer("other") == ["other"] + + +def test_none_fallback_stays_silent(): + assert WebSocketRoute(fallback=None).answer("x") == [] + + +def test_handler_overrides_replies(): + route = WebSocketRoute(replies={"x": "no"}, handler=lambda message: message.upper()) + assert route.answer("x") == ["X"] + + +def test_route_from_config(): + route = route_from_config({"greeting": "hi", "replies": {"a": "b"}, "fallback": None}) + assert (route.greeting, route.replies, route.fallback) == ("hi", {"a": "b"}, None) + + +@pytest.mark.parametrize("config", [ + {"unknown": 1}, {"replies": []}, {"fallback": 3}, {"greeting": ["x"]}, +]) +def test_route_from_config_rejects_bad_values(config): + with pytest.raises(MockServerException): + route_from_config(config) + + +def test_route_path_must_be_absolute(): + with pytest.raises(MockServerException): + WebSocketMockServer(port=0).add_route("chat") + + +def test_missing_websockets_raises(monkeypatch): + real_import = builtins.__import__ + + def _fake_import(name, *args, **kwargs): + if name.startswith("websockets"): + raise ImportError("nope") + return real_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", _fake_import) + with pytest.raises(MockServerException): + WebSocketMockServer(port=0).start() + + +def test_stop_without_start_is_a_no_op(): + WebSocketMockServer(port=0).stop() + + +@pytest.fixture +def chat_server(): + pytest.importorskip("websockets") + server = WebSocketMockServer(port=0) + server.add_route("/chat", WebSocketRoute(greeting="welcome", replies={"ping": "pong"})) + with server: + yield server + + +def test_round_trip_with_greeting_replies_and_echo(chat_server): + from websockets.sync.client import connect + + with connect(f"{chat_server.url}/chat?room=1", open_timeout=5) as connection: + assert connection.recv(timeout=5) == "welcome" + connection.send("ping") + assert connection.recv(timeout=5) == "pong" + connection.send("hello") + assert connection.recv(timeout=5) == "hello" + assert chat_server.received("/chat") == ["ping", "hello"] + + +def test_apitestka_websocket_backend_against_the_mock(chat_server): + record = ws_round_trip(f"{chat_server.url}/chat", messages=["ping"], expected_replies=2, timeout=5) + assert record["messages_received"] == ["welcome", "pong"] + + +def test_unknown_path_is_refused(chat_server): + from websockets.exceptions import InvalidStatus + from websockets.sync.client import connect + + with pytest.raises(InvalidStatus) as caught: + connect(f"{chat_server.url}/missing", open_timeout=5) + assert caught.value.response.status_code == 404 + + +def test_second_start_raises(chat_server): + with pytest.raises(MockServerException): + chat_server.start() diff --git a/uv.lock b/uv.lock index d0d10f7..7a76e19 100644 --- a/uv.lock +++ b/uv.lock @@ -452,6 +452,77 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/7f/9c/34f6962f9b9e9c71f6e5ed806e0d0ff03c9d1b0b2340088a0cf4bce09b18/flask-3.1.3-py3-none-any.whl", hash = "sha256:f4bcbefc124291925f1a26446da31a5178f9483862233b23c0c96a20701f670c", size = 103424, upload-time = "2026-02-19T05:00:56.027Z" }, ] +[[package]] +name = "grpcio" +version = "1.84.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/3f/4f/4435c0aae54657258d9cfcba78598f3d9e5fe4c82ff18d78558567b90faf/grpcio-1.84.0.tar.gz", hash = "sha256:19aaf172fc2edbefccce3f6e92c5150975dbe56c45744e9e87cf72ebdf85bfbe", size = 13493876, upload-time = "2026-09-14T06:59:33.291Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3a/4b/a0dc421d049b743093eae90caeb5dd92ced7226cd4919dc4de34c81455b6/grpcio-1.84.0-cp310-cp310-linux_armv7l.whl", hash = "sha256:71fd60e6e426d293d0a2f685115ad0a0845117602cf13605a4be7524fb5f7bba", size = 6450049, upload-time = "2026-09-14T06:56:48.72Z" }, + { url = "https://files.pythonhosted.org/packages/f7/41/90292bf55af7aa09de0e3ec928d1b8c56d477f85244f7928d2231630b781/grpcio-1.84.0-cp310-cp310-macosx_11_0_universal2.whl", hash = "sha256:8e1a45d174b6b8589f51dce1cea804aa6c1f72c9c80cba91ae2caabeb6d90540", size = 12344932, upload-time = "2026-09-14T06:56:51.685Z" }, + { url = "https://files.pythonhosted.org/packages/e9/68/b6c0248266a378b1bde08e4de7d69f3cc08ee6f5937a5dce7d3c3ba0fe1d/grpcio-1.84.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:efb29f8633bf6630dc89de4fe0353ac3d7e4b70ef7b6e29fb40f00e68c127fa5", size = 7030162, upload-time = "2026-09-14T06:56:54.412Z" }, + { url = "https://files.pythonhosted.org/packages/70/2b/0a2a2cbcf48847f83eb51fb982116d0965f2fe068e73f28b2d17facf30a4/grpcio-1.84.0-cp310-cp310-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:d0fdd25faece8a1f95e8a3a8006e29701b5cf8dadb4a8132e68f3134637004a5", size = 7781546, upload-time = "2026-09-14T06:56:56.571Z" }, + { url = "https://files.pythonhosted.org/packages/a6/7c/da97476f3c2e90e9f00bfb19def7cbb5f841b7661e3cd09c6a89beaa5b98/grpcio-1.84.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:393d8a78bff6731ecc5ad2151a821f8fbc1709b137ebb9c25a4ef399fbdcc914", size = 7186279, upload-time = "2026-09-14T06:56:58.817Z" }, + { url = "https://files.pythonhosted.org/packages/14/16/27fa3aed1ee6fdcbb978a1bd4bce255dc0122b90179535177a96543d14fc/grpcio-1.84.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:fc66cb50c93554b86db0b6625ab5c6e9051dbf8847c08d93c84918e02e413fb7", size = 7731191, upload-time = "2026-09-14T06:57:02.447Z" }, + { url = "https://files.pythonhosted.org/packages/4c/78/75644af37af85afb381376aef99cad92da8bc2d56ba3e5ae070a7cb59682/grpcio-1.84.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:455ed6083353b8e938f1d58c765eab2fbb165731e5b507be30fee344915a2a11", size = 8790443, upload-time = "2026-09-14T06:57:04.623Z" }, + { url = "https://files.pythonhosted.org/packages/95/4d/ce57fa986e93c06ef867f64e1ebe419e924fdc2395115607f0725f4855e4/grpcio-1.84.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:3d6a82c4fc6c85f2fb7572c86bdb86f84c97b6580e5f6599f711800bac48a5d8", size = 8138068, upload-time = "2026-09-14T06:57:07.348Z" }, + { url = "https://files.pythonhosted.org/packages/ba/a6/22a73111c4f75da9450bf0481fac805396ec9bf6f949a90bb2969de07cf4/grpcio-1.84.0-cp310-cp310-win32.whl", hash = "sha256:8e3f508d0e9e6236ba2f08d56e33355e434e785e813149a1b8477d3edf69779d", size = 4496545, upload-time = "2026-09-14T06:57:09.236Z" }, + { url = "https://files.pythonhosted.org/packages/31/ff/dc048bc3d8ebd8d4b7f6f6803c76142a9a5ca1e1e9fa34e79597f0f9ed77/grpcio-1.84.0-cp310-cp310-win_amd64.whl", hash = "sha256:ed2c1493c44d0932f1e55fdb5d1ead658c68288ec5d51b8c4928422d98633ef9", size = 5258144, upload-time = "2026-09-14T06:57:11.403Z" }, + { url = "https://files.pythonhosted.org/packages/2d/b9/46146728b3f4a5c7e34c17d0ab724d58b5456b116e76dc77d3ef4e79b135/grpcio-1.84.0-cp311-cp311-linux_armv7l.whl", hash = "sha256:4aaeceeb7fa7d824c322d1ec3208c8495c88478a927295553235435fc49043ad", size = 6454572, upload-time = "2026-09-14T06:57:14.651Z" }, + { url = "https://files.pythonhosted.org/packages/e3/63/5d668b4102637410d700153fd12d6a798e3ff8308bd9dcbaeae93f191060/grpcio-1.84.0-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:06619ba1515e5ee69fb2a514e95dd8be05ce74cb3928d5b34f87f87c86fe3c27", size = 12359529, upload-time = "2026-09-14T06:57:17.202Z" }, + { url = "https://files.pythonhosted.org/packages/18/2a/52e29c02047a493f15a78c0502bde4d3fab7c19c7813944d367cd501811c/grpcio-1.84.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:158c1c11cfb61b4849c3caf4d52de6f5ecd376e14446feb4a90dc95a90d616f5", size = 7029927, upload-time = "2026-09-14T06:57:19.767Z" }, + { url = "https://files.pythonhosted.org/packages/0a/11/9962b313553647abb091943e0721e4a1662ecc63cdfe930abf00abcce47a/grpcio-1.84.0-cp311-cp311-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:a9383401d9f116f98cacd4eba6c505a6edb80ba65badfc8e8ed8ae64983bcc44", size = 7782268, upload-time = "2026-09-14T06:57:22.381Z" }, + { url = "https://files.pythonhosted.org/packages/e2/b7/14a9413cb7d4b2e782b4f79c81a918610caedf55138ab5916f5fdd4b002f/grpcio-1.84.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:bd8ea8eb3817b226057cc1c0e7ec4b378dcda52043b972b6ff12b1152178967d", size = 7187959, upload-time = "2026-09-14T06:57:24.686Z" }, + { url = "https://files.pythonhosted.org/packages/ee/3b/6cc8e6aed8f23be40f52af341e5d4595ec3ec8d7572271a692b5c1212178/grpcio-1.84.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:756ea5c2da00fa65c930284892d2a9706828704ca3ba40b4c51c4834eb39fcfd", size = 7737554, upload-time = "2026-09-14T06:57:27.5Z" }, + { url = "https://files.pythonhosted.org/packages/3c/7e/6f61002a01802ca9675e1b3599c9b0f9f3cf168ded94ebacc02199309f88/grpcio-1.84.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:28d2609691da93051e998495108bbddd2a9f7a561253bae94828d81290f30c15", size = 8792681, upload-time = "2026-09-14T06:57:29.731Z" }, + { url = "https://files.pythonhosted.org/packages/eb/84/8bec1ae7e6732a9b435a394ddfdfffde46c2620ae0109823f7cce1a54455/grpcio-1.84.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:27b8b36200a9fbee6e120246f4a8a41657549107ef19fb2c819c4b2fd524f39a", size = 8145493, upload-time = "2026-09-14T06:57:32.672Z" }, + { url = "https://files.pythonhosted.org/packages/59/84/c8c7bd210d657288f18af06522f150f61e81ea14fd3c7c135beed697c5fd/grpcio-1.84.0-cp311-cp311-win32.whl", hash = "sha256:465eef3d17e59ad22a556fc0138f7c7c799df426734344daec42c797d49fda99", size = 4495596, upload-time = "2026-09-14T06:57:34.799Z" }, + { url = "https://files.pythonhosted.org/packages/da/1e/da99356b3b573af357d059753a47fba54f1ca1a9c0e4deccd0210cb7f4ba/grpcio-1.84.0-cp311-cp311-win_amd64.whl", hash = "sha256:f9a456bdbed52a01c9ab8423bdebab04a5363c78676edc55ab9b58bd13bdf9e1", size = 5259900, upload-time = "2026-09-14T06:57:37.067Z" }, + { url = "https://files.pythonhosted.org/packages/0a/c1/4c9a2e0e6b0aaf02781404cad2f79211f989f2c827cf672a4a48d1604d3e/grpcio-1.84.0-cp312-cp312-linux_armv7l.whl", hash = "sha256:b5c6f20d657ae09ae4e30d9d3a21edd13f1219d58cc6f999b9d1bb63be9c1baa", size = 6415756, upload-time = "2026-09-14T06:57:39.345Z" }, + { url = "https://files.pythonhosted.org/packages/b1/57/131e7007bdee9acb77a8dbe8a16fa9fef75f88c1695242d8ee0993ac2d3d/grpcio-1.84.0-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:406583b4e8fb2282ebd392e12b963e601c1f82e07125a8c2cb5b144e7e024796", size = 12339195, upload-time = "2026-09-14T06:57:42.373Z" }, + { url = "https://files.pythonhosted.org/packages/db/d1/a7b7cda98fcab9b3d2916204a872d87371158a7a34e41768f524584fb64d/grpcio-1.84.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:fbdbcd06986ede3ce584083b1dc2afe6808e8943e5cf50ad11183c03aceda25a", size = 6984468, upload-time = "2026-09-14T06:57:45.035Z" }, + { url = "https://files.pythonhosted.org/packages/19/81/c5be83e3ac9416f73c4c51fe1ea9c41a0c42fc3509e3505faa46f5046abe/grpcio-1.84.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:23e6e8e8a75cff88e0a793bfd3becea03a13e2763ae90c1ff573bc19ca5b429a", size = 7749432, upload-time = "2026-09-14T06:57:47.395Z" }, + { url = "https://files.pythonhosted.org/packages/a0/bf/258cd7c0a7ed92745dc93c31666d462d05b702807a689744bd49fb833bde/grpcio-1.84.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:b44f0a0fc7bc6677d38cc80bca1a32814ce6c8f200fb8b3c1a61c9d77eaefbf3", size = 7156115, upload-time = "2026-09-14T06:57:49.657Z" }, + { url = "https://files.pythonhosted.org/packages/2b/4b/7f829418dbfcf91b875e55e2973f1059a95decb4f081313416317ef04ec1/grpcio-1.84.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:210e4c32f907045eb8158273e60c6ab69a3947697df6245dbda381f26c59485b", size = 7708010, upload-time = "2026-09-14T06:57:52.496Z" }, + { url = "https://files.pythonhosted.org/packages/34/f0/9932e2fec6a04205f8bf3f8f4d2020479dcdac88feb6f93822ed31bf0eba/grpcio-1.84.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:a71d24f40b0cc6798feaa978c7411dc1135b7018e9fc0442db611c139bf58344", size = 8759980, upload-time = "2026-09-14T06:57:55.312Z" }, + { url = "https://files.pythonhosted.org/packages/2c/5c/b67407c6dbc480dfc0715f6eccdb1061e7c88d85f9a330a241d357a538c5/grpcio-1.84.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:f6c972474ce691aca74e58d17625450cef153dc4760364cadeb167983ea6d589", size = 8124904, upload-time = "2026-09-14T06:57:58.569Z" }, + { url = "https://files.pythonhosted.org/packages/02/37/2bfdae2df8dfcfc0df619b628e0c7153ce703adae827243f44720322ccc1/grpcio-1.84.0-cp312-cp312-win32.whl", hash = "sha256:0d532ade4486dad9b302ffa4d4683d67561051c26d17c4023322845e9fa10140", size = 4478915, upload-time = "2026-09-14T06:58:00.714Z" }, + { url = "https://files.pythonhosted.org/packages/85/2c/309268b7b39f6deb2342f634841e105623a0b67982e8b10ec516782ff1c6/grpcio-1.84.0-cp312-cp312-win_amd64.whl", hash = "sha256:49717e857899f4136d7657bf5aded61ac479110a075438290923a4d86af7cd02", size = 5253534, upload-time = "2026-09-14T06:58:03.336Z" }, + { url = "https://files.pythonhosted.org/packages/5d/51/40f99701adb01d4e5316a2aaf13838da1a24d5c879cd8c95156d7c364454/grpcio-1.84.0-cp313-cp313-linux_armv7l.whl", hash = "sha256:209414080da8c20af94df1395b635da52dd57b5edc9e917e1deca0dc1c4bb55e", size = 6427619, upload-time = "2026-09-14T06:58:06.025Z" }, + { url = "https://files.pythonhosted.org/packages/c5/4b/ed8e22a1237e6b2be6ef4f221d074a5b0e0dd8a0da8c944c04aea731f0eb/grpcio-1.84.0-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:e41c3993eee896c617dbd8a505085d28b6e84a0445ed9a1f40f95808473cf678", size = 12336549, upload-time = "2026-09-14T06:58:08.583Z" }, + { url = "https://files.pythonhosted.org/packages/d3/50/00165b05cd73f45996748ea67ce9e55d08936f2fea94a7fd8541cc2d0e54/grpcio-1.84.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:fff5ef3fe1bba7d6147e5f19e01e5e122ac2c076486887ddcb8d42e663400fbe", size = 6989458, upload-time = "2026-09-14T06:58:11.884Z" }, + { url = "https://files.pythonhosted.org/packages/26/38/d0486230e684d916f97429a53041db88410e662a38f2a8d09e2d90375840/grpcio-1.84.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:b8c62888c3e49debf37ad9773e3c02f77b0c1e811f8fb0962f2b6c3bbab5b97a", size = 7757778, upload-time = "2026-09-14T06:58:14.849Z" }, + { url = "https://files.pythonhosted.org/packages/da/56/548a643decb059ca244499c675ae2c13a15f523ba94592c2774bd80a13c1/grpcio-1.84.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:986e9751d416d7a6eaa2fecdac38da63153d63a4b340ba7d624889c490451500", size = 7159572, upload-time = "2026-09-14T06:58:17.87Z" }, + { url = "https://files.pythonhosted.org/packages/db/f5/42caac81a79ec680f1f7a8eaf7ca90d2f93936ce0c3a073141ba96757f77/grpcio-1.84.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:5933a052946873d01a42119a05420d669bdca436aeba2d1851988ccb12b421c0", size = 7710547, upload-time = "2026-09-14T06:58:20.607Z" }, + { url = "https://files.pythonhosted.org/packages/57/a4/828ad990b2410fee0a55cc73aa1bf98eb5b911c54847374ef4f24b9e877b/grpcio-1.84.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:e094dd21f077af8194923fc263cad872eaa1802bb0156fd7e5ae18e99cd86715", size = 8761519, upload-time = "2026-09-14T06:58:23.875Z" }, + { url = "https://files.pythonhosted.org/packages/d5/a5/1f91af098919eaf5d80d5a61126ad9fae074e5190c25a3014ce1d8d0d890/grpcio-1.84.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:08735e3d08d24ab3132cf87e2e5dea8746cabcc7d676c2b0b7362f195feef9d9", size = 8121424, upload-time = "2026-09-14T06:58:27.006Z" }, + { url = "https://files.pythonhosted.org/packages/8c/8f/77fd4a7a913b636785479922349c4cb98d94d05d15652e556b3ca0df6663/grpcio-1.84.0-cp313-cp313-win32.whl", hash = "sha256:70bb4ce8be0c5606bec259cbd7152374470396413b7863a658a08c849e6b29ff", size = 4477974, upload-time = "2026-09-14T06:58:29.528Z" }, + { url = "https://files.pythonhosted.org/packages/d0/9a/1fa59ddbfc8898e5518d1447e46f771f387f0ed6132ad531395338e51a5c/grpcio-1.84.0-cp313-cp313-win_amd64.whl", hash = "sha256:b61692f0069b3eee2fc8a3a1b7f6c044df9e03fede6ce69b3ca832e1c39f26c5", size = 5255326, upload-time = "2026-09-14T06:58:31.781Z" }, + { url = "https://files.pythonhosted.org/packages/26/6f/e25ca89ca5b0b7b95464c907a5c21a77c0ac8c4ee1dca164c4dd8f153ddb/grpcio-1.84.0-cp314-cp314-linux_armv7l.whl", hash = "sha256:026d757df86c5b7a41de8200b9a2cda454aaa5004cb0c7e3374c66eb82f61499", size = 6428207, upload-time = "2026-09-14T06:58:34.401Z" }, + { url = "https://files.pythonhosted.org/packages/cd/b4/6b76b429f3f9b901cdbc306c81364d708bc957f847a05cbd1046cd2d05d8/grpcio-1.84.0-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:3de427b05f244ba2c2a9bdc67e7a6731c8340811524ecc4435466549f8af1d17", size = 12342420, upload-time = "2026-09-14T06:58:37.416Z" }, + { url = "https://files.pythonhosted.org/packages/af/64/ac86d638ba7f73bee0dccb608ba551d4f63adf75151f00d2c43e46d3979e/grpcio-1.84.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:e90e3bdf7b5eac005fef631adae9cafde16f922def207b80a7c46b253c18ad20", size = 6998396, upload-time = "2026-09-14T06:58:40.535Z" }, + { url = "https://files.pythonhosted.org/packages/4a/65/fa12e9ec9d7ebf8cc3e81428fa9e1ca0d30d22d546ce2baa4c64bc917cbc/grpcio-1.84.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:e88d304f094f4937bc27ec6a435e218a084168f11ec630c8d5d39b431d08d81d", size = 7757538, upload-time = "2026-09-14T06:58:43.297Z" }, + { url = "https://files.pythonhosted.org/packages/21/d7/94240c7fae121ff1f116dcf04a3b7ee0216a06832c704310363f72638d4c/grpcio-1.84.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:57dc36a5ab0e676f5f6e171de2917fd0aef73f32a9aaf23956bfe19997a30bd1", size = 7161480, upload-time = "2026-09-14T06:58:45.939Z" }, + { url = "https://files.pythonhosted.org/packages/23/c9/7033e95d4b344969818b09185721c7608b47fc2498d97b5e4eec4995dbf3/grpcio-1.84.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:5deda5b4bf62769eb98c119cca43d40e1231e34846b19db5cdea821d446a2253", size = 7720191, upload-time = "2026-09-14T06:58:48.308Z" }, + { url = "https://files.pythonhosted.org/packages/95/22/b45df2deba81d55069076859480bae7109c9eec02bce5515c799530cc2aa/grpcio-1.84.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:9bab4cf571653a8afffb83ce21aa27b51dfe629b526b7b6adec35491fe1fc2ea", size = 8762792, upload-time = "2026-09-14T06:58:51.068Z" }, + { url = "https://files.pythonhosted.org/packages/de/c4/3e1c3d6155c16b8737cc31d5b477d6cf1fc7cdd10d58320cf0ec9b446f42/grpcio-1.84.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:c5559b492007dc09b4de9b95dab05f0b5e53547aad230cf07e46c7dd017a3be5", size = 8123299, upload-time = "2026-09-14T06:58:54.332Z" }, + { url = "https://files.pythonhosted.org/packages/56/fe/f4864de5b815e5ba18858771f99381a398fac14117f89ef5291ed43d3c4e/grpcio-1.84.0-cp314-cp314-win32.whl", hash = "sha256:2c024da73b296f040b8360e60bd73a659b230093684a438da0e1260f34cc724e", size = 4562560, upload-time = "2026-09-14T06:58:56.894Z" }, + { url = "https://files.pythonhosted.org/packages/44/03/640811d4d8c84f5e603995c5a9bab725223aa472cad9ca4286c3bbf1c3e3/grpcio-1.84.0-cp314-cp314-win_amd64.whl", hash = "sha256:800b7e00d92553313c0463c200087930aa78678ec1d528193aeb50906f55989b", size = 5394092, upload-time = "2026-09-14T06:58:59.61Z" }, + { url = "https://files.pythonhosted.org/packages/4a/1a/9e3d2c9f005f680f03308fa894b1db91d4ab3f0fe65ff630c69561e91e95/grpcio-1.84.0-cp315-cp315-linux_armv7l.whl", hash = "sha256:47ecf0d9b81d981f07b61bd89eced9d2582f5eaacc3aaa36ad27f81aef70a27f", size = 6428252, upload-time = "2026-09-14T06:59:02.597Z" }, + { url = "https://files.pythonhosted.org/packages/77/34/0bc9f52ebf091311651eeab3a452fb557985604a3088cb5406f4d6df85d3/grpcio-1.84.0-cp315-cp315-macosx_10_15_universal2.whl", hash = "sha256:61386101ecaa096b694d0dd278caf99a56aeec78440cc17e918eef0b50f2d567", size = 12359488, upload-time = "2026-09-14T06:59:05.646Z" }, + { url = "https://files.pythonhosted.org/packages/93/0e/c31052712f241cb6ecae9c226fabd519b7f8c64a7a40bac27e9ca0405b78/grpcio-1.84.0-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f6d178ba6dc8e82976c184b65fddde172d054c17237993a3e083efe4f134d55b", size = 7019339, upload-time = "2026-09-14T06:59:08.76Z" }, + { url = "https://files.pythonhosted.org/packages/55/b9/b9b33ea4f1eb4cad28833cade604febf357385b5ebb0c9c7562d020e167a/grpcio-1.84.0-cp315-cp315-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:15bb76489e337fc492685c9758e2fd4d4ab516b901ad830dc5a91987decf00be", size = 7107974, upload-time = "2026-09-14T06:59:11.568Z" }, + { url = "https://files.pythonhosted.org/packages/0e/9e/799d4c45db91bbdcd8c54b3982932dbcf3d059f7ce67dca3e8540faa1ece/grpcio-1.84.0-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:82da34ae4f639c73ac46e521e00c0a49bf86f717b9fb1f405f133e98731e38dc", size = 7200036, upload-time = "2026-09-14T06:59:14.401Z" }, + { url = "https://files.pythonhosted.org/packages/45/dc/dcfdd13ada41aff9098f0c2c6f260eb7debbc88b84b7e5fcbd085165427d/grpcio-1.84.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:9b73836ba0e16fcbb57c31cf6cbc2907c8d8c790b83679df454b74bd15e0be04", size = 7742281, upload-time = "2026-09-14T06:59:17.348Z" }, + { url = "https://files.pythonhosted.org/packages/55/31/75eab2ec77b80804bc5e21cec99b57598e726fca6484cd3e8920a97639d5/grpcio-1.84.0-cp315-cp315-musllinux_1_2_i686.whl", hash = "sha256:42959bd50dd660ffc3f2a9bec15a6da4f9aaa0dda555d59ff2d2e80b908456a8", size = 8113629, upload-time = "2026-09-14T06:59:20.584Z" }, + { url = "https://files.pythonhosted.org/packages/34/f0/fdcf6bdc1df9ca11679a1187bef8e6b81df31a2baae69497e17344f05ea3/grpcio-1.84.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:659728f20fc7a0933ed7b1945435e31014b97ab8a5a7edcbaa70da4794aeb191", size = 8152972, upload-time = "2026-09-14T06:59:24.523Z" }, + { url = "https://files.pythonhosted.org/packages/5c/cf/6720e720bfa80fcb1ace873f66724eb3c8b03bba2fa078a30c12cab3212e/grpcio-1.84.0-cp315-cp315-win32.whl", hash = "sha256:edb6f87fc60ff438557291501b3e16c7a77c3b01a52d782cf276dccc7c5dd89c", size = 4561981, upload-time = "2026-09-14T06:59:27.275Z" }, + { url = "https://files.pythonhosted.org/packages/7f/b9/69d8a709df225bc2e06e028e9465166b174c24b3da07cc72d9a5ddc63194/grpcio-1.84.0-cp315-cp315-win_amd64.whl", hash = "sha256:4119efa6519871719ad81f33bc95ab87857dcb1c5801f30a6e592f2c41164169", size = 5394757, upload-time = "2026-09-14T06:59:30.118Z" }, +] + [[package]] name = "h11" version = "0.16.0" @@ -558,6 +629,9 @@ dependencies = [ ] [package.optional-dependencies] +grpc = [ + { name = "grpcio" }, +] gui = [ { name = "pyside6" }, { name = "qt-material" }, @@ -587,6 +661,7 @@ requires-dist = [ { name = "botocore", marker = "extra == 'security'", specifier = ">=1.34" }, { name = "defusedxml" }, { name = "flask" }, + { name = "grpcio", marker = "extra == 'grpc'", specifier = ">=1.60" }, { name = "httpx" }, { name = "jsonpath-ng", marker = "extra == 'schema'", specifier = ">=1.6" }, { name = "jsonschema", marker = "extra == 'schema'", specifier = ">=4.0" }, @@ -599,7 +674,7 @@ requires-dist = [ { name = "requests" }, { name = "websockets", marker = "extra == 'websocket'", specifier = ">=12.0" }, ] -provides-extras = ["gui", "websocket", "schema", "security", "otel", "mcp"] +provides-extras = ["gui", "websocket", "grpc", "schema", "security", "otel", "mcp"] [[package]] name = "jinja2" @@ -1516,8 +1591,8 @@ name = "uvicorn" version = "0.53.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "click" }, - { name = "h11" }, + { name = "click", marker = "python_full_version < '3.12' or python_full_version >= '3.14' or sys_platform != 'emscripten'" }, + { name = "h11", marker = "python_full_version < '3.12' or python_full_version >= '3.14' or sys_platform != 'emscripten'" }, { name = "typing-extensions", marker = "python_full_version < '3.11'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/5d/ad/04bbb797c84fc1f26cb171f7394716f4865ffb8d8c5e1eef42565c2dfa6b/uvicorn-0.53.0.tar.gz", hash = "sha256:a9356f0cb89b3b8621529c5d5eebd69bfe154f4c3f68b4cf2de47e45fa855c2e", size = 110881, upload-time = "2026-09-14T07:44:23.815Z" } From cac9ee4d1ce68e52345a988d0ff1f68ca68757d3 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 01:10:16 +0800 Subject: [PATCH 11/43] feat: add Pact-style bidirectional contract testing New je_api_testka.contract package: consumer contracts in the Pact v2 file format recorded from test runs (type-matched bodies by default), Pact matching rules, provider verification against a running provider with provider states, and a check of the contract against the provider's OpenAPI document without running it. Reachable as `apitestka contract record|verify|compare` and as AT_write_contract / AT_verify_contract / AT_check_contract_against_openapi. --- CLAUDE.md | 3 +- README.md | 45 ++++- README/README_zh-CN.md | 42 +++- README/README_zh-TW.md | 42 +++- architecture.md | 6 +- docs/source/Eng/Eng_index.rst | 6 +- docs/source/Eng/doc/cli/cli_doc.rst | 2 + docs/source/Eng/doc/contract/contract_doc.rst | 89 +++++++++ docs/source/Zh/Zh_index.rst | 5 +- docs/source/Zh/doc/cli/cli_doc.rst | 2 + docs/source/Zh/doc/contract/contract_doc.rst | 82 ++++++++ docs/updates/2026-10.md | 13 ++ docs/updates/README.md | 3 +- je_api_testka/cli/cli_common.py | 25 +++ je_api_testka/cli/cli_main.py | 22 +-- je_api_testka/cli/completion.py | 2 +- je_api_testka/cli/contract_cli.py | 106 +++++++++++ je_api_testka/contract/__init__.py | 32 ++++ je_api_testka/contract/commands.py | 64 +++++++ je_api_testka/contract/matching.py | 155 +++++++++++++++ je_api_testka/contract/openapi_compat.py | 152 +++++++++++++++ je_api_testka/contract/openapi_schema.py | 80 ++++++++ je_api_testka/contract/pact.py | 179 ++++++++++++++++++ je_api_testka/contract/verifier.py | 162 ++++++++++++++++ je_api_testka/spec/openapi_export.py | 13 +- je_api_testka/spec/path_templates.py | 47 +++++ je_api_testka/spec/records_to_openapi.py | 10 +- je_api_testka/utils/exception/exceptions.py | 4 + .../utils/executor/action_executor.py | 5 + progress.md | 1 - test/test_contract/__init__.py | 0 test/test_contract/conftest.py | 51 +++++ test/test_contract/test_commands.py | 123 ++++++++++++ test/test_contract/test_matching.py | 72 +++++++ test/test_contract/test_openapi_compat.py | 116 ++++++++++++ test/test_contract/test_pact.py | 98 ++++++++++ test/test_contract/test_verifier.py | 70 +++++++ test/test_executor_integration.py | 2 + 38 files changed, 1897 insertions(+), 34 deletions(-) create mode 100644 docs/source/Eng/doc/contract/contract_doc.rst create mode 100644 docs/source/Zh/doc/contract/contract_doc.rst create mode 100644 je_api_testka/cli/cli_common.py create mode 100644 je_api_testka/cli/contract_cli.py create mode 100644 je_api_testka/contract/__init__.py create mode 100644 je_api_testka/contract/commands.py create mode 100644 je_api_testka/contract/matching.py create mode 100644 je_api_testka/contract/openapi_compat.py create mode 100644 je_api_testka/contract/openapi_schema.py create mode 100644 je_api_testka/contract/pact.py create mode 100644 je_api_testka/contract/verifier.py create mode 100644 je_api_testka/spec/path_templates.py create mode 100644 test/test_contract/__init__.py create mode 100644 test/test_contract/conftest.py create mode 100644 test/test_contract/test_commands.py create mode 100644 test/test_contract/test_matching.py create mode 100644 test/test_contract/test_openapi_compat.py create mode 100644 test/test_contract/test_pact.py create mode 100644 test/test_contract/test_verifier.py diff --git a/CLAUDE.md b/CLAUDE.md index d092b44..46b9bf0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -40,7 +40,7 @@ je_api_testka/ │ ├── callback/ # Observer pattern - post-request callbacks │ ├── executor/ # Command pattern - JSON keyword-driven actions (AT_* commands) │ ├── generate_report/ # Template Method - HTML/JSON/XML/JUnit/Allure/Markdown reports -│ ├── mock_server/ # Flask-based mock server +│ ├── mock_server/ # Flask mock server plus WebSocket and gRPC mocks │ ├── socket_server/ # TCP remote automation server │ ├── project/ # Factory pattern - project scaffolding │ ├── json/, xml/ # JSON and XML I/O utilities @@ -54,6 +54,7 @@ je_api_testka/ ├── data/ # Variable store, templates, env profiles, fake data ├── connection/ # Connection options, DNS override, record/replay cassettes ├── diff/, spec/ # Response and contract diff, SLA checks; schema inference, OpenAPI export +├── contract/ # Pact-style consumer contracts, provider verification, OpenAPI check ├── security/ # Auth helpers, header scan, fuzzing, pip-audit wrapper ├── runner/ # Parallel runner, tag filter, dependency ordering ├── integrations/ # cURL and HAR import, webhook notify, GitHub PR comment diff --git a/README.md b/README.md index 4001f64..0195ae1 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,7 @@ server for Claude, and more. - [CLI / Developer Experience](#cli--developer-experience) - [Security Probes](#security-probes) - [OpenAPI Inference](#openapi-inference) + - [Contract Testing](#contract-testing) - [GUI](#gui) - [Pluggable AI Backend](#pluggable-ai-backend) - [MCP Server for Claude](#mcp-server-for-claude) @@ -54,13 +55,14 @@ server for Claude, and more. | **Data layer** | Variable store, `{{var}}` templating, CSV/JSON data-driven loops, env profiles, fake data | | **Assertions** | Field assertions, JSON Schema, JSONPath, snapshot, structural diffs, OpenAPI contract drift, response-time SLAs | | **Connection** | mTLS, proxies, DNS override, VCR-style cassette record/replay | -| **Mock server** | Static, dynamic, stateful, fault injection, OpenAPI-driven, Jinja templating, webhook receiver, record-replay proxy | +| **Mock server** | Static, dynamic, stateful, fault injection, OpenAPI-driven, Jinja templating, webhook receiver, record-replay proxy, WebSocket and gRPC endpoints | | **Runner** | Sequential & parallel execution, tag filters, dependency-aware ordering, retry policies | | **Reports** | HTML / JSON / XML / **JUnit / Allure / Markdown** / shields.io badge / SQLite trend store / run diff | | **Integrations** | Slack / Teams / Discord webhook, GitHub PR comment, cURL & HAR importers, OpenAPI / Postman importer | | **CLI / DX** | Subcommand CLI, REPL, terminal summary, shell completion, scaffold | | **Security** | Auth helpers (Basic / Bearer / JWT / AWS SigV4), header / CORS / rate-limit / SSRF probes, pip-audit wrapper, fuzz inputs | | **Spec inference** | Test record → OpenAPI, JSON Schema inference, OpenAPI changelog | +| **Contract testing** | Pact v2 consumer contracts from test runs, provider verification, bidirectional check against OpenAPI | | **AI** | Pluggable backend with deterministic fallback for test generation, fake data, failure classification | | **MCP** | First-class Claude Code / MCP server exposing the framework as tools | | **GUI** | Optional PySide6 GUI (English / 繁中 / 简中 / 日本語) plus Swagger UI embed | @@ -353,6 +355,7 @@ apitestka completion bash # source >> ~/.bashrc apitestka mcp # MCP server over stdio apitestka openapi --report run_success.json -o openapi.json # infer a spec from a saved run apitestka openapi --run actions.json # run first, print the inferred spec +apitestka contract verify pacts/web-shop.json --base-url http://localhost:8000 # see Contract Testing ``` ### Security Probes @@ -390,6 +393,45 @@ bodies become response and request-body schemas. A saved report is the `_success.json` file from `generate_json_report`; the `AT_export_openapi` command and `apitestka openapi` read it too. +### Contract Testing + +Pact-style, bidirectional consumer contracts (`je_api_testka.contract`). The consumer's test run +becomes a contract in the Pact v2 file format, so it can go to a Pact Broker or Pactflow as it is. Every +response body gets a `type` matching rule by default: the provider must return the same structure +and value types, not the same values. The contract is then checked from two sides: + +- **Provider verification** replays each interaction against a running provider: equal status, + the contract's headers present (`Content-Type` by media type), and the body satisfying the + matching rules (`type`, `regex`, `min`; objects may carry extra keys). Provider states + (`providerState`) are set up through a Python callable or a setup URL that receives + `{"consumer": ..., "state": ...}`. +- **Bidirectional check** compares the contract with the provider's OpenAPI document without + running the provider: the operation exists (path templates match), query parameters are declared + and required ones sent, the request body fits its schema, the status is declared (exact, `2XX` or + `default`), and the response body fits its schema (`$ref`, `allOf` / `anyOf` / `oneOf`, `nullable`). + The provider keeps its OpenAPI document honest from its own test run, e.g. with `apitestka openapi`. + +```python +from je_api_testka.contract import write_contract, verify_contract, check_contract_against_openapi + +# consumer: its test run (current record plus saved reports) becomes the contract +write_contract("pacts/web-shop.json", consumer="web", provider="shop", report_paths=["run_success.json"]) +# provider: replay the contract against a running provider +verify_contract("pacts/web-shop.json", base_url="http://localhost:8000") +# bidirectional: check the contract against the provider's OpenAPI document, no provider needed +check_contract_against_openapi("pacts/web-shop.json", "openapi.json") +``` + +```bash +apitestka contract record --report run_success.json --consumer web --provider shop -o pacts/web-shop.json +apitestka contract verify pacts/web-shop.json --base-url http://localhost:8000 +apitestka contract compare pacts/web-shop.json openapi.json +``` + +`verify` and `compare` print a report (`--json` for JSON) and exit with 1 when the contract does not +hold. JSON actions: `AT_write_contract`, `AT_verify_contract`, `AT_check_contract_against_openapi` +(the last two fail the action with the report). + ### GUI ```bash @@ -470,6 +512,7 @@ je_api_testka/ ├── ai/ # Pluggable AI backend + helpers ├── cli/ # apitestka CLI subcommands + REPL + completions ├── connection/ # ConnectionOptions, DNS override, Cassette +├── contract/ # Pact-style consumer contracts, provider verification, OpenAPI check ├── data/ # VariableStore, templates, faker, env profiles ├── diff/ # Response diff / contract drift / SLA ├── graphql_wrapper/ # GraphQL helper diff --git a/README/README_zh-CN.md b/README/README_zh-CN.md index 576761a..0843082 100644 --- a/README/README_zh-CN.md +++ b/README/README_zh-CN.md @@ -33,6 +33,7 @@ record-replay 代理、安全检测、并行执行 runner,以及给 Claude 用 - [CLI / 开发体验](#cli--开发体验) - [安全检测](#安全检测) - [OpenAPI 反推](#openapi-反推) + - [契约测试](#契约测试) - [GUI](#gui) - [可插拔 AI 后端](#可插拔-ai-后端) - [Claude 的 MCP Server](#claude-的-mcp-server) @@ -53,13 +54,14 @@ record-replay 代理、安全检测、并行执行 runner,以及给 Claude 用 | **数据层** | 变量存储、`{{var}}` 模板、CSV/JSON 数据驱动、环境配置、假数据 | | **断言** | 字段断言、JSON Schema、JSONPath、Snapshot、结构化 diff、OpenAPI contract drift、响应时间 SLA | | **连接** | mTLS、代理、DNS override、VCR-style cassette 录制/回放 | -| **模拟服务器** | 静态、动态、stateful、故障注入、OpenAPI 驱动、Jinja 模板、Webhook 接收、record-replay 代理 | +| **模拟服务器** | 静态、动态、stateful、故障注入、OpenAPI 驱动、Jinja 模板、Webhook 接收、record-replay 代理、WebSocket 与 gRPC 端点 | | **Runner** | 顺序与并行、Tag 过滤、Dependency-aware 排序、Retry 策略 | | **报告** | HTML / JSON / XML / **JUnit / Allure / Markdown** / shields.io badge / SQLite 趋势库 / Run diff | | **生态集成** | Slack / Teams / Discord webhook、GitHub PR comment、cURL & HAR 导入、OpenAPI / Postman 导入 | | **CLI / DX** | 子命令式 CLI、REPL、终端摘要、Shell completion、Scaffold | | **安全** | Auth helper(Basic / Bearer / JWT / AWS SigV4)、Header / CORS / Rate limit / SSRF probe、pip-audit、Fuzz | | **Spec 反推** | 测试记录 → OpenAPI、JSON Schema 推断、OpenAPI changelog | +| **契约测试** | 从测试运行生成 Pact v2 消费者契约、提供端验证、与 OpenAPI 双向比较 | | **AI** | 可插拔后端,LLM 不可用时自动回退到确定性 fallback | | **MCP** | 一级支持 Claude Code,将框架作为 MCP 工具暴露 | | **GUI** | 可选 PySide6 GUI(英 / 繁中 / 简中 / 日)+ 嵌入 Swagger UI | @@ -349,6 +351,7 @@ apitestka completion bash # source >> ~/.bashrc apitestka mcp # 走 stdio 启动 MCP server apitestka openapi --report run_success.json -o openapi.json # 从保存的运行结果反推规格 apitestka openapi --run actions.json # 先运行再打印反推的规格 +apitestka contract verify pacts/web-shop.json --base-url http://localhost:8000 # 见契约测试 ``` ### 安全检测 @@ -385,6 +388,42 @@ openapi_changelog(prev_spec, current_spec) # markdown changelog 保存的报告是 `generate_json_report` 生成的 `_success.json`;`AT_export_openapi` 命令与 `apitestka openapi` 也能读取它。 +### 契约测试 + +Pact 风格的双向消费者契约(`je_api_testka.contract`)。消费端的测试运行会变成 Pact v2 文件格式的 +契约,可以直接发送到 Pact Broker 或 Pactflow。响应体默认带 `type` 匹配规则:提供端必须返回相同的 +结构与值的类型,而不是相同的值。契约接着从两边检查: + +- **提供端验证**:把每个交互重放到运行中的提供端,检查状态码相同、契约列出的 header 都存在 + (`Content-Type` 只比较 media type)、响应体符合匹配规则(`type`、`regex`、`min`;对象可以多带字段)。 + 提供端状态(`providerState`)通过 Python 函数或设置 URL 建立,设置 URL 会收到 + `{"consumer": ..., "state": ...}`。 +- **双向检查**:不用启动提供端,直接把契约和提供端的 OpenAPI 文档比较:operation 存在(路径模板可匹配)、 + 查询参数已声明且必填的已发送、请求体符合 schema、状态码已声明(完全相同、`2XX` 或 `default`)、 + 响应体符合 schema(`$ref`、`allOf` / `anyOf` / `oneOf`、`nullable`)。提供端用自己的测试运行 + 保持 OpenAPI 文档正确,例如用 `apitestka openapi`。 + +```python +from je_api_testka.contract import write_contract, verify_contract, check_contract_against_openapi + +# 消费端:测试运行(当前记录加上保存的报告)变成契约 +write_contract("pacts/web-shop.json", consumer="web", provider="shop", report_paths=["run_success.json"]) +# 提供端:把契约重放到运行中的提供端 +verify_contract("pacts/web-shop.json", base_url="http://localhost:8000") +# 双向:不用提供端,直接和它的 OpenAPI 文档比较 +check_contract_against_openapi("pacts/web-shop.json", "openapi.json") +``` + +```bash +apitestka contract record --report run_success.json --consumer web --provider shop -o pacts/web-shop.json +apitestka contract verify pacts/web-shop.json --base-url http://localhost:8000 +apitestka contract compare pacts/web-shop.json openapi.json +``` + +`verify` 与 `compare` 会打印报告(`--json` 输出 JSON),契约不成立时退出码为 1。JSON action: +`AT_write_contract`、`AT_verify_contract`、`AT_check_contract_against_openapi`(后两者失败时该 action +会带着报告失败)。 + ### GUI ```bash @@ -463,6 +502,7 @@ je_api_testka/ ├── ai/ # 可插拔 AI 后端 + 周边 ├── cli/ # apitestka CLI 子命令、REPL、shell completion ├── connection/ # ConnectionOptions、DNS override、Cassette +├── contract/ # Pact 风格消费者契约、提供端验证、OpenAPI 比较 ├── data/ # VariableStore、template、faker、env profile ├── diff/ # Response diff / contract drift / SLA ├── graphql_wrapper/ # GraphQL helper diff --git a/README/README_zh-TW.md b/README/README_zh-TW.md index b1c5a71..c355e51 100644 --- a/README/README_zh-TW.md +++ b/README/README_zh-TW.md @@ -33,6 +33,7 @@ record-replay proxy、安全性檢測、平行執行 runner,以及給 Claude 用 - [CLI / 開發體驗](#cli--開發體驗) - [安全檢測](#安全檢測) - [OpenAPI 反推](#openapi-反推) + - [契約測試](#契約測試) - [GUI](#gui) - [可插拔 AI 後端](#可插拔-ai-後端) - [Claude 用的 MCP Server](#claude-用的-mcp-server) @@ -53,13 +54,14 @@ record-replay proxy、安全性檢測、平行執行 runner,以及給 Claude 用 | **資料層** | 變數儲存區、`{{var}}` 模板、CSV/JSON 資料驅動、環境設定檔、假資料 | | **斷言** | 欄位斷言、JSON Schema、JSONPath、Snapshot、結構化 diff、OpenAPI contract drift、回應時間 SLA | | **連線** | mTLS、Proxy、DNS override、VCR-style cassette 錄製/回放 | -| **模擬伺服器** | 靜態、動態、stateful、故障注入、OpenAPI 驅動、Jinja 模板、Webhook 接收、record-replay proxy | +| **模擬伺服器** | 靜態、動態、stateful、故障注入、OpenAPI 驅動、Jinja 模板、Webhook 接收、record-replay proxy、WebSocket 與 gRPC 端點 | | **Runner** | 順序與平行執行、Tag 過濾、Dependency-aware 排序、Retry 策略 | | **報告** | HTML / JSON / XML / **JUnit / Allure / Markdown** / shields.io badge / SQLite 趨勢資料庫 / Run diff | | **生態整合** | Slack / Teams / Discord webhook、GitHub PR comment、cURL & HAR 匯入、OpenAPI / Postman 匯入 | | **CLI / DX** | 子命令式 CLI、REPL、終端摘要、Shell completion、Scaffold | | **安全** | Auth helper(Basic / Bearer / JWT / AWS SigV4)、Header / CORS / Rate limit / SSRF probe、pip-audit、Fuzz | | **Spec 反推** | 測試紀錄 → OpenAPI、JSON Schema 推斷、OpenAPI changelog | +| **契約測試** | 從測試執行產生 Pact v2 消費者契約、提供端驗證、與 OpenAPI 雙向比對 | | **AI** | 可插拔後端,LLM 不可用時自動退回確定性 fallback | | **MCP** | 一級支援 Claude Code,將框架曝露為 MCP 工具 | | **GUI** | 可選 PySide6 GUI(英 / 繁中 / 简中 / 日)+ 嵌入 Swagger UI | @@ -349,6 +351,7 @@ apitestka completion bash # source >> ~/.bashrc apitestka mcp # 走 stdio 啟動 MCP server apitestka openapi --report run_success.json -o openapi.json # 從存下的執行結果反推規格 apitestka openapi --run actions.json # 先執行再印出反推的規格 +apitestka contract verify pacts/web-shop.json --base-url http://localhost:8000 # 見契約測試 ``` ### 安全檢測 @@ -385,6 +388,42 @@ openapi_changelog(prev_spec, current_spec) # markdown changelog 存下的報告是 `generate_json_report` 產生的 `_success.json`;`AT_export_openapi` 指令與 `apitestka openapi` 也讀得到它。 +### 契約測試 + +Pact 風格的雙向消費者契約(`je_api_testka.contract`)。消費端的測試執行會變成 Pact v2 檔案格式的 +契約,可以直接送到 Pact Broker 或 Pactflow。回應內容預設帶 `type` 比對規則:提供端必須回傳相同的 +結構與值的型別,而不是相同的值。契約接著從兩邊檢查: + +- **提供端驗證**:把每個互動重播到執行中的提供端,檢查狀態碼相同、契約列出的 header 都在 + (`Content-Type` 只比 media type)、內容符合比對規則(`type`、`regex`、`min`;物件可以多帶欄位)。 + 提供端狀態(`providerState`)透過 Python 函式或設定 URL 建立,設定 URL 會收到 + `{"consumer": ..., "state": ...}`。 +- **雙向檢查**:不用啟動提供端,直接把契約和提供端的 OpenAPI 文件比對:operation 存在(路徑樣板可比對)、 + 查詢參數有宣告且必填的有送、請求內容符合 schema、狀態碼有宣告(完全相同、`2XX` 或 `default`)、 + 回應內容符合 schema(`$ref`、`allOf` / `anyOf` / `oneOf`、`nullable`)。提供端用自己的測試執行 + 維持 OpenAPI 文件正確,例如用 `apitestka openapi`。 + +```python +from je_api_testka.contract import write_contract, verify_contract, check_contract_against_openapi + +# 消費端:測試執行(目前紀錄加上存下的報告)變成契約 +write_contract("pacts/web-shop.json", consumer="web", provider="shop", report_paths=["run_success.json"]) +# 提供端:把契約重播到執行中的提供端 +verify_contract("pacts/web-shop.json", base_url="http://localhost:8000") +# 雙向:不用提供端,直接和它的 OpenAPI 文件比對 +check_contract_against_openapi("pacts/web-shop.json", "openapi.json") +``` + +```bash +apitestka contract record --report run_success.json --consumer web --provider shop -o pacts/web-shop.json +apitestka contract verify pacts/web-shop.json --base-url http://localhost:8000 +apitestka contract compare pacts/web-shop.json openapi.json +``` + +`verify` 與 `compare` 會印出報告(`--json` 輸出 JSON),契約不成立時結束碼為 1。JSON action: +`AT_write_contract`、`AT_verify_contract`、`AT_check_contract_against_openapi`(後兩者失敗時該 action +會帶著報告失敗)。 + ### GUI ```bash @@ -463,6 +502,7 @@ je_api_testka/ ├── ai/ # 可插拔 AI 後端 + 周邊 ├── cli/ # apitestka CLI 子命令、REPL、shell completion ├── connection/ # ConnectionOptions、DNS override、Cassette +├── contract/ # Pact 風格消費者契約、提供端驗證、OpenAPI 比對 ├── data/ # VariableStore、template、faker、env profile ├── diff/ # Response diff / contract drift / SLA ├── graphql_wrapper/ # GraphQL helper diff --git a/architecture.md b/architecture.md index f262f6d..3be8f5c 100644 --- a/architecture.md +++ b/architecture.md @@ -32,7 +32,8 @@ CLI, a TCP socket server, an MCP server, a pytest plugin and an optional PySide6 | `je_api_testka/utils/{json,xml,file_process,logging,exception,retry,observability}/` | JSON and XML I/O, directory listing, `apitestka_logger` (file at `$APITESTKA_LOG_FILE` or `~/.je_api_testka/logs/APITestka.log`, opened on first use; the root logger is left alone), exception hierarchy, `RetryPolicy`, OpenTelemetry hooks | | `je_api_testka/data/` | Variable store, template rendering, env profiles, fake-data helpers, data rows | | `je_api_testka/connection/` | Connection options, DNS override, record/replay cassettes | -| `je_api_testka/diff/`, `spec/` | Response and contract diff, SLA checks; schema inference, records → OpenAPI (`records_to_openapi`; `openapi_export` reads saved JSON reports and writes the spec), OpenAPI changelog | +| `je_api_testka/diff/`, `spec/` | Response and contract diff, SLA checks; schema inference, records → OpenAPI (`records_to_openapi`; `openapi_export` reads saved JSON reports and writes the spec), OpenAPI changelog, path-template matching (`path_templates`) | +| `je_api_testka/contract/` | Pact v2 consumer contracts from records (`pact.py`), matching rules (`matching.py`), provider verification (`verifier.py`), bidirectional check against an OpenAPI document (`openapi_compat.py`, `openapi_schema.py`), file-level steps for the executor and CLI (`commands.py`) | | `je_api_testka/security/` | Auth header helpers, CORS/SSRF/rate-limit probes, header scan, fuzzing, `pip-audit` wrapper | | `je_api_testka/runner/` | Parallel runner, tag filter, dependency ordering of actions | | `je_api_testka/integrations/` | cURL and HAR import, webhook notify, GitHub PR comment | @@ -60,7 +61,8 @@ CLI, a TCP socket server, an MCP server, a pytest plugin and an optional PySide6 `-c/--create_project` and `--execute_str`. On `win32`/`cygwin`/`msys`, `--execute_str` is decoded with `json.loads` twice. Errors print `repr(error)` to stderr and exit with code 1. - **Subcommand CLI**: `apitestka` (`je_api_testka.cli.cli_main:main`) with `run`, `create`, `mock`, - `import`, `repl`, `summary`, `scaffold`, `completion`, `mcp` and `openapi`. `cli/completion.py` + `import`, `repl`, `summary`, `scaffold`, `completion`, `mcp`, `openapi` and `contract record|verify|compare` + (`cli/contract_cli.py`); shared helpers are in `cli/cli_common.py`. `cli/completion.py` `SUBCOMMANDS` lists the same names (a test compares them with the parser). - **MCP**: `apitestka-mcp`, `python -m je_api_testka.mcp_server` or `apitestka mcp` (stdio, needs the `mcp` extra, mcp 1.x or 2.x: `build_server` registers the handlers with the 1.x `list_tools()`/`call_tool()` diff --git a/docs/source/Eng/Eng_index.rst b/docs/source/Eng/Eng_index.rst index e2b2780..449f1d9 100644 --- a/docs/source/Eng/Eng_index.rst +++ b/docs/source/Eng/Eng_index.rst @@ -30,7 +30,7 @@ Features - mTLS, proxies, DNS override, VCR-style cassette record/replay * - **Mock server** - Static, dynamic, stateful, fault injection, OpenAPI-driven, Jinja - templating, webhook receiver, record-replay proxy + templating, webhook receiver, record-replay proxy, WebSocket and gRPC endpoints * - **Runner** - Sequential & parallel execution, tag filters, dependency-aware ordering, retry policies @@ -47,6 +47,9 @@ Features rate-limit / SSRF probes, pip-audit wrapper, fuzz inputs * - **Spec inference** - Test record → OpenAPI, JSON Schema inference, OpenAPI changelog + * - **Contract testing** + - Pact v2 consumer contracts from test runs, provider verification, + bidirectional check against OpenAPI * - **AI** - Pluggable backend with deterministic fallback for test generation, fake data, failure classification @@ -68,6 +71,7 @@ Features doc/data_layer/data_layer_doc doc/assertion/assertion_doc doc/diff/diff_doc + doc/contract/contract_doc doc/connection/connection_doc doc/runner/runner_doc doc/report/report_doc diff --git a/docs/source/Eng/doc/cli/cli_doc.rst b/docs/source/Eng/doc/cli/cli_doc.rst index 7c5789a..b5f080f 100644 --- a/docs/source/Eng/doc/cli/cli_doc.rst +++ b/docs/source/Eng/doc/cli/cli_doc.rst @@ -80,6 +80,8 @@ The ``apitestka`` command (installed with the package) groups the everyday tasks - Run the MCP server over stdio * - ``openapi`` - Infer an OpenAPI document from recorded traffic (below) + * - ``contract record|verify|compare`` + - Record, verify and compare Pact-style consumer contracts (see Contract Testing) Inferring an OpenAPI document ----------------------------- diff --git a/docs/source/Eng/doc/contract/contract_doc.rst b/docs/source/Eng/doc/contract/contract_doc.rst new file mode 100644 index 0000000..2ceb103 --- /dev/null +++ b/docs/source/Eng/doc/contract/contract_doc.rst @@ -0,0 +1,89 @@ +================ +Contract Testing +================ + +Pact-style, bidirectional consumer contracts live in ``je_api_testka.contract``. + +Consumer contracts +------------------ + +The consumer's test run becomes a contract in the Pact specification v2 file format, so the file +can go to a Pact Broker or Pactflow as it is. Records with the same method, path, query and status +become one interaction. Response bodies get a ``type`` matching rule on ``$.body`` by default, so +the provider must return the same structure and value types, not the same values; pass +``body_rule="equality"`` for exact values. ``base_path`` drops a prefix such as ``/api/v1`` from +request paths. + +.. code-block:: python + + from je_api_testka.contract import add_interaction, new_pact, write_contract, write_pact + + write_contract("pacts/web-shop.json", consumer="web", provider="shop", + report_paths=["run_success.json"], base_path="/api/v1") + + pact = new_pact("web", "shop") # or write interactions by hand + add_interaction(pact, "get item 1", {"method": "GET", "path": "/items/1"}, + {"status": 200, "body": {"id": 1, "name": "a"}}, + provider_state="item 1 exists", + matching_rules={"$.body": {"match": "type"}}) + write_pact(pact, "pacts/web-shop.json") + +Matching rules +-------------- + +Without a rule, values must be equal. A rule applies to its path and everything below it, and the +most specific rule wins; ``[*]`` and ``.*`` match any index or key. + +* ``{"match": "type"}``: same JSON type. Every element of an actual array must match the first + expected element; ``"min"`` sets the minimum length. +* ``{"match": "regex", "regex": "..."}``: the value's text must fully match. + +Objects may carry keys the contract does not mention. Headers compare by name case-insensitively, +and ``Content-Type`` compares the media type only. + +Provider verification +--------------------- + +``verify_pact`` replays each interaction against a running provider. Provider states are set up +first, through a Python callable or a setup URL that receives ``{"consumer": ..., "state": ...}`` +as a JSON POST. + +.. code-block:: python + + from je_api_testka.contract import ProviderTarget, read_pact, verify_pact + + target = ProviderTarget("http://localhost:8000", state_handler=seed_database, timeout=10) + report = verify_pact(read_pact("pacts/web-shop.json"), target) + print(report.render_text()) # report.ok, report.to_dict() + +Bidirectional check +------------------- + +``check_pact_against_openapi`` compares the contract with the provider's OpenAPI document without +running the provider. For every interaction it checks that the operation exists (path templates +such as ``/items/{id}`` match, and the path of the first server URL is ignored), that query +parameters are declared and required ones sent, that the request body fits its schema, that the +status is declared (exactly, as ``2XX`` or as ``default``) and that the response body fits its +schema. Schemas may use local ``$ref``, ``allOf`` / ``anyOf`` / ``oneOf``, ``nullable``, ``enum``, +``required`` and ``additionalProperties: false``. The provider keeps its document honest from its +own test run, for example with ``apitestka openapi``. + +.. code-block:: python + + from je_api_testka.contract import check_contract_against_openapi + + check_contract_against_openapi("pacts/web-shop.json", "openapi.json") # raises on a problem + +Command line and JSON actions +----------------------------- + +.. code-block:: bash + + apitestka contract record --report run_success.json --consumer web --provider shop -o pacts/web-shop.json + apitestka contract verify pacts/web-shop.json --base-url http://localhost:8000 [--provider-states-url URL] + apitestka contract compare pacts/web-shop.json openapi.json + +``verify`` and ``compare`` print a report (``--json`` for JSON) and exit with 1 when the contract does +not hold, and with 2 when a file cannot be read. The executor commands are ``AT_write_contract``, +``AT_verify_contract`` and ``AT_check_contract_against_openapi``; the last two raise +``APIContractException`` carrying the report, so the action fails. diff --git a/docs/source/Zh/Zh_index.rst b/docs/source/Zh/Zh_index.rst index e373119..af41ae4 100644 --- a/docs/source/Zh/Zh_index.rst +++ b/docs/source/Zh/Zh_index.rst @@ -29,7 +29,7 @@ HAR / cURL 匯入器、record-replay proxy、安全檢測、平行 runner、給 - mTLS、Proxy、DNS override、VCR-style cassette 錄製/回放 * - **模擬伺服器** - 靜態、動態、stateful、故障注入、OpenAPI 驅動、Jinja 模板、Webhook - 接收、record-replay proxy + 接收、record-replay proxy、WebSocket 與 gRPC 端點 * - **Runner** - 順序 & 平行執行、Tag 過濾、Dependency-aware 排序、Retry 策略 * - **報告** @@ -45,6 +45,8 @@ HAR / cURL 匯入器、record-replay proxy、安全檢測、平行 runner、給 Rate limit / SSRF probe、pip-audit、Fuzz * - **Spec 反推** - 測試紀錄 → OpenAPI、JSON Schema 推斷、OpenAPI changelog + * - **契約測試** + - 從測試執行產生 Pact v2 消費者契約、提供端驗證、與 OpenAPI 雙向比對 * - **AI** - 可插拔後端,LLM 不可用時自動退回確定性 fallback * - **MCP** @@ -65,6 +67,7 @@ HAR / cURL 匯入器、record-replay proxy、安全檢測、平行 runner、給 doc/data_layer/data_layer_doc doc/assertion/assertion_doc doc/diff/diff_doc + doc/contract/contract_doc doc/connection/connection_doc doc/runner/runner_doc doc/report/report_doc diff --git a/docs/source/Zh/doc/cli/cli_doc.rst b/docs/source/Zh/doc/cli/cli_doc.rst index ff897bb..98f547e 100644 --- a/docs/source/Zh/doc/cli/cli_doc.rst +++ b/docs/source/Zh/doc/cli/cli_doc.rst @@ -80,6 +80,8 @@ CLI 參數 - 以 stdio 啟動 MCP server * - ``openapi`` - 從錄下的流量反推 OpenAPI 文件（見下節） + * - ``contract record|verify|compare`` + - 產生、驗證與比對 Pact 風格消費者契約（見契約測試一章） 反推 OpenAPI 文件 ---------------------- diff --git a/docs/source/Zh/doc/contract/contract_doc.rst b/docs/source/Zh/doc/contract/contract_doc.rst new file mode 100644 index 0000000..33f10d1 --- /dev/null +++ b/docs/source/Zh/doc/contract/contract_doc.rst @@ -0,0 +1,82 @@ +======== +契約測試 +======== + +Pact 風格的雙向消費者契約放在 ``je_api_testka.contract``。 + +消費者契約 +---------- + +消費端的測試執行會變成 Pact 規格 v2 檔案格式的契約，檔案可以直接送到 Pact Broker 或 Pactflow。 +method、路徑、查詢字串與狀態碼都相同的紀錄會合成一個互動。回應內容預設在 ``$.body`` 帶 ``type`` +比對規則，所以提供端必須回傳相同的結構與值的型別，而不是相同的值；要比對確切的值就傳 +``body_rule="equality"``。``base_path`` 會從請求路徑拿掉 ``/api/v1`` 這類前綴。 + +.. code-block:: python + + from je_api_testka.contract import add_interaction, new_pact, write_contract, write_pact + + write_contract("pacts/web-shop.json", consumer="web", provider="shop", + report_paths=["run_success.json"], base_path="/api/v1") + + pact = new_pact("web", "shop") # 也可以手寫互動 + add_interaction(pact, "get item 1", {"method": "GET", "path": "/items/1"}, + {"status": 200, "body": {"id": 1, "name": "a"}}, + provider_state="item 1 exists", + matching_rules={"$.body": {"match": "type"}}) + write_pact(pact, "pacts/web-shop.json") + +比對規則 +-------- + +沒有規則時值必須相同。規則作用在自己的路徑以及底下所有節點，最具體的規則優先；``[*]`` 與 ``.*`` +可比對任何索引或鍵。 + +* ``{"match": "type"}``：JSON 型別相同。實際陣列的每個元素都必須符合預期的第一個元素；``"min"`` + 設定最少元素數。 +* ``{"match": "regex", "regex": "..."}``：值的文字必須完全符合。 + +物件可以多帶契約沒提到的鍵。header 名稱不分大小寫，``Content-Type`` 只比 media type。 + +提供端驗證 +---------- + +``verify_pact`` 把每個互動重播到執行中的提供端。會先建立提供端狀態，方法是呼叫 Python 函式，或對 +設定 URL 送出內容為 ``{"consumer": ..., "state": ...}`` 的 JSON POST。 + +.. code-block:: python + + from je_api_testka.contract import ProviderTarget, read_pact, verify_pact + + target = ProviderTarget("http://localhost:8000", state_handler=seed_database, timeout=10) + report = verify_pact(read_pact("pacts/web-shop.json"), target) + print(report.render_text()) # report.ok、report.to_dict() + +雙向檢查 +-------- + +``check_pact_against_openapi`` 不用啟動提供端，直接把契約和提供端的 OpenAPI 文件比對。每個互動都檢查： +operation 存在（``/items/{id}`` 這類路徑樣板可比對，第一個 server URL 的路徑會被忽略）、查詢參數 +有宣告且必填的有送、請求內容符合 schema、狀態碼有宣告（完全相同、``2XX`` 或 ``default``）、回應內容 +符合 schema。schema 可以用本地 ``$ref``、``allOf`` / ``anyOf`` / ``oneOf``、``nullable``、``enum``、 +``required`` 與 ``additionalProperties: false``。提供端用自己的測試執行維持文件正確，例如用 +``apitestka openapi``。 + +.. code-block:: python + + from je_api_testka.contract import check_contract_against_openapi + + check_contract_against_openapi("pacts/web-shop.json", "openapi.json") # 有問題就丟例外 + +命令列與 JSON action +-------------------- + +.. code-block:: bash + + apitestka contract record --report run_success.json --consumer web --provider shop -o pacts/web-shop.json + apitestka contract verify pacts/web-shop.json --base-url http://localhost:8000 [--provider-states-url URL] + apitestka contract compare pacts/web-shop.json openapi.json + +``verify`` 與 ``compare`` 會印出報告（``--json`` 輸出 JSON），契約不成立時結束碼為 1，檔案讀不到時為 2。 +executor 指令是 ``AT_write_contract``、``AT_verify_contract`` 與 ``AT_check_contract_against_openapi``； +後兩者會丟出帶著報告的 ``APIContractException``，讓該 action 失敗。 diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 9c6fe61..52d66e7 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -37,3 +37,16 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Tests**: `test/test_mock_server/test_websocket_mock.py` (new; includes a round trip through APITestka's own `test_api_method_websocket`), `test_grpc_stub.py` (extended), `test_protocol_mocks.py` and `test_mock_config.py` (new, including `apitestka mock --config` with the blocking Flask run patched out). - **Result / numbers**: `pytest` 522 passed, 2 skipped (460 before); `sphinx -W` clean. - **Files**: `je_api_testka/utils/mock_server/{websocket_mock,grpc_stub,protocol_mocks,mock_config}.py`, `je_api_testka/utils/executor/action_executor.py`, `je_api_testka/cli/cli_main.py`, `pyproject.toml`, `dev.toml`, `.github/requirements/ci.{in,txt}`, `uv.lock`; README ×3, `docs/source/{Eng,Zh}/doc/{mock_server,cli}`, `architecture.md` §2, §3, §5, §7. + +## U-20261001-04 · 2026-10-01 · Pact-style bidirectional contract testing · #done #contract + +- **What** (`progress.md` #9): new package `je_api_testka/contract/`. + - **Consumer contracts** (`pact.py`): records become Pact specification v2 files that a Pact Broker or Pactflow accepts as they are. Records with the same method, path, query and status make one interaction. The response body gets a `type` rule on `$.body` by default (`body_rule="equality"` for exact values), and `base_path` drops a prefix such as `/api/v1`. `add_interaction` writes interactions by hand with `providerState` and matching rules; `read_pact` validates a file. + - **Matching** (`matching.py`): Pact v2 `type` / `regex` / `min` rules with cascading, most-specific-wins lookup and `[*]` / `.*` wildcards; objects may carry extra keys; headers compare case-insensitively and `Content-Type` by media type. `1` and `true` are not equal. + - **Provider verification** (`verifier.py`): replays each interaction against a running provider through `requests` (with timeout, no redirects). Provider states are set through a callable or a setup URL that receives `{"consumer", "state"}`. A failed connection counts as a mismatch, and a contract with no interactions does not pass. + - **Bidirectional check** (`openapi_compat.py`, `openapi_schema.py`): checks the contract against the provider's OpenAPI document without running the provider. The operation must exist: path templates match, the most literal template wins, and the first server URL's path is ignored. Query parameters must be declared, and required ones sent (path-level and `$ref` parameters count). Request and response bodies must fit their schemas, and the status must be declared exactly or as `2XX` / `default`. A dependency-free schema check covers local `$ref`, 3.1 type lists, 3.0 `nullable`, `enum`, `required`, `additionalProperties: false`, `items`, `allOf` / `anyOf` / `oneOf`. + - **Entry points**: `apitestka contract record|verify|compare` (`cli/contract_cli.py`; exit 1 when the contract fails, 2 for unreadable files), `AT_write_contract`, `AT_verify_contract`, `AT_check_contract_against_openapi` (the last two raise `APIContractException` with the report). `APIContractException` joins the exception hierarchy. +- **Shared pieces**: `spec/path_templates.py` (template matching, server base path) for reuse by #13; `records_to_openapi.decode_body` is now public; `openapi_export.collect_records` feeds both OpenAPI export and contracts; `cli/cli_common.run_action_path` replaces the private `_run_path` in `cli_main.py`. +- **Tests**: new `test/test_contract/` (pact, matching, schema and compatibility, verifier against a Flask provider on a free port, commands, CLI and executor). It includes an end-to-end flow: the consumer run becomes a contract, the provider run becomes its OpenAPI document, the two are compatible, and the contract verifies against the live provider. +- **Result / numbers**: `pytest` 587 passed, 2 skipped (522 before); `sphinx -W` clean. +- **Docs**: README ×3 (highlights, table of contents, new Contract Testing section, CLI, project tree; the Mock server highlight also gains the #8 WebSocket and gRPC endpoints), new `docs/source/{Eng,Zh}/doc/contract/contract_doc.rst`, CLI tables, `architecture.md` §2–§3, `CLAUDE.md` package tree. diff --git a/docs/updates/README.md b/docs/updates/README.md index 065330c..5115db7 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261001-04 | 2026-10-01 | Pact-style bidirectional contract testing | #done #contract | [2026-10](2026-10.md) | | U-20261001-03 | 2026-10-01 | Mock server serves WebSocket and gRPC endpoints | #done #mock #grpc #websocket | [2026-10](2026-10.md) | | U-20261001-02 | 2026-10-01 | Imported, scaffolded and AI-generated actions run as they are | #bugfix #executor | [2026-10](2026-10.md) | | U-20261001-01 | 2026-10-01 | records_to_openapi reachable from the CLI and MCP | #done #openapi #cli #mcp | [2026-10](2026-10.md) | @@ -89,5 +90,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 3 | +| [2026-10.md](2026-10.md) | 2026-10 | 4 | | [2026-09.md](2026-09.md) | 2026-09 | 23 | diff --git a/je_api_testka/cli/cli_common.py b/je_api_testka/cli/cli_common.py new file mode 100644 index 0000000..de1eec4 --- /dev/null +++ b/je_api_testka/cli/cli_common.py @@ -0,0 +1,25 @@ +"""Helpers shared by the ``apitestka`` subcommands.""" +from __future__ import annotations + +from pathlib import Path + +from je_api_testka.utils.executor.action_executor import execute_action, execute_files +from je_api_testka.utils.file_process.get_dir_file_list import get_dir_files_as_list +from je_api_testka.utils.json.json_file.json_file import read_action_json +from je_api_testka.utils.logging.loggin_instance import apitestka_logger + +EXIT_OK: int = 0 +EXIT_FAILED: int = 1 +EXIT_USAGE: int = 2 + + +def run_action_path(target: Path) -> int: + """Execute an action JSON file, or every JSON file in a directory; return 2 when ``target`` is missing.""" + if target.is_dir(): + execute_files(get_dir_files_as_list(str(target))) + elif target.is_file(): + execute_action(read_action_json(str(target))) + else: + apitestka_logger.error(f"cli: path not found: {target}") + return EXIT_USAGE + return EXIT_OK diff --git a/je_api_testka/cli/cli_main.py b/je_api_testka/cli/cli_main.py index 7d29885..36c8d47 100644 --- a/je_api_testka/cli/cli_main.py +++ b/je_api_testka/cli/cli_main.py @@ -12,6 +12,7 @@ completion Print a shell completion script. mcp Run the MCP server over stdio. openapi Infer an OpenAPI document from recorded traffic. + contract Record, verify and compare Pact-style consumer contracts. """ from __future__ import annotations @@ -21,11 +22,10 @@ from pathlib import Path from typing import Optional, Sequence +from je_api_testka.cli.cli_common import run_action_path +from je_api_testka.cli.contract_cli import configure_contract_parser from je_api_testka.spec.openapi_export import build_openapi from je_api_testka.spec.records_to_openapi import DEFAULT_SPEC_TITLE, DEFAULT_SPEC_VERSION -from je_api_testka.utils.executor.action_executor import execute_action, execute_files -from je_api_testka.utils.file_process.get_dir_file_list import get_dir_files_as_list -from je_api_testka.utils.json.json_file.json_file import read_action_json from je_api_testka.utils.logging.loggin_instance import apitestka_logger from je_api_testka.utils.project.create_project_structure import create_project_dir @@ -33,19 +33,8 @@ DEFAULT_MOCK_PORT: int = 8090 -def _run_path(target: Path) -> int: - if target.is_dir(): - execute_files(get_dir_files_as_list(str(target))) - elif target.is_file(): - execute_action(read_action_json(str(target))) - else: - apitestka_logger.error(f"cli run: path not found: {target}") - return 2 - return 0 - - def _cmd_run(args: argparse.Namespace) -> int: - return _run_path(Path(args.path)) + return run_action_path(Path(args.path)) def _cmd_create(args: argparse.Namespace) -> int: @@ -118,7 +107,7 @@ def _cmd_openapi(args: argparse.Namespace) -> int: apitestka_logger.error("cli openapi: give at least one --report or --run") return 2 for path in args.run: - status = _run_path(Path(path)) + status = run_action_path(Path(path)) if status: return status spec = build_openapi(args.report, title=args.title, version=args.api_version) @@ -193,6 +182,7 @@ def build_parser() -> argparse.ArgumentParser: mcp_parser.set_defaults(func=_cmd_mcp) _configure_openapi_parser(sub.add_parser("openapi", help="Infer an OpenAPI document from recorded traffic")) + configure_contract_parser(sub.add_parser("contract", help="Record, verify and compare consumer contracts")) return parser diff --git a/je_api_testka/cli/completion.py b/je_api_testka/cli/completion.py index 193f871..979ef94 100644 --- a/je_api_testka/cli/completion.py +++ b/je_api_testka/cli/completion.py @@ -7,7 +7,7 @@ from __future__ import annotations SUBCOMMANDS = ( - "run", "create", "mock", "import", "repl", "summary", "scaffold", "completion", "mcp", "openapi", + "run", "create", "mock", "import", "repl", "summary", "scaffold", "completion", "mcp", "openapi", "contract", ) SUPPORTED_SHELLS = ("bash", "zsh", "fish", "powershell") diff --git a/je_api_testka/cli/contract_cli.py b/je_api_testka/cli/contract_cli.py new file mode 100644 index 0000000..5c503d5 --- /dev/null +++ b/je_api_testka/cli/contract_cli.py @@ -0,0 +1,106 @@ +""" +``apitestka contract``: Pact-style consumer contracts from the command line. + + apitestka contract record --report run_success.json --consumer web --provider shop -o pacts/web-shop.json + apitestka contract verify pacts/web-shop.json --base-url http://localhost:8000 + apitestka contract compare pacts/web-shop.json openapi.json + +``verify`` and ``compare`` print a report and exit with 1 when the contract does not hold. +""" +from __future__ import annotations + +import argparse +import json +from pathlib import Path +from typing import Union + +from je_api_testka.cli.cli_common import EXIT_FAILED, EXIT_OK, EXIT_USAGE, run_action_path +from je_api_testka.contract.commands import read_openapi, write_contract +from je_api_testka.contract.openapi_compat import CompatibilityReport, check_pact_against_openapi +from je_api_testka.contract.pact import BODY_RULE_EQUALITY, BODY_RULE_TYPE, read_pact +from je_api_testka.contract.verifier import ( + DEFAULT_VERIFY_TIMEOUT_SECONDS, + ProviderTarget, + VerificationReport, + verify_pact, +) +from je_api_testka.utils.exception.exceptions import APIContractException +from je_api_testka.utils.logging.loggin_instance import apitestka_logger + + +def _print_report(report: Union[VerificationReport, CompatibilityReport], as_json: bool) -> int: + print(json.dumps(report.to_dict(), indent=2, ensure_ascii=False) if as_json else report.render_text()) + return EXIT_OK if report.ok else EXIT_FAILED + + +def _cmd_record(args: argparse.Namespace) -> int: + if not args.report and not args.run: + apitestka_logger.error("cli contract record: give at least one --report or --run") + return EXIT_USAGE + for path in args.run: + status = run_action_path(Path(path)) + if status: + return status + try: + written = write_contract(args.output, args.consumer, args.provider, args.report, + base_path=args.base_path, body_rule=args.body_rule) + except APIContractException as error: + apitestka_logger.error(f"cli contract record: {error}") + return EXIT_FAILED + print(written) + return EXIT_OK + + +def _cmd_verify(args: argparse.Namespace) -> int: + try: + pact = read_pact(args.contract) + except APIContractException as error: + apitestka_logger.error(f"cli contract verify: {error}") + return EXIT_USAGE + target = ProviderTarget(args.base_url, provider_states_url=args.provider_states_url, timeout=args.timeout) + return _print_report(verify_pact(pact, target), args.json) + + +def _cmd_compare(args: argparse.Namespace) -> int: + try: + pact = read_pact(args.contract) + spec = read_openapi(args.openapi) + except APIContractException as error: + apitestka_logger.error(f"cli contract compare: {error}") + return EXIT_USAGE + return _print_report(check_pact_against_openapi(pact, spec), args.json) + + +def _add_record_parser(actions) -> None: + record = actions.add_parser("record", help="Record a consumer contract from test records") + record.add_argument("--report", action="append", default=[], metavar="FILE", + help="Saved JSON success report (_success.json); repeatable") + record.add_argument("--run", action="append", default=[], metavar="PATH", + help="Action JSON file or directory to execute first; repeatable") + record.add_argument("--consumer", required=True) + record.add_argument("--provider", required=True) + record.add_argument("--base-path", default="", help="Prefix to drop from request paths, e.g. /api/v1") + record.add_argument("--body-rule", choices=(BODY_RULE_TYPE, BODY_RULE_EQUALITY), default=BODY_RULE_TYPE, + help="Match response bodies by type (default) or by exact value") + record.add_argument("-o", "--output", required=True, help="Contract file to write") + record.set_defaults(func=_cmd_record) + + +def configure_contract_parser(contract_parser: argparse.ArgumentParser) -> None: + """Add the ``record``, ``verify`` and ``compare`` actions to the ``contract`` subcommand.""" + actions = contract_parser.add_subparsers(dest="contract_command", required=True) + _add_record_parser(actions) + + verify = actions.add_parser("verify", help="Replay a contract against a running provider") + verify.add_argument("contract", help="Contract file") + verify.add_argument("--base-url", required=True, help="Provider base URL") + verify.add_argument("--provider-states-url", help="URL that receives provider-state setup POSTs") + verify.add_argument("--timeout", type=float, default=DEFAULT_VERIFY_TIMEOUT_SECONDS) + verify.add_argument("--json", action="store_true", help="Print the report as JSON") + verify.set_defaults(func=_cmd_verify) + + compare = actions.add_parser("compare", help="Check a contract against the provider's OpenAPI document") + compare.add_argument("contract", help="Contract file") + compare.add_argument("openapi", help="Provider OpenAPI document (JSON)") + compare.add_argument("--json", action="store_true", help="Print the report as JSON") + compare.set_defaults(func=_cmd_compare) diff --git a/je_api_testka/contract/__init__.py b/je_api_testka/contract/__init__.py new file mode 100644 index 0000000..513d827 --- /dev/null +++ b/je_api_testka/contract/__init__.py @@ -0,0 +1,32 @@ +from je_api_testka.contract.commands import check_contract_against_openapi, verify_contract, write_contract +from je_api_testka.contract.matching import body_mismatches, header_mismatches +from je_api_testka.contract.openapi_compat import CompatibilityReport, check_pact_against_openapi +from je_api_testka.contract.pact import ( + add_interaction, + interaction_from_record, + new_pact, + pact_from_records, + read_pact, + write_pact, +) +from je_api_testka.contract.verifier import InteractionResult, ProviderTarget, VerificationReport, verify_pact + +__all__ = [ + "CompatibilityReport", + "InteractionResult", + "ProviderTarget", + "VerificationReport", + "add_interaction", + "body_mismatches", + "check_contract_against_openapi", + "check_pact_against_openapi", + "header_mismatches", + "interaction_from_record", + "new_pact", + "pact_from_records", + "read_pact", + "verify_contract", + "verify_pact", + "write_contract", + "write_pact", +] diff --git a/je_api_testka/contract/commands.py b/je_api_testka/contract/commands.py new file mode 100644 index 0000000..342ac56 --- /dev/null +++ b/je_api_testka/contract/commands.py @@ -0,0 +1,64 @@ +""" +File-level contract steps used by the executor (``AT_*``) and ``apitestka contract``. + +* consumer side: :func:`write_contract` records a contract from test records; +* provider side: :func:`verify_contract` replays it against a running provider; +* bidirectional: :func:`check_contract_against_openapi` compares it with the + provider's OpenAPI document without running the provider. + +The checking steps return a JSON-ready report and raise +:class:`APIContractException` (carrying the text report) when the contract does +not hold, so a failing step shows up as a failed action. +""" +from __future__ import annotations + +import json +from pathlib import Path +from typing import Optional, Sequence + +from je_api_testka.contract.openapi_compat import assert_pact_compatible +from je_api_testka.contract.pact import BODY_RULE_TYPE, pact_from_records, read_pact, write_pact +from je_api_testka.contract.verifier import DEFAULT_VERIFY_TIMEOUT_SECONDS, ProviderTarget, assert_pact_verified +from je_api_testka.spec.openapi_export import collect_records +from je_api_testka.utils.exception.exceptions import APIContractException + + +def write_contract(output_path: str, consumer: str, provider: str, report_paths: Optional[Sequence[str]] = None, + base_path: str = "", body_rule: str = BODY_RULE_TYPE) -> str: + """ + Record a consumer contract from the current test record plus saved JSON reports and write it. + + :raises APIContractException: when there is no record to build an interaction from. + """ + pact = pact_from_records(collect_records(report_paths), consumer, provider, + base_path=base_path, body_rule=body_rule) + if not pact["interactions"]: + raise APIContractException("no successful records to build a contract from") + return write_pact(pact, output_path) + + +def verify_contract(contract_path: str, base_url: str, provider_states_url: Optional[str] = None, + timeout: float = DEFAULT_VERIFY_TIMEOUT_SECONDS) -> dict: + """Replay the contract at ``contract_path`` against the provider at ``base_url``.""" + target = ProviderTarget(base_url, provider_states_url=provider_states_url, timeout=timeout) + return assert_pact_verified(read_pact(contract_path), target) + + +def read_openapi(openapi_path: str) -> dict: + """ + Read an OpenAPI JSON document. + + :raises APIContractException: when the file is missing, not JSON or not a JSON object. + """ + try: + spec = json.loads(Path(openapi_path).read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as error: + raise APIContractException(f"{openapi_path}: cannot read OpenAPI document: {error!r}") from error + if not isinstance(spec, dict): + raise APIContractException(f"{openapi_path}: an OpenAPI document is a JSON object") + return spec + + +def check_contract_against_openapi(contract_path: str, openapi_path: str) -> dict: + """Check the contract at ``contract_path`` against the OpenAPI JSON document at ``openapi_path``.""" + return assert_pact_compatible(read_pact(contract_path), read_openapi(openapi_path)) diff --git a/je_api_testka/contract/matching.py b/je_api_testka/contract/matching.py new file mode 100644 index 0000000..8598d0e --- /dev/null +++ b/je_api_testka/contract/matching.py @@ -0,0 +1,155 @@ +""" +Compare an actual response with a contract's expected response (Pact v2 rules). + +Without a rule, values must be equal. A rule applies to its path and everything +below it, and the most specific rule wins. Paths look like ``$.body.items[*].id``, +where ``[*]`` and ``.*`` match any index or key. + +* ``{"match": "type"}``: same JSON type (numbers are one type). Under a type rule + every element of an actual array must match the first expected element, and + ``"min"`` sets the minimum length. +* ``{"match": "regex", "regex": "..."}``: the value's text must fully match. + +Objects may carry keys the contract does not mention, as in Pact. +""" +from __future__ import annotations + +import re +from typing import Any, List, Mapping, Optional, Sequence + +RuleMap = Mapping[str, Mapping[str, Any]] + +_TOKEN = re.compile(r"\['([^']*)'\]|\[(\*|\d+)\]|\.?([^.\[\]]+)") +_EQUALITY: str = "equality" + + +def _pattern_tokens(pattern: str) -> List[str]: + return [quoted or index or plain for quoted, index, plain in _TOKEN.findall(pattern)] + + +def _render(tokens: Sequence[str]) -> str: + text = tokens[0] + for token in tokens[1:]: + text += f"[{token}]" if token.isdigit() else f".{token}" + return text + + +def _rule_for(tokens: Sequence[str], rules: RuleMap) -> Optional[Mapping[str, Any]]: + best: Optional[Mapping[str, Any]] = None + best_length = -1 + for pattern, rule in rules.items(): + pattern_tokens = _pattern_tokens(pattern) + if len(pattern_tokens) > len(tokens) or len(pattern_tokens) <= best_length: + continue + # The pattern may be shorter than the path: a rule covers everything below it. + if all(expected in ("*", actual) for expected, actual in zip(pattern_tokens, tokens, strict=False)): + best, best_length = rule, len(pattern_tokens) + return best + + +def json_type(value: Any) -> str: + """Return the JSON type name of ``value`` (``number`` for int and float, ``boolean`` for bool).""" + if isinstance(value, bool): + return "boolean" + if isinstance(value, (int, float)): + return "number" + if isinstance(value, str): + return "string" + if isinstance(value, list): + return "array" + if isinstance(value, dict): + return "object" + return "null" if value is None else type(value).__name__ + + +class _Matcher: + def __init__(self, rules: RuleMap) -> None: + self.rules = rules + self.mismatches: List[str] = [] + + def compare(self, expected: Any, actual: Any, tokens: List[str]) -> None: + rule = _rule_for(tokens, self.rules) or {} + kind = rule.get("match", _EQUALITY) + if kind == "regex": + self._regex(rule, actual, tokens) + elif isinstance(expected, dict): + self._object(expected, actual, tokens) + elif isinstance(expected, list): + self._array(expected, actual, tokens, rule, kind) + elif kind == "type": + self._same_type(expected, actual, tokens) + elif expected != actual or json_type(expected) != json_type(actual): + self.mismatches.append(f"{_render(tokens)}: expected {expected!r}, got {actual!r}") + + def _regex(self, rule: Mapping[str, Any], actual: Any, tokens: List[str]) -> None: + if actual is None or not re.fullmatch(str(rule.get("regex", "")), str(actual)): + self.mismatches.append(f"{_render(tokens)}: {actual!r} does not match /{rule.get('regex')}/") + + def _same_type(self, expected: Any, actual: Any, tokens: List[str]) -> None: + if json_type(expected) != json_type(actual): + self.mismatches.append( + f"{_render(tokens)}: expected a {json_type(expected)}, got {json_type(actual)} {actual!r}") + + def _object(self, expected: dict, actual: Any, tokens: List[str]) -> None: + if not isinstance(actual, dict): + self.mismatches.append(f"{_render(tokens)}: expected an object, got {json_type(actual)}") + return + for key, value in expected.items(): + if key not in actual: + self.mismatches.append(f"{_render([*tokens, key])}: missing") + else: + self.compare(value, actual[key], [*tokens, key]) + + def _array(self, expected: list, actual: Any, tokens: List[str], rule: Mapping[str, Any], kind: str) -> None: + if not isinstance(actual, list): + self.mismatches.append(f"{_render(tokens)}: expected an array, got {json_type(actual)}") + return + minimum = rule.get("min") + if isinstance(minimum, int) and len(actual) < minimum: + self.mismatches.append(f"{_render(tokens)}: expected at least {minimum} items, got {len(actual)}") + if kind == "type": + for index, item in enumerate(actual if expected else []): + self.compare(expected[0], item, [*tokens, str(index)]) + return + if len(expected) != len(actual): + self.mismatches.append(f"{_render(tokens)}: expected {len(expected)} items, got {len(actual)}") + return + for index, (want, got) in enumerate(zip(expected, actual, strict=True)): + self.compare(want, got, [*tokens, str(index)]) + + +def body_mismatches(expected: Any, actual: Any, rules: Optional[RuleMap] = None) -> List[str]: + """Return why ``actual`` does not satisfy the expected body under ``rules`` (empty when it does).""" + matcher = _Matcher(rules or {}) + matcher.compare(expected, actual, ["$", "body"]) + return matcher.mismatches + + +def _media_type(value: str) -> str: + return value.split(";", 1)[0].strip().lower() + + +def header_mismatches(expected: Mapping[str, str], actual: Mapping[str, str], + rules: Optional[RuleMap] = None) -> List[str]: + """ + Return why ``actual`` headers do not satisfy ``expected`` (names compare case-insensitively). + + Without a rule, ``Content-Type`` compares the media type only, so a ``charset`` parameter does not matter. + """ + lowered = {str(name).lower(): str(value) for name, value in actual.items()} + mismatches: List[str] = [] + for name, want in expected.items(): + tokens = ["$", "headers", name] + got = lowered.get(name.lower()) + rule = _rule_for(tokens, rules or {}) + if got is None: + mismatches.append(f"{_render(tokens)}: missing") + elif rule and rule.get("match") == "regex": + if not re.fullmatch(str(rule.get("regex", "")), got): + mismatches.append(f"{_render(tokens)}: {got!r} does not match /{rule.get('regex')}/") + elif name.lower() == "content-type": + if _media_type(got) != _media_type(want): + mismatches.append(f"{_render(tokens)}: expected {want!r}, got {got!r}") + elif got.strip() != str(want).strip(): + mismatches.append(f"{_render(tokens)}: expected {want!r}, got {got!r}") + return mismatches diff --git a/je_api_testka/contract/openapi_compat.py b/je_api_testka/contract/openapi_compat.py new file mode 100644 index 0000000..900e422 --- /dev/null +++ b/je_api_testka/contract/openapi_compat.py @@ -0,0 +1,152 @@ +""" +Bidirectional contract check: a consumer contract against the provider's OpenAPI document. + +The provider publishes an OpenAPI document (kept honest by its own tests, e.g. +``apitestka openapi`` over its test run); the consumer publishes a contract. +This module checks that every interaction the consumer relies on is something +the provider's document allows, without running the provider: + +* an operation exists for the method and path (path templates such as + ``/items/{id}`` match, and the path of the first server URL is ignored); +* every query parameter the consumer sends is declared, and every required one is sent; +* a JSON request body satisfies the declared request body schema; +* the expected status code is declared (exactly, as ``2XX`` or as ``default``); +* the expected response body satisfies that response's JSON schema. +""" +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any, List, Mapping, Optional, Tuple +from urllib.parse import parse_qsl + +from je_api_testka.contract.openapi_schema import resolve_ref, schema_errors +from je_api_testka.spec.path_templates import match_path_template, server_base_path, strip_base_path +from je_api_testka.utils.exception.exceptions import APIContractException + +_JSON_MEDIA_SUFFIX: str = "json" + + +@dataclass +class CompatibilityReport: + """Problems found per interaction; empty ``problems`` means the contract fits the document.""" + + consumer: str + provider: str + checked: int = 0 + problems: List[Tuple[str, str]] = field(default_factory=list) + + @property + def ok(self) -> bool: + """True when at least one interaction was checked and none had a problem.""" + return self.checked > 0 and not self.problems + + def to_dict(self) -> dict: + """JSON-ready form: ``ok``, counts and ``{interaction, problem}`` pairs.""" + return {"ok": self.ok, "consumer": self.consumer, "provider": self.provider, "checked": self.checked, + "problems": [{"interaction": name, "problem": problem} for name, problem in self.problems]} + + def render_text(self) -> str: + """Plain-text summary with one line per problem.""" + lines = [f"Checking {self.consumer} contract against the {self.provider} OpenAPI document"] + lines.extend(f" [FAIL] {name}: {problem}" for name, problem in self.problems) + failing = len({name for name, _problem in self.problems}) + lines.append(f"{self.checked - failing}/{self.checked} interactions compatible") + return "\n".join(lines) + + +def _json_schema(content: Any) -> Optional[Mapping[str, Any]]: + if not isinstance(content, Mapping): + return None + for media_type, media in content.items(): + if _JSON_MEDIA_SUFFIX in str(media_type) and isinstance(media, Mapping) and "schema" in media: + return media["schema"] + return None + + +def _declared_response(operation: Mapping[str, Any], status: int) -> Optional[Mapping[str, Any]]: + responses = operation.get("responses") or {} + for key in (str(status), f"{str(status)[0]}XX", f"{str(status)[0]}xx", "default"): + if key in responses: + return responses[key] or {} + return None + + +def _parameters(path_item: Mapping[str, Any], operation: Mapping[str, Any], spec: Mapping[str, Any]) -> List[dict]: + merged = {} + for parameter in [*path_item.get("parameters", []), *operation.get("parameters", [])]: + resolved = resolve_ref(parameter, spec) + merged[(resolved.get("name"), resolved.get("in"))] = dict(resolved) + return list(merged.values()) + + +def _query_problems(request: Mapping[str, Any], parameters: List[dict]) -> List[str]: + sent = {name for name, _value in parse_qsl(str(request.get("query") or ""), keep_blank_values=True)} + declared = {parameter["name"] for parameter in parameters if parameter.get("in") == "query"} + required = {parameter["name"] for parameter in parameters + if parameter.get("in") == "query" and parameter.get("required")} + problems = [f"query parameter '{name}' is not declared" for name in sorted(sent - declared)] + problems.extend(f"required query parameter '{name}' is not sent" for name in sorted(required - sent)) + return problems + + +def _body_problems(request: Mapping[str, Any], operation: Mapping[str, Any], spec: Mapping[str, Any]) -> List[str]: + request_body = resolve_ref(operation.get("requestBody") or {}, spec) + if "body" not in request: + return ["request body is required but not sent"] if request_body.get("required") else [] + if not request_body: + return ["request body is sent but the operation declares none"] + schema = _json_schema(request_body.get("content")) + if schema is None or not isinstance(request["body"], (dict, list)): + return [] + return [f"request body {error}" for error in schema_errors(request["body"], schema, spec, "$.body")] + + +def _response_problems(response: Mapping[str, Any], operation: Mapping[str, Any], + spec: Mapping[str, Any]) -> List[str]: + declared = _declared_response(operation, response["status"]) + if declared is None: + return [f"status {response['status']} is not declared"] + declared = resolve_ref(declared, spec) + schema = _json_schema(declared.get("content")) + if "body" not in response or schema is None: + return [] + return [f"response body {error}" for error in schema_errors(response["body"], schema, spec, "$.body")] + + +def interaction_problems(interaction: Mapping[str, Any], spec: Mapping[str, Any]) -> List[str]: + """Return why ``interaction`` is not allowed by ``spec`` (empty when it is).""" + request = interaction["request"] + path = strip_base_path(str(request["path"]), server_base_path(spec)) + paths = spec.get("paths") or {} + template = match_path_template(path, paths) + method = str(request["method"]).lower() + if template is None or method not in (paths[template] or {}): + return [f"no operation for {method.upper()} {path}"] + path_item = paths[template] + operation = path_item[method] + problems = _query_problems(request, _parameters(path_item, operation, spec)) + problems.extend(_body_problems(request, operation, spec)) + problems.extend(_response_problems(interaction["response"], operation, spec)) + return problems + + +def check_pact_against_openapi(pact: Mapping[str, Any], spec: Mapping[str, Any]) -> CompatibilityReport: + """Check every interaction of ``pact`` against the provider's OpenAPI document ``spec``.""" + report = CompatibilityReport(pact["consumer"]["name"], pact["provider"]["name"]) + for interaction in pact["interactions"]: + report.checked += 1 + name = str(interaction["description"]) + report.problems.extend((name, problem) for problem in interaction_problems(interaction, spec)) + return report + + +def assert_pact_compatible(pact: Mapping[str, Any], spec: Mapping[str, Any]) -> dict: + """ + Check ``pact`` against ``spec`` and return the report as a dict. + + :raises APIContractException: with the text report when any interaction does not fit. + """ + report = check_pact_against_openapi(pact, spec) + if not report.ok: + raise APIContractException(report.render_text()) + return report.to_dict() diff --git a/je_api_testka/contract/openapi_schema.py b/je_api_testka/contract/openapi_schema.py new file mode 100644 index 0000000..59a9732 --- /dev/null +++ b/je_api_testka/contract/openapi_schema.py @@ -0,0 +1,80 @@ +""" +Check a JSON value against an OpenAPI 3.0 / 3.1 schema, without extra dependencies. + +Covers what API descriptions commonly use: local ``$ref`` (``#/...``), ``type`` +(a string or, in 3.1, a list), ``nullable`` (3.0), ``enum``, ``properties``, +``required``, ``additionalProperties: false``, ``items``, ``allOf``, ``anyOf`` +and ``oneOf`` (treated like ``anyOf``). Unknown keywords are ignored, so the +check never rejects what a full validator would accept for those keywords. +""" +from __future__ import annotations + +from typing import Any, List, Mapping + +from je_api_testka.contract.matching import json_type + +_MAX_REF_DEPTH: int = 32 + + +def resolve_ref(schema: Mapping[str, Any], spec: Mapping[str, Any], depth: int = 0) -> Mapping[str, Any]: + """Follow a local ``$ref`` (``#/components/...``) in ``spec``; other nodes come back unchanged.""" + ref = schema.get("$ref") + if not isinstance(ref, str) or not ref.startswith("#/") or depth > _MAX_REF_DEPTH: + return schema + node: Any = spec + for part in ref[2:].split("/"): + node = node.get(part.replace("~1", "/").replace("~0", "~"), {}) if isinstance(node, Mapping) else {} + return resolve_ref(node, spec, depth + 1) if isinstance(node, Mapping) else {} + + +def _type_matches(value: Any, schema: Mapping[str, Any]) -> bool: + declared = schema.get("type") + if declared is None: + return True + allowed = set(declared) if isinstance(declared, list) else {declared} + if schema.get("nullable"): + allowed.add("null") + actual = json_type(value) + if actual == "number" and isinstance(value, int): + return bool(allowed & {"number", "integer"}) + return actual in allowed + + +def _object_errors(value: dict, schema: Mapping[str, Any], spec: Mapping[str, Any], path: str) -> List[str]: + errors = [f"{path}.{name}: required" for name in schema.get("required", []) if name not in value] + properties = schema.get("properties") or {} + for name, item in value.items(): + if name in properties: + errors.extend(schema_errors(item, properties[name], spec, f"{path}.{name}")) + elif schema.get("additionalProperties") is False: + errors.append(f"{path}.{name}: not allowed") + return errors + + +def _combinator_errors(value: Any, schema: Mapping[str, Any], spec: Mapping[str, Any], path: str) -> List[str]: + errors: List[str] = [] + for part in schema.get("allOf", []): + errors.extend(schema_errors(value, part, spec, path)) + for keyword in ("anyOf", "oneOf"): + options = schema.get(keyword) + if options and all(schema_errors(value, option, spec, path) for option in options): + errors.append(f"{path}: matches none of {keyword}") + return errors + + +def schema_errors(value: Any, schema: Mapping[str, Any], spec: Mapping[str, Any], path: str = "$") -> List[str]: + """Return why ``value`` does not satisfy ``schema`` (empty when it does); ``spec`` resolves ``$ref``.""" + schema = resolve_ref(schema, spec) + if value is None and schema.get("nullable"): + return [] + if not _type_matches(value, schema): + return [f"{path}: expected type {schema.get('type')}, got {json_type(value)}"] + if "enum" in schema and value not in schema["enum"]: + return [f"{path}: {value!r} is not one of {schema['enum']}"] + errors = _combinator_errors(value, schema, spec, path) + if isinstance(value, dict): + errors.extend(_object_errors(value, schema, spec, path)) + elif isinstance(value, list) and isinstance(schema.get("items"), Mapping): + for index, item in enumerate(value): + errors.extend(schema_errors(item, schema["items"], spec, f"{path}[{index}]")) + return errors diff --git a/je_api_testka/contract/pact.py b/je_api_testka/contract/pact.py new file mode 100644 index 0000000..c15f69a --- /dev/null +++ b/je_api_testka/contract/pact.py @@ -0,0 +1,179 @@ +""" +Consumer contracts in the Pact specification v2 file format. + +A contract lists the interactions a consumer relies on: the request it sends +and the response it expects, plus ``matchingRules`` that loosen exact equality +(``type`` or ``regex``). Files written here can go to a Pact Broker or Pactflow +as they are. + +Contracts are usually recorded from the consumer's own test run +(:func:`pact_from_records`). By default the response body gets a ``type`` rule +on ``$.body``, so the provider must return the same structure and value types, +not the same values. +""" +from __future__ import annotations + +import json +from pathlib import Path +from typing import Iterable, Mapping, Optional +from urllib.parse import urlparse + +from je_api_testka.spec.path_templates import strip_base_path +from je_api_testka.spec.records_to_openapi import decode_body +from je_api_testka.utils.exception.exceptions import APIContractException + +PACT_SPECIFICATION_VERSION: str = "2.0.0" +BODY_RULE_TYPE: str = "type" +BODY_RULE_EQUALITY: str = "equality" +_BODY_RULES = (BODY_RULE_TYPE, BODY_RULE_EQUALITY) +_CONTENT_TYPE: str = "Content-Type" + + +def new_pact(consumer: str, provider: str) -> dict: + """Return an empty Pact v2 document for ``consumer`` and ``provider``.""" + return { + "consumer": {"name": consumer}, + "provider": {"name": provider}, + "interactions": [], + "metadata": {"pactSpecification": {"version": PACT_SPECIFICATION_VERSION}}, + } + + +def add_interaction(pact: dict, description: str, request: Mapping[str, object], response: Mapping[str, object], + provider_state: Optional[str] = None, + matching_rules: Optional[Mapping[str, Mapping[str, object]]] = None) -> dict: + """ + Append one interaction to ``pact`` and return it. + + ``request`` needs ``method`` and ``path`` (``query`` as a query string, ``headers``, ``body`` optional); + ``response`` needs ``status`` (``headers``, ``body`` optional). ``matching_rules`` are + Pact v2 rules such as ``{"$.body.id": {"match": "type"}}``. + + :raises APIContractException: on a missing field or a description already in the contract. + """ + if not request.get("method") or not str(request.get("path", "")).startswith("/"): + raise APIContractException("an interaction request needs a method and a path starting with '/'") + if not isinstance(response.get("status"), int): + raise APIContractException("an interaction response needs an integer status") + if any(existing["description"] == description for existing in pact["interactions"]): + raise APIContractException(f"duplicate interaction description: {description!r}") + interaction: dict = {"description": description} + if provider_state: + interaction["providerState"] = provider_state + interaction["request"] = {**request, "method": str(request["method"]).upper()} + interaction["response"] = dict(response) + if matching_rules: + interaction["response"]["matchingRules"] = dict(matching_rules) + pact["interactions"].append(interaction) + return interaction + + +def _content_type(headers: object) -> Optional[str]: + if not isinstance(headers, Mapping): + return None + for name, value in headers.items(): + if str(name).lower() == _CONTENT_TYPE.lower(): + return str(value) + return None + + +def _request_part(record: Mapping[str, object], base_path: str) -> dict: + parsed = urlparse(str(record["request_url"])) + request: dict = {"method": str(record.get("request_method") or "GET").upper(), + "path": strip_base_path(parsed.path or "/", base_path)} + if parsed.query: + request["query"] = parsed.query + body = decode_body(record.get("request_body")) + if body is not None: + request["body"] = body[1] + return request + + +def _response_part(record: Mapping[str, object]) -> dict: + response: dict = {"status": int(str(record.get("status_code", 200)))} + content_type = _content_type(record.get("headers")) + body = decode_body(record.get("text")) + if content_type: + response["headers"] = {_CONTENT_TYPE: content_type} + if body is not None: + response["body"] = body[1] + return response + + +def interaction_from_record(record: Mapping[str, object], base_path: str = "", + body_rule: str = BODY_RULE_TYPE) -> dict: + """ + Turn one success record into an interaction (not yet added to a contract). + + ``base_path`` is removed from the request path, e.g. ``/api/v1`` when the provider is + verified at ``https://host/api/v1``. ``body_rule`` is ``"type"`` (the default) or ``"equality"``. + """ + if body_rule not in _BODY_RULES: + raise APIContractException(f"body_rule must be one of {_BODY_RULES}, not {body_rule!r}") + request = _request_part(record, base_path) + response = _response_part(record) + query = f"?{request['query']}" if "query" in request else "" + interaction = { + "description": f"{request['method']} {request['path']}{query} -> {response['status']}", + "request": request, + "response": response, + } + if body_rule == BODY_RULE_TYPE and "body" in response: + response["matchingRules"] = {"$.body": {"match": "type"}} + return interaction + + +def pact_from_records(records: Iterable[Mapping[str, object]], consumer: str, provider: str, + base_path: str = "", body_rule: str = BODY_RULE_TYPE) -> dict: + """ + Build a contract from success records; records without a ``request_url`` are skipped. + + Records that yield the same description (method, path, query and status) are kept once. + """ + pact = new_pact(consumer, provider) + for record in records: + if not record.get("request_url"): + continue + interaction = interaction_from_record(record, base_path=base_path, body_rule=body_rule) + if all(existing["description"] != interaction["description"] for existing in pact["interactions"]): + pact["interactions"].append(interaction) + return pact + + +def write_pact(pact: Mapping[str, object], path: str) -> str: + """Write ``pact`` to ``path`` as UTF-8 JSON (creating parent folders) and return the path.""" + target = Path(path) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(json.dumps(pact, indent=2, ensure_ascii=False), encoding="utf-8") + return str(target) + + +def _check_interaction(interaction: object, index: int) -> None: + if not isinstance(interaction, dict) or not isinstance(interaction.get("description"), str): + raise APIContractException(f"interaction {index} needs a description") + request = interaction.get("request") + response = interaction.get("response") + if not isinstance(request, dict) or not request.get("method") or not isinstance(request.get("path"), str): + raise APIContractException(f"interaction {index} needs request.method and request.path") + if not isinstance(response, dict) or not isinstance(response.get("status"), int): + raise APIContractException(f"interaction {index} needs an integer response.status") + + +def read_pact(path: str) -> dict: + """ + Read and validate a contract file. + + :raises APIContractException: when the file is not JSON or misses consumer, provider or interaction fields. + """ + try: + pact = json.loads(Path(path).read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as error: + raise APIContractException(f"{path}: cannot read contract: {error!r}") from error + if not isinstance(pact, dict) or not isinstance(pact.get("interactions"), list): + raise APIContractException(f"{path}: a contract is an object with an interactions list") + for party in ("consumer", "provider"): + if not isinstance(pact.get(party), dict) or not pact[party].get("name"): + raise APIContractException(f"{path}: missing {party}.name") + for index, interaction in enumerate(pact["interactions"]): + _check_interaction(interaction, index) + return pact diff --git a/je_api_testka/contract/verifier.py b/je_api_testka/contract/verifier.py new file mode 100644 index 0000000..391f72b --- /dev/null +++ b/je_api_testka/contract/verifier.py @@ -0,0 +1,162 @@ +""" +Provider verification: replay a consumer contract against a running provider. + +Each interaction's request is sent to ``base_url`` and the response is checked +with :mod:`je_api_testka.contract.matching`: the status must be equal, the +contract's headers must be present, and the body must satisfy the matching +rules. When an interaction names a ``providerState``, the provider is first put +into that state through a Python callable or a setup URL, which receives +``{"consumer": ..., "state": ...}`` as a JSON POST (the Pact convention). +""" +from __future__ import annotations + +import json +from dataclasses import dataclass, field +from typing import Callable, List, Mapping, Optional + +import requests + +from je_api_testka.contract.matching import body_mismatches, header_mismatches +from je_api_testka.utils.exception.exceptions import APIContractException +from je_api_testka.utils.logging.loggin_instance import apitestka_logger + +DEFAULT_VERIFY_TIMEOUT_SECONDS: float = 10.0 + +StateHandler = Callable[[str], None] + + +@dataclass +class InteractionResult: + """Outcome of one interaction.""" + + description: str + mismatches: List[str] = field(default_factory=list) + + @property + def ok(self) -> bool: + """True when the provider satisfied the interaction.""" + return not self.mismatches + + +@dataclass +class VerificationReport: + """Outcome of verifying one contract against one provider.""" + + consumer: str + provider: str + results: List[InteractionResult] = field(default_factory=list) + + @property + def ok(self) -> bool: + """True when every interaction passed (and there was at least one).""" + return bool(self.results) and all(result.ok for result in self.results) + + def to_dict(self) -> dict: + """JSON-ready form: ``ok``, the parties, and each interaction's mismatches.""" + return { + "ok": self.ok, + "consumer": self.consumer, + "provider": self.provider, + "interactions": [{"description": result.description, "ok": result.ok, "mismatches": result.mismatches} + for result in self.results], + } + + def render_text(self) -> str: + """Plain-text summary, one line per interaction and one indented line per mismatch.""" + lines = [f"Verifying {self.consumer} -> {self.provider}"] + for result in self.results: + lines.append(f" [{'PASS' if result.ok else 'FAIL'}] {result.description}") + lines.extend(f" {mismatch}" for mismatch in result.mismatches) + passed = sum(result.ok for result in self.results) + lines.append(f"{passed}/{len(self.results)} interactions passed") + return "\n".join(lines) + + +@dataclass +class ProviderTarget: + """Where and how to reach the provider under verification.""" + + base_url: str + state_handler: Optional[StateHandler] = None + provider_states_url: Optional[str] = None + timeout: float = DEFAULT_VERIFY_TIMEOUT_SECONDS + + +def _set_up_state(target: ProviderTarget, consumer: str, state: str) -> None: + if target.state_handler is not None: + target.state_handler(state) + if target.provider_states_url: + response = requests.post(target.provider_states_url, json={"consumer": consumer, "state": state}, + timeout=target.timeout) + response.raise_for_status() + + +def _send(target: ProviderTarget, request: Mapping[str, object]) -> requests.Response: + url = target.base_url.rstrip("/") + str(request["path"]) + if request.get("query"): + url += f"?{request['query']}" + body = request.get("body") + return requests.request( + str(request["method"]), url, headers=dict(request.get("headers") or {}), + json=body if isinstance(body, (dict, list)) else None, + data=None if body is None or isinstance(body, (dict, list)) else str(body), + timeout=target.timeout, allow_redirects=False, + ) + + +def _actual_body(response: requests.Response, expected: object) -> object: + if isinstance(expected, str): + return response.text + try: + return response.json() + except (json.JSONDecodeError, ValueError): + return response.text + + +def _check_response(expected: Mapping[str, object], response: requests.Response) -> List[str]: + rules = expected.get("matchingRules") or {} + mismatches: List[str] = [] + if response.status_code != expected["status"]: + mismatches.append(f"status: expected {expected['status']}, got {response.status_code}") + mismatches.extend(header_mismatches(expected.get("headers") or {}, response.headers, rules)) + if "body" in expected: + mismatches.extend(body_mismatches(expected["body"], _actual_body(response, expected["body"]), rules)) + return mismatches + + +def verify_interaction(interaction: Mapping[str, object], target: ProviderTarget, + consumer: str = "") -> InteractionResult: + """Replay one interaction against ``target`` and return its result.""" + result = InteractionResult(str(interaction["description"])) + try: + if interaction.get("providerState"): + _set_up_state(target, consumer, str(interaction["providerState"])) + response = _send(target, interaction["request"]) + except requests.RequestException as error: + result.mismatches.append(f"request failed: {error!r}") + return result + result.mismatches.extend(_check_response(interaction["response"], response)) + return result + + +def verify_pact(pact: Mapping[str, object], target: ProviderTarget) -> VerificationReport: + """Replay every interaction of ``pact`` against ``target``; the report says which ones failed and why.""" + consumer = pact["consumer"]["name"] + report = VerificationReport(consumer, pact["provider"]["name"]) + for interaction in pact["interactions"]: + result = verify_interaction(interaction, target, consumer) + apitestka_logger.info(f"contract verify {result.description}: {'ok' if result.ok else result.mismatches}") + report.results.append(result) + return report + + +def assert_pact_verified(pact: Mapping[str, object], target: ProviderTarget) -> dict: + """ + Verify ``pact`` and return the report as a dict. + + :raises APIContractException: with the text report when any interaction fails. + """ + report = verify_pact(pact, target) + if not report.ok: + raise APIContractException(report.render_text()) + return report.to_dict() diff --git a/je_api_testka/spec/openapi_export.py b/je_api_testka/spec/openapi_export.py index ea0aab0..58b4381 100644 --- a/je_api_testka/spec/openapi_export.py +++ b/je_api_testka/spec/openapi_export.py @@ -67,13 +67,18 @@ def load_report_records(path: str) -> List[dict]: return records -def build_openapi(report_paths: Optional[Sequence[str]] = None, title: str = DEFAULT_SPEC_TITLE, - version: str = DEFAULT_SPEC_VERSION) -> dict: - """Return an OpenAPI document inferred from the current test record plus each saved report.""" +def collect_records(report_paths: Optional[Sequence[str]] = None) -> List[dict]: + """Return the current test record's successes followed by the records of each saved report.""" records = list(test_record_instance.test_record_list) for path in report_paths or (): records.extend(load_report_records(path)) - return records_to_openapi(records, title=title, version=version) + return records + + +def build_openapi(report_paths: Optional[Sequence[str]] = None, title: str = DEFAULT_SPEC_TITLE, + version: str = DEFAULT_SPEC_VERSION) -> dict: + """Return an OpenAPI document inferred from the current test record plus each saved report.""" + return records_to_openapi(collect_records(report_paths), title=title, version=version) def export_openapi(output_path: str, report_paths: Optional[Sequence[str]] = None, diff --git a/je_api_testka/spec/path_templates.py b/je_api_testka/spec/path_templates.py new file mode 100644 index 0000000..779cda9 --- /dev/null +++ b/je_api_testka/spec/path_templates.py @@ -0,0 +1,47 @@ +""" +Match concrete request paths against OpenAPI path templates. + +``/items/{id}`` matches ``/items/42``: each ``{name}`` segment matches one +non-empty path segment. When several templates match, the one with the most +literal segments wins, so ``/items/latest`` beats ``/items/{id}`` for +``/items/latest``, as OpenAPI requires. +""" +from __future__ import annotations + +import re +from typing import Iterable, Optional, Pattern +from urllib.parse import urlparse + +_PARAMETER = re.compile(r"\{[^/{}]+\}") + + +def template_regex(template: str) -> Pattern[str]: + """Return a regex that matches the concrete paths of ``template``.""" + parts = _PARAMETER.split(template) + return re.compile("^" + "[^/]+".join(re.escape(part) for part in parts) + "$") + + +def _literal_segments(template: str) -> int: + return sum(1 for segment in template.split("/") if segment and not _PARAMETER.fullmatch(segment)) + + +def match_path_template(path: str, templates: Iterable[str]) -> Optional[str]: + """Return the template in ``templates`` that best matches ``path``, or ``None``.""" + matches = [template for template in templates if template_regex(template).match(path)] + if not matches: + return None + return max(matches, key=_literal_segments) + + +def server_base_path(spec: dict) -> str: + """Return the path part of the spec's first server URL without a trailing slash (``""`` if none).""" + servers = spec.get("servers") or [] + url = servers[0].get("url", "") if servers and isinstance(servers[0], dict) else "" + return urlparse(url).path.rstrip("/") + + +def strip_base_path(path: str, base_path: str) -> str: + """Remove ``base_path`` from the front of ``path`` when present (``/api/v1/items`` → ``/items``).""" + if base_path and (path == base_path or path.startswith(base_path + "/")): + return path[len(base_path):] or "/" + return path diff --git a/je_api_testka/spec/records_to_openapi.py b/je_api_testka/spec/records_to_openapi.py index ab4309e..5553a44 100644 --- a/je_api_testka/spec/records_to_openapi.py +++ b/je_api_testka/spec/records_to_openapi.py @@ -23,8 +23,12 @@ TEXT_MEDIA_TYPE: str = "text/plain" -def _media_sample(raw: Any) -> Optional[Tuple[str, Any]]: - """Return ``(media type, sample)`` for a body, or ``None`` when it is empty.""" +def decode_body(raw: Any) -> Optional[Tuple[str, Any]]: + """ + Return ``(media type, value)`` for a recorded body, or ``None`` when it is empty or not UTF-8. + + JSON text becomes ``("application/json", parsed value)``; any other text ``("text/plain", text)``. + """ if isinstance(raw, (bytes, bytearray)): try: raw = raw.decode("utf-8") @@ -40,7 +44,7 @@ def _media_sample(raw: Any) -> Optional[Tuple[str, Any]]: def _content(raw: Any) -> Optional[dict]: """Return an OpenAPI ``content`` map for a body, or ``None`` when it is empty.""" - sample = _media_sample(raw) + sample = decode_body(raw) if sample is None: return None media_type, value = sample diff --git a/je_api_testka/utils/exception/exceptions.py b/je_api_testka/utils/exception/exceptions.py index c9ba3a8..f2d4bf5 100644 --- a/je_api_testka/utils/exception/exceptions.py +++ b/je_api_testka/utils/exception/exceptions.py @@ -156,3 +156,7 @@ class CallbackExecutorException(APITesterException): class MockServerException(APITesterException): pass + + +class APIContractException(APITesterException): + """A consumer contract is malformed, or it does not hold against a provider or its OpenAPI document.""" diff --git a/je_api_testka/utils/executor/action_executor.py b/je_api_testka/utils/executor/action_executor.py index e0f6ac5..6f5b681 100644 --- a/je_api_testka/utils/executor/action_executor.py +++ b/je_api_testka/utils/executor/action_executor.py @@ -6,6 +6,7 @@ from je_api_testka.ai.fake_data_generator import generate_fake_payload from je_api_testka.ai.test_generator import generate_tests_from_openapi from je_api_testka.connection.cassette import Cassette, CassetteRecord +from je_api_testka.contract.commands import check_contract_against_openapi, verify_contract, write_contract from je_api_testka.data.env_profile import load_env_profile from je_api_testka.data.faker_helpers import fake_email, fake_uuid, fake_word from je_api_testka.data.template_render import render_template @@ -153,6 +154,10 @@ def __init__(self): "AT_records_to_openapi": records_to_openapi, "AT_export_openapi": export_openapi, "AT_openapi_changelog": openapi_changelog, + # Consumer contracts (Pact v2 files) + "AT_write_contract": write_contract, + "AT_verify_contract": verify_contract, + "AT_check_contract_against_openapi": check_contract_against_openapi, # AI integrations "AT_classify_failures": classify_failures, "AT_generate_fake_payload": generate_fake_payload, diff --git a/progress.md b/progress.md index 23135c3..90561f9 100644 --- a/progress.md +++ b/progress.md @@ -8,7 +8,6 @@ Cross-repo and workspace items live in `D:\Codes\progress.md` (relevant here: X- - **#3** [DECIDE] The PyPI classifier still says `Development Status :: 2 - Pre-Alpha`. - **#6** GUI redesign: restructure and restyle. `gui/main_widget.py` is one 605-line widget (over the 500-line limit) with six flat tabs. It should become a sidebar-navigated window with a request/response split view, and history and environment panels docked (`gui/history_panel.py`, `gui/env_manager_model.py` exist but are not wired in). It needs a refreshed theme (`gui/main_window.py:21` hard-codes `dark_amber.xml`), pages for the newer features (#8–#13), and all four language wrappers (English, Traditional Chinese, Simplified Chinese, Japanese). `APITestkaWidget` stays the embeddable entry point (PyBreeze contract, `architecture.md` §6). -- **#9** Pact-style bidirectional contract testing. `diff/contract_diff.py` only compares two OpenAPI documents. Missing: a consumer contract recorded from the consumer's tests (interactions plus matchers), provider verification replaying it against a running provider, and a cross-check of the consumer contract against the provider's OpenAPI. - **#10** Reference AI backend on the Anthropic API. `ai/backend.py:1` ships only `NoOpAIBackend` and `StaticAIBackend`. Missing: an optional `ai` extra, an `AnthropicAIBackend` (key from the environment, never stored), and a way to select it (CLI, executor). - **#11** Load-test bridge to LoadDensity. No way to turn APITestka actions into a LoadDensity run. It needs a converter to LoadDensity's action JSON (`LD_start_test` with `tasks`) and a runner that uses LoadDensity's `--execute_file` CLI. That is a new cross-project contract, so it also needs §6 in both repos and the root `AI_CONTEXT.md`. - **#12** Response-time trends and anomaly detection. `utils/generate_report/trend_store.py:13` stores only run totals (success, failure, average elapsed). Missing: per-endpoint latency history, a baseline (median/MAD or EWMA), anomaly flags, and a trend report. diff --git a/test/test_contract/__init__.py b/test/test_contract/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/test/test_contract/conftest.py b/test/test_contract/conftest.py new file mode 100644 index 0000000..f262ea8 --- /dev/null +++ b/test/test_contract/conftest.py @@ -0,0 +1,51 @@ +"""A small provider app on a free port for contract verification tests.""" +from __future__ import annotations + +import socket +import threading + +import pytest +from flask import Flask, jsonify, request +from werkzeug.serving import make_server + + +def _provider_app(states: list) -> Flask: + # Test-only provider on loopback; no form auth surface, CSRF not applicable. + app = Flask(__name__) # NOSONAR S4502 + + @app.get("/items/") + def get_item(item_id: int): + return jsonify({"id": item_id, "name": f"item {item_id}", "tags": ["a", "b"]}) + + @app.post("/items") + def create_item(): + return jsonify({"id": 99, **(request.get_json(silent=True) or {})}), 201 + + @app.get("/ping") + def ping(): + return "pong", 200, {"Content-Type": "text/plain; charset=utf-8"} + + @app.post("/_states") + def set_state(): + states.append(request.get_json()) + return "", 204 + + return app + + +@pytest.fixture +def provider(): + """Yield ``(base_url, states)``; ``states`` collects provider-state setup calls.""" + states: list = [] + original_getfqdn = socket.getfqdn + socket.getfqdn = lambda name="": name or "127.0.0.1" # avoid reverse DNS on some CI runners + try: + server = make_server("127.0.0.1", 0, _provider_app(states), threaded=True) + finally: + socket.getfqdn = original_getfqdn + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + # Local in-process provider on loopback; HTTPS not applicable. + yield f"http://127.0.0.1:{server.server_port}", states # NOSONAR S5332 + server.shutdown() + thread.join(5) diff --git a/test/test_contract/test_commands.py b/test/test_contract/test_commands.py new file mode 100644 index 0000000..af73ed6 --- /dev/null +++ b/test/test_contract/test_commands.py @@ -0,0 +1,123 @@ +"""Tests for the file-level contract steps, their CLI and the executor commands.""" +from __future__ import annotations + +import json + +import pytest + +from je_api_testka.cli.cli_main import main +from je_api_testka.contract.commands import ( + check_contract_against_openapi, + read_openapi, + verify_contract, + write_contract, +) +from je_api_testka.contract.pact import read_pact +from je_api_testka.spec.openapi_export import export_openapi +from je_api_testka.utils.exception.exceptions import APIContractException +from je_api_testka.utils.executor.action_executor import execute_action +from je_api_testka.utils.test_record.test_record_class import test_record_instance + + +def _call_provider(base_url: str) -> None: + execute_action([ + ["AT_test_api_method", {"http_method": "get", "test_url": f"{base_url}/items/3", "timeout": 5}], + ["AT_test_api_method", {"http_method": "post", "test_url": f"{base_url}/items", + "json": {"name": "n"}, "timeout": 5}], + ]) + + +def test_bidirectional_flow_end_to_end(provider, tmp_path): + base_url, _states = provider + # Consumer side: its tests talk to the provider (or a mock) and the run becomes the contract. + _call_provider(base_url) + contract = write_contract(str(tmp_path / "web-shop.json"), "web", "shop") + # Provider side: its own test run becomes the OpenAPI document it publishes. + openapi = export_openapi(str(tmp_path / "openapi.json")) + test_record_instance.clean_record() + + assert [i["description"] for i in read_pact(contract)["interactions"]] == [ + "GET /items/3 -> 200", "POST /items -> 201"] + assert check_contract_against_openapi(contract, openapi)["ok"] is True + assert verify_contract(contract, base_url, timeout=5)["ok"] is True + + +def test_write_contract_needs_records(tmp_path): + with pytest.raises(APIContractException): + write_contract(str(tmp_path / "c.json"), "web", "shop") + + +def test_verify_contract_raises_with_the_report(provider, tmp_path): + base_url, _states = provider + test_record_instance.test_record_list.append({ + "request_url": f"{base_url}/items/1", "request_method": "GET", "status_code": 200, "text": '{"id": "x"}'}) + contract = write_contract(str(tmp_path / "c.json"), "web", "shop") + with pytest.raises(APIContractException, match="expected a string"): + verify_contract(contract, base_url, timeout=5) + + +def test_read_openapi_errors(tmp_path): + (tmp_path / "list.json").write_text("[]", encoding="utf-8") + for name in ("list.json", "missing.json"): + with pytest.raises(APIContractException): + read_openapi(str(tmp_path / name)) + + +def test_executor_commands(provider, tmp_path): + base_url, _states = provider + _call_provider(base_url) + contract = str(tmp_path / "c.json") + openapi = str(tmp_path / "o.json") + record = execute_action([ + ["AT_write_contract", {"output_path": contract, "consumer": "web", "provider": "shop"}], + ["AT_export_openapi", {"output_path": openapi}], + ["AT_check_contract_against_openapi", {"contract_path": contract, "openapi_path": openapi}], + ["AT_verify_contract", {"contract_path": contract, "base_url": base_url, "timeout": 5}], + ]) + values = list(record.values()) + assert values[2]["ok"] is True and values[3]["ok"] is True + + +def _report(tmp_path, base_url: str) -> str: + path = tmp_path / "run_success.json" + path.write_text(json.dumps({"Success_Test1": { + "request_url": f"{base_url}/items/1", "request_method": "GET", "status_code": "200", + "text": '{"id": 1, "name": "a"}', "request_body": "None", + }}), encoding="utf-8") + return str(path) + + +def test_cli_record_verify_compare(provider, tmp_path, capsys): + base_url, _states = provider + contract = str(tmp_path / "pacts" / "web-shop.json") + assert main(["contract", "record", "--report", _report(tmp_path, base_url), + "--consumer", "web", "--provider", "shop", "-o", contract]) == 0 + assert main(["contract", "verify", contract, "--base-url", base_url, "--json"]) == 0 + assert json.loads(capsys.readouterr().out.split("\n", 1)[1])["ok"] is True + + spec = tmp_path / "openapi.json" + spec.write_text(json.dumps({"paths": {"/items/{id}": {"get": {"responses": {"200": {}}}}}}), encoding="utf-8") + assert main(["contract", "compare", contract, str(spec)]) == 0 + spec.write_text(json.dumps({"paths": {}}), encoding="utf-8") + assert main(["contract", "compare", contract, str(spec)]) == 1 + assert "no operation for GET /items/1" in capsys.readouterr().out + + +def test_cli_verify_failure_exit_code(provider, tmp_path): + base_url, _states = provider + contract = tmp_path / "c.json" + contract.write_text(json.dumps({ + "consumer": {"name": "web"}, "provider": {"name": "shop"}, + "interactions": [{"description": "x", "request": {"method": "GET", "path": "/items/1"}, + "response": {"status": 404}}]}), encoding="utf-8") + assert main(["contract", "verify", str(contract), "--base-url", base_url]) == 1 + + +def test_cli_usage_errors(tmp_path): + assert main(["contract", "record", "--consumer", "a", "--provider", "b", "-o", str(tmp_path / "c.json")]) == 2 + empty = tmp_path / "empty.json" + empty.write_text("{}", encoding="utf-8") + assert main(["contract", "record", "--report", str(empty), "--consumer", "a", "--provider", "b", + "-o", str(tmp_path / "c.json")]) == 1 + assert main(["contract", "verify", str(tmp_path / "missing.json"), "--base-url", "http://127.0.0.1:9"]) == 2 + assert main(["contract", "compare", str(tmp_path / "missing.json"), str(empty)]) == 2 diff --git a/test/test_contract/test_matching.py b/test/test_contract/test_matching.py new file mode 100644 index 0000000..cb30529 --- /dev/null +++ b/test/test_contract/test_matching.py @@ -0,0 +1,72 @@ +"""Tests for Pact v2 matching rules.""" +from __future__ import annotations + +from je_api_testka.contract.matching import body_mismatches, header_mismatches, json_type + +TYPE = {"$.body": {"match": "type"}} + + +def test_equality_without_rules(): + assert body_mismatches({"id": 1}, {"id": 1, "extra": True}) == [] + assert body_mismatches({"id": 1}, {"id": 2}) == ["$.body.id: expected 1, got 2"] + assert body_mismatches({"id": 1}, {}) == ["$.body.id: missing"] + + +def test_true_is_not_one(): + assert body_mismatches({"flag": 1}, {"flag": True}) + + +def test_type_rule_cascades(): + assert body_mismatches({"id": 1, "name": "a", "price": 1.5}, {"id": 7, "name": "b", "price": 3}, TYPE) == [] + assert body_mismatches({"id": 1}, {"id": "7"}, TYPE) == ["$.body.id: expected a number, got string '7'"] + + +def test_type_rule_on_arrays_uses_first_element(): + expected = {"items": [{"id": 1}]} + assert body_mismatches(expected, {"items": [{"id": 2}, {"id": 3}]}, TYPE) == [] + assert body_mismatches(expected, {"items": []}, TYPE) == [] + assert body_mismatches(expected, {"items": [{"id": "x"}]}, TYPE) == [ + "$.body.items[0].id: expected a number, got string 'x'"] + + +def test_min_rule_on_array(): + rules = {"$.body.items": {"match": "type", "min": 1}} + assert body_mismatches({"items": [1]}, {"items": []}, rules) == ["$.body.items: expected at least 1 items, got 0"] + + +def test_equality_arrays_compare_length_and_items(): + assert body_mismatches([1, 2], [1, 2, 3]) == ["$.body: expected 2 items, got 3"] + assert body_mismatches([1, 2], [1, 3]) == ["$.body[1]: expected 2, got 3"] + + +def test_regex_and_wildcards_and_specific_rule_wins(): + rules = { + "$.body": {"match": "type"}, + "$.body.items[*].sku": {"match": "regex", "regex": "[A-Z]{3}-\\d+"}, + "$.body.code": {"match": "regex", "regex": "\\d{3}"}, + } + expected = {"code": "200", "items": [{"sku": "ABC-1"}]} + assert body_mismatches(expected, {"code": "404", "items": [{"sku": "XYZ-99"}]}, rules) == [] + assert body_mismatches(expected, {"code": "4O4", "items": [{"sku": "bad"}]}, rules) == [ + "$.body.code: '4O4' does not match /\\d{3}/", + "$.body.items[0].sku: 'bad' does not match /[A-Z]{3}-\\d+/", + ] + + +def test_structure_mismatches(): + assert body_mismatches({"a": 1}, [1]) == ["$.body: expected an object, got array"] + assert body_mismatches([1], {"a": 1}) == ["$.body: expected an array, got object"] + + +def test_headers(): + actual = {"content-type": "application/json; charset=utf-8", "X-Id": "abc-123"} + assert header_mismatches({"Content-Type": "application/json"}, actual) == [] + assert header_mismatches({"X-Id": "abc-124"}, actual) == ["$.headers.X-Id: expected 'abc-124', got 'abc-123'"] + assert header_mismatches({"X-Id": "x"}, actual, {"$.headers.X-Id": {"match": "regex", "regex": "abc-\\d+"}}) == [] + assert header_mismatches({"X-Missing": "x"}, actual) == ["$.headers.X-Missing: missing"] + assert header_mismatches({"Content-Type": "text/plain"}, actual) + + +def test_json_type_names(): + assert [json_type(v) for v in (True, 1, 1.5, "s", [], {}, None)] == [ + "boolean", "number", "number", "string", "array", "object", "null"] diff --git a/test/test_contract/test_openapi_compat.py b/test/test_contract/test_openapi_compat.py new file mode 100644 index 0000000..1ea4276 --- /dev/null +++ b/test/test_contract/test_openapi_compat.py @@ -0,0 +1,116 @@ +"""Tests for the bidirectional check of a contract against an OpenAPI document.""" +from __future__ import annotations + +import pytest + +from je_api_testka.contract.openapi_compat import ( + assert_pact_compatible, + check_pact_against_openapi, + interaction_problems, +) +from je_api_testka.contract.openapi_schema import schema_errors +from je_api_testka.contract.pact import add_interaction, new_pact +from je_api_testka.utils.exception.exceptions import APIContractException + +SPEC = { + "openapi": "3.0.3", + "servers": [{"url": "https://shop.invalid/api/v1"}], + "components": { + "schemas": {"Item": {"type": "object", "required": ["id", "name"], "properties": { + "id": {"type": "integer"}, "name": {"type": "string"}, "note": {"type": "string", "nullable": True}}}}, + "parameters": {"Full": {"name": "full", "in": "query", "schema": {"type": "string"}}}, + }, + "paths": { + "/items/{id}": { + "parameters": [{"$ref": "#/components/parameters/Full"}], + "get": {"responses": { + "200": {"content": {"application/json": {"schema": {"$ref": "#/components/schemas/Item"}}}}, + "4XX": {"description": "client error"}}}, + }, + "/items": {"post": { + "parameters": [{"name": "dry", "in": "query", "required": True, "schema": {"type": "string"}}], + "requestBody": {"required": True, "content": {"application/json": {"schema": { + "type": "object", "required": ["name"], "properties": {"name": {"type": "string"}}}}}}, + "responses": {"default": {"description": "any"}}}}, + }, +} + + +def _interaction(method="GET", path="/items/1", status=200, body=None, query=None, request_body=None): + request = {"method": method, "path": path} + if query: + request["query"] = query + if request_body is not None: + request["body"] = request_body + response = {"status": status} + if body is not None: + response["body"] = body + return {"description": f"{method} {path}", "request": request, "response": response} + + +def test_compatible_interaction_through_refs_and_templates(): + interaction = _interaction(query="full=1", body={"id": 1, "name": "a", "note": None}) + assert interaction_problems(interaction, SPEC) == [] + + +def test_server_base_path_is_ignored(): + assert interaction_problems(_interaction(path="/api/v1/items/1", body={"id": 1, "name": "a"}), SPEC) == [] + + +@pytest.mark.parametrize(("interaction", "problem"), [ + (_interaction(path="/orders/1"), "no operation for GET /orders/1"), + (_interaction(method="DELETE"), "no operation for DELETE /items/1"), + (_interaction(status=500), "status 500 is not declared"), + (_interaction(query="page=2"), "query parameter 'page' is not declared"), + (_interaction(body={"id": "1", "name": "a"}), "response body $.body.id: expected type integer, got string"), + (_interaction(body={"id": 1}), "response body $.body.name: required"), + (_interaction(method="POST", path="/items", request_body={"name": "a"}), + "required query parameter 'dry' is not sent"), + (_interaction(method="POST", path="/items", query="dry=1"), "request body is required but not sent"), + (_interaction(method="POST", path="/items", query="dry=1", request_body={"name": 3}), + "request body $.body.name: expected type string, got number"), + (_interaction(request_body={"x": 1}), "request body is sent but the operation declares none"), +]) +def test_problems(interaction, problem): + assert problem in interaction_problems(interaction, SPEC) + + +def test_status_ranges_and_default(): + assert interaction_problems(_interaction(status=404), SPEC) == [] + assert interaction_problems(_interaction(method="POST", path="/items", query="dry=1", + request_body={"name": "a"}, status=418), SPEC) == [] + + +def test_report_and_assert(): + pact = new_pact("web", "shop") + add_interaction(pact, "ok", {"method": "GET", "path": "/items/1"}, {"status": 200, "body": {"id": 1, "name": "a"}}) + add_interaction(pact, "bad", {"method": "GET", "path": "/nope"}, {"status": 200}) + report = check_pact_against_openapi(pact, SPEC) + assert not report.ok + assert report.to_dict()["problems"] == [{"interaction": "bad", "problem": "no operation for GET /nope"}] + assert "1/2 interactions compatible" in report.render_text() + with pytest.raises(APIContractException): + assert_pact_compatible(pact, SPEC) + pact["interactions"].pop() + assert assert_pact_compatible(pact, SPEC)["ok"] is True + + +def test_empty_contract_is_not_ok(): + assert not check_pact_against_openapi(new_pact("web", "shop"), SPEC).ok + + +@pytest.mark.parametrize(("value", "schema", "errors"), [ + (1, {"type": ["integer", "null"]}, 0), + (None, {"type": ["integer", "null"]}, 0), + (1.5, {"type": "integer"}, 1), + (True, {"type": "integer"}, 1), + ("b", {"enum": ["a"]}, 1), + ({"a": 1, "b": 2}, {"type": "object", "properties": {"a": {}}, "additionalProperties": False}, 1), + ([1, "x"], {"type": "array", "items": {"type": "integer"}}, 1), + ({"a": 1}, {"allOf": [{"required": ["a"]}, {"required": ["b"]}]}, 1), + ("x", {"anyOf": [{"type": "integer"}, {"type": "string"}]}, 0), + ([], {"oneOf": [{"type": "integer"}, {"type": "string"}]}, 1), + ({}, {"$ref": "#/components/schemas/Missing"}, 0), +]) +def test_schema_errors(value, schema, errors): + assert len(schema_errors(value, schema, SPEC)) == errors diff --git a/test/test_contract/test_pact.py b/test/test_contract/test_pact.py new file mode 100644 index 0000000..162298c --- /dev/null +++ b/test/test_contract/test_pact.py @@ -0,0 +1,98 @@ +"""Tests for Pact v2 contract files.""" +from __future__ import annotations + +import json + +import pytest + +from je_api_testka.contract.pact import ( + add_interaction, + interaction_from_record, + new_pact, + pact_from_records, + read_pact, + write_pact, +) +from je_api_testka.utils.exception.exceptions import APIContractException + + +def _record(url="https://api.invalid/api/v1/items/1?full=1", method="GET", status=200, + text='{"id": 1}', body=None, headers=None): + return {"request_url": url, "request_method": method, "status_code": status, "text": text, + "request_body": body, "headers": headers or {"content-type": "application/json"}} + + +def test_new_pact_is_pact_v2(): + pact = new_pact("web", "shop") + assert pact["consumer"] == {"name": "web"} + assert pact["provider"] == {"name": "shop"} + assert pact["metadata"]["pactSpecification"]["version"] == "2.0.0" + + +def test_interaction_from_record_with_type_rule(): + interaction = interaction_from_record(_record(), base_path="/api/v1") + assert interaction["description"] == "GET /items/1?full=1 -> 200" + assert interaction["request"] == {"method": "GET", "path": "/items/1", "query": "full=1"} + assert interaction["response"] == { + "status": 200, "headers": {"Content-Type": "application/json"}, "body": {"id": 1}, + "matchingRules": {"$.body": {"match": "type"}}, + } + + +def test_equality_rule_and_request_body(): + interaction = interaction_from_record( + _record(url="https://api.invalid/items", method="post", status=201, body=b'{"name": "a"}'), + body_rule="equality") + assert interaction["request"]["body"] == {"name": "a"} + assert "matchingRules" not in interaction["response"] + + +def test_unknown_body_rule_raises(): + with pytest.raises(APIContractException): + interaction_from_record(_record(), body_rule="loose") + + +def test_pact_from_records_skips_and_deduplicates(): + pact = pact_from_records([_record(), _record(), {"status_code": 200}, _record(status=404)], "web", "shop") + assert [i["description"] for i in pact["interactions"]] == [ + "GET /api/v1/items/1?full=1 -> 200", "GET /api/v1/items/1?full=1 -> 404"] + + +def test_add_interaction_validates(): + pact = new_pact("web", "shop") + add_interaction(pact, "get item", {"method": "get", "path": "/items/1"}, {"status": 200}, + provider_state="item 1 exists", matching_rules={"$.body.id": {"match": "type"}}) + interaction = pact["interactions"][0] + assert interaction["providerState"] == "item 1 exists" + assert interaction["request"]["method"] == "GET" + assert interaction["response"]["matchingRules"] == {"$.body.id": {"match": "type"}} + with pytest.raises(APIContractException): + add_interaction(pact, "get item", {"method": "GET", "path": "/x"}, {"status": 200}) + with pytest.raises(APIContractException): + add_interaction(pact, "no path", {"method": "GET", "path": "x"}, {"status": 200}) + with pytest.raises(APIContractException): + add_interaction(pact, "bad status", {"method": "GET", "path": "/x"}, {"status": "200"}) + + +def test_write_and_read_round_trip(tmp_path): + pact = pact_from_records([_record()], "web", "shop") + path = write_pact(pact, str(tmp_path / "pacts" / "web-shop.json")) + assert read_pact(path) == pact + + +@pytest.mark.parametrize("document", [ + [], {"interactions": []}, {"consumer": {"name": "a"}, "provider": {}, "interactions": []}, + {"consumer": {"name": "a"}, "provider": {"name": "b"}, "interactions": [{"description": "x"}]}, + {"consumer": {"name": "a"}, "provider": {"name": "b"}, "interactions": [ + {"description": "x", "request": {"method": "GET", "path": "/"}, "response": {"status": "200"}}]}, +]) +def test_read_pact_rejects_malformed(tmp_path, document): + path = tmp_path / "bad.json" + path.write_text(json.dumps(document), encoding="utf-8") + with pytest.raises(APIContractException): + read_pact(str(path)) + + +def test_read_pact_rejects_unreadable(tmp_path): + with pytest.raises(APIContractException): + read_pact(str(tmp_path / "missing.json")) diff --git a/test/test_contract/test_verifier.py b/test/test_contract/test_verifier.py new file mode 100644 index 0000000..b39d354 --- /dev/null +++ b/test/test_contract/test_verifier.py @@ -0,0 +1,70 @@ +"""Tests for provider verification against a live provider.""" +from __future__ import annotations + +import pytest + +from je_api_testka.contract.pact import add_interaction, new_pact +from je_api_testka.contract.verifier import ProviderTarget, assert_pact_verified, verify_pact +from je_api_testka.utils.exception.exceptions import APIContractException + +TYPE = {"$.body": {"match": "type"}} + + +def _pact() -> dict: + pact = new_pact("web", "shop") + add_interaction(pact, "get item", {"method": "GET", "path": "/items/1"}, + {"status": 200, "headers": {"Content-Type": "application/json"}, + "body": {"id": 5, "name": "x", "tags": ["t"]}}, matching_rules=TYPE) + add_interaction(pact, "create item", {"method": "POST", "path": "/items", "body": {"name": "n"}}, + {"status": 201, "body": {"id": 99, "name": "n"}}, provider_state="catalogue is empty") + add_interaction(pact, "ping", {"method": "GET", "path": "/ping"}, + {"status": 200, "headers": {"Content-Type": "text/plain"}, "body": "pong"}) + return pact + + +def test_passing_contract_with_state_handler(provider): + base_url, _states = provider + seen = [] + report = verify_pact(_pact(), ProviderTarget(base_url, state_handler=seen.append, timeout=5)) + assert [result.mismatches for result in report.results] == [[], [], []] + assert report.ok + assert seen == ["catalogue is empty"] + assert "3/3 interactions passed" in report.render_text() + + +def test_provider_states_url_gets_the_pact_convention_body(provider): + base_url, states = provider + verify_pact(_pact(), ProviderTarget(base_url, provider_states_url=f"{base_url}/_states", timeout=5)) + assert states == [{"consumer": "web", "state": "catalogue is empty"}] + + +def test_mismatches_are_reported(provider): + base_url, _states = provider + pact = new_pact("web", "shop") + add_interaction(pact, "wrong status", {"method": "GET", "path": "/items/1"}, {"status": 201}) + add_interaction(pact, "wrong type", {"method": "GET", "path": "/items/1"}, + {"status": 200, "body": {"id": "1"}}, matching_rules=TYPE) + add_interaction(pact, "wrong header", {"method": "GET", "path": "/ping"}, + {"status": 200, "headers": {"Content-Type": "application/json"}}) + report = verify_pact(pact, ProviderTarget(base_url, timeout=5)) + assert [result.mismatches for result in report.results] == [ + ["status: expected 201, got 200"], + ["$.body.id: expected a string, got number 1"], + ["$.headers.Content-Type: expected 'application/json', got 'text/plain; charset=utf-8'"], + ] + assert report.to_dict()["ok"] is False + with pytest.raises(APIContractException): + assert_pact_verified(pact, ProviderTarget(base_url, timeout=5)) + + +def test_unreachable_provider_is_a_mismatch(): + pact = new_pact("web", "shop") + add_interaction(pact, "get", {"method": "GET", "path": "/"}, {"status": 200}) + # Closed loopback port; nothing listens there. + report = verify_pact(pact, ProviderTarget("http://127.0.0.1:9", timeout=2)) # NOSONAR S5332 + assert report.results[0].mismatches[0].startswith("request failed:") + + +def test_empty_contract_does_not_pass(provider): + base_url, _states = provider + assert not verify_pact(new_pact("web", "shop"), ProviderTarget(base_url)).ok diff --git a/test/test_executor_integration.py b/test/test_executor_integration.py index 8ae97a1..0eaeb9c 100644 --- a/test/test_executor_integration.py +++ b/test/test_executor_integration.py @@ -41,6 +41,8 @@ "AT_build_jwt", "AT_aws_sigv4_headers", # Spec "AT_infer_schema", "AT_records_to_openapi", "AT_export_openapi", "AT_openapi_changelog", + # Contracts + "AT_write_contract", "AT_verify_contract", "AT_check_contract_against_openapi", # AI "AT_classify_failures", "AT_generate_fake_payload", "AT_generate_tests_from_openapi", From b2a3c7ac2c3a05aedd0889369e9282fb33c6c410 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 01:21:51 +0800 Subject: [PATCH 12/43] feat: add an Anthropic reference AI backend and backend selection AnthropicAIBackend (ai extra, anthropic>=1.0,<2) answers the AI helpers through the Messages API: claude-opus-5-5 at effort medium, server-side refusal fallbacks on, credentials from the SDK's environment. Refusals, truncation, rate limits, 5xx and network errors fall back to the deterministic helpers; rejected requests raise APIAIBackendException. Backends are chosen with select_ai_backend, AT_select_ai_backend, `apitestka generate-tests --ai`, or APITESTKA_AI_BACKEND on first use. Test generation now describes the action format and validates replies, and the failure classifier asks the backend about unmatched errors. --- .github/requirements/ci.in | 3 +- .github/requirements/ci.txt | 140 ++++++++++++++- CLAUDE.md | 2 +- README.md | 43 +++-- README/README_zh-CN.md | 39 ++-- README/README_zh-TW.md | 39 ++-- architecture.md | 7 +- dev.toml | 2 + docs/source/Eng/Eng_index.rst | 3 +- docs/source/Eng/doc/ai/ai_doc.rst | 53 ++++++ docs/source/Eng/doc/cli/cli_doc.rst | 2 + docs/source/Zh/Zh_index.rst | 3 +- docs/source/Zh/doc/ai/ai_doc.rst | 47 +++++ docs/source/Zh/doc/cli/cli_doc.rst | 2 + docs/updates/2026-10.md | 14 ++ docs/updates/README.md | 3 +- je_api_testka/ai/__init__.py | 8 +- je_api_testka/ai/anthropic_backend.py | 149 ++++++++++++++++ je_api_testka/ai/backend.py | 84 +++++++-- je_api_testka/ai/failure_classifier.py | 48 ++++- je_api_testka/ai/fake_data_generator.py | 5 +- je_api_testka/ai/reply.py | 21 +++ je_api_testka/ai/test_generator.py | 41 +++-- je_api_testka/cli/cli_main.py | 34 ++++ je_api_testka/cli/completion.py | 3 +- je_api_testka/utils/exception/exceptions.py | 4 + .../utils/executor/action_executor.py | 2 + progress.md | 1 - pyproject.toml | 2 + test/test_ai/test_anthropic_backend.py | 147 +++++++++++++++ test/test_ai/test_backend_selection.py | 123 +++++++++++++ test/test_executor_integration.py | 2 +- uv.lock | 168 +++++++++++++++++- 33 files changed, 1172 insertions(+), 72 deletions(-) create mode 100644 docs/source/Eng/doc/ai/ai_doc.rst create mode 100644 docs/source/Zh/doc/ai/ai_doc.rst create mode 100644 je_api_testka/ai/anthropic_backend.py create mode 100644 je_api_testka/ai/reply.py create mode 100644 test/test_ai/test_anthropic_backend.py create mode 100644 test/test_ai/test_backend_selection.py diff --git a/.github/requirements/ci.in b/.github/requirements/ci.in index 9235366..85b7730 100644 --- a/.github/requirements/ci.in +++ b/.github/requirements/ci.in @@ -1,5 +1,5 @@ # What the CI test job installs; ci.txt is generated from this file (the command is in ci.yml). -# Runtime dependencies (pyproject.toml), the gui, mcp, websocket and grpc extras, and the test tools. +# Runtime dependencies (pyproject.toml), the gui, mcp, websocket, grpc and ai extras, and the test tools. requests Flask httpx @@ -9,6 +9,7 @@ qt-material mcp>=1.28.1,<3 websockets>=12.0 grpcio>=1.60 +anthropic>=1.0,<2 pytest pytest-asyncio pytest-timeout diff --git a/.github/requirements/ci.txt b/.github/requirements/ci.txt index 357cf67..93f6c1f 100644 --- a/.github/requirements/ci.txt +++ b/.github/requirements/ci.txt @@ -4,10 +4,15 @@ annotated-types==0.8.0 \ --hash=sha256:13b2beaad985e05e2d6407ee4c4f35590b11f8d693a258a561055cac8f64cab7 \ --hash=sha256:f072f4d804ea359e4eaf198b1af7a8b0943881a87f31bb764f8bf219bb9419e0 # via pydantic +anthropic==1.9.0 \ + --hash=sha256:a7fe05a9e4b72113b934501698df5d56484ac89f05677f0f793f8bfddb15cfff \ + --hash=sha256:da5191a0af0b853a797fc0c6b09fc5b05cb5d9da063ba80fdd03c736abe81b6c + # via -r .github/requirements/ci.in anyio==4.15.1 \ --hash=sha256:6152fdbbf9a77fdec97731721bebf7c4c44f7c29b424b0065826173efc7ed101 \ --hash=sha256:9f28306018cbd6d329e64a36d58256edff76dd996fe423bc957326e578b82a94 # via + # anthropic # httpx # httpx2 # mcp @@ -372,6 +377,10 @@ defusedxml==0.7.1 \ --hash=sha256:1bb3032db185915b62d7c6209c5a8792be6a32ab2fedacc84e01b52c51aa3e69 \ --hash=sha256:a352e7e428770286cc899e2542b6cdaedb2b4953ff269a210103ec58f6198a61 # via -r .github/requirements/ci.in +docstring-parser==0.18.0 \ + --hash=sha256:292510982205c12b1248696f44959db3cdd1740237a968ea1e2e7a900eeb2015 \ + --hash=sha256:b3fcbed555c47d8479be0796ef7e19c2670d428d72e96da63f3a40122860374b + # via anthropic exceptiongroup==1.3.1 ; python_full_version < '3.11' \ --hash=sha256:8b412432c6055b0b7d14c310000ae93352ed6754f70fa8f7c34141f91c4e3219 \ --hash=sha256:a7a39a3bd276781e98394987d3a5701d0c4edffb633bb7a5144577f82c773598 @@ -467,7 +476,9 @@ httpx==0.28.1 \ httpx2==2.13.1 \ --hash=sha256:6dff50fabc270ee5fd25d845d0b078ed20564579744d6d962850975996d2f9a4 \ --hash=sha256:e48744a19e3af5ee48313d0ce5fe941d5422fae5705ea922a4aabf94d7800dfa - # via mcp + # via + # anthropic + # mcp httpx2-jsfetch==1.0 ; python_full_version >= '3.12' and sys_platform == 'emscripten' \ --hash=sha256:70a0e3eabfef7cce5ad9c629f7d01ca05e418f586646f4ddf14782e4c1454c60 \ --hash=sha256:cb916b707601e69a07721aabc8f3f6659be3a6893bc1ff5c6f9e02241df2da32 @@ -494,6 +505,127 @@ jinja2==3.1.6 \ # via # flask # qt-material +jiter==0.17.0 \ + --hash=sha256:00b5a98df3e3a3e8cf7b619f4ac2f8bf975bbf3d95d02c5d17b8dbfe5c8b8245 \ + --hash=sha256:00d783a779c5664e16dbad5e3a3c3a75e128b07dd5f4765159658d9210a50ca5 \ + --hash=sha256:0239520085cac678e77a606fd7e3f1c60c371d719790c5e3807388d3da4354c2 \ + --hash=sha256:02a360707033d8cef53f7f3480817a1489177a259ec6ec01e98c37e0b922ddca \ + --hash=sha256:02adebb7ce6413c44d40af9ad59d1c1cd79630ccdcb6f7bdd2d461e48c03d8f9 \ + --hash=sha256:03e432f226a453851079fb84cd17c6da9991eab723e28d716f14ae3d906e0c12 \ + --hash=sha256:0619d806e260ecf0c2a64521942c94af5d547c9ec99b55ae4f51b538b5576a76 \ + --hash=sha256:073dc68c1a700c8fc480e877864a6b6ffc887533e261f4380c08c16bf09d057a \ + --hash=sha256:0b52d52035b3907c5b1f6277857b29c1cbfc965e24e0f27330dbed83edb591ec \ + --hash=sha256:10c5349312e5cb02b7a21e123a57665afa895953f05bf252a9dd4c13a572b7ab \ + --hash=sha256:10cd64a5720ad7f809ac5466ff1705813f1b6b510f195a73acafba0ac0e1f675 \ + --hash=sha256:10f5558eed511b830488003449d942bd75829ad6257dc58cb9a03e596a7777b1 \ + --hash=sha256:11902505d401691720f5785c15b02204248526edee11b635cd6c40cd52b81599 \ + --hash=sha256:155be7355bdb7ca76ab0961be8982c225f964a5c073a83984183f22391cc29fc \ + --hash=sha256:16dd0c1baf098ae70b8f3616574eb3fedf34e26670b89e16a7e67561f737ed2d \ + --hash=sha256:1b18434638228c0c184281609bf3d9459026a0f1ea48fb76c205e3ef72069caa \ + --hash=sha256:29f49b325e0234e4ad9ecca5b861ffbd09b95ccac9bd46fa55841b6e56eea5fe \ + --hash=sha256:2c45ad7c973ef33fe5114a953377b35a95240f4542c0724d9f781e47dc24bac7 \ + --hash=sha256:300ce01ab0215e3dea4d00090143c909aedc65c0f809b3c07983e1d038f291b9 \ + --hash=sha256:30793a24a31e968969757c9e08d830cbb15a2cd3c4959b4498b38f4b1c2258eb \ + --hash=sha256:30c692d567ba206c7cca38c9d1d0ccc70c9786290173c184d871ca12e9981ed7 \ + --hash=sha256:32aaaa764604496610a3ad2d98503ae88ccb2fbe769e892ff4533e778e85f708 \ + --hash=sha256:362bb47423886d45a9f705d2d9d4008c6eedd4e41eb1bab4e96fb6daa06b33fd \ + --hash=sha256:36ee6e69027396664e59995b9a635a947a5304ee9837279584a0bb8145c8f6b8 \ + --hash=sha256:370d8fe5bf201dc6925e8a84c81ac7291f74d9fd1778234fc79d517064a5c76b \ + --hash=sha256:37150a9e02e869475854fa20b7d0d5e26d18d0f8bc17293999973ff27e99ae7a \ + --hash=sha256:37f33d327900bf2879613b3363fd48df97b4232d0c41f54bcf2e790c2fc40a71 \ + --hash=sha256:3ad556afc289f15d2b181b941982d01f06190863c07440185b9f354e1bd2def3 \ + --hash=sha256:3bf4dc2b84a464117fb097d15a25c58d100d2692888e3b0d92df5b48ed16b7c0 \ + --hash=sha256:3c1a5336c04a41b1f1cf9572e294aec27cc569767ff73de7bf87a91f0bea7cb9 \ + --hash=sha256:3e05f5adbf68c4bd11e1610f394034d984152988e84be6f8314235ce6f2139e5 \ + --hash=sha256:40d2c240f8f80b5b0f201b29f0ae129c81448c60c772227a41747b5e0026f6a2 \ + --hash=sha256:42b0260445251b1bc520a63baa94a32d88e0f931fba234f1764db7feb7c72174 \ + --hash=sha256:454c4997d73cc466c71fd565d91e603b0274e48ea0c6b0b7a7aee6967e4ceb7c \ + --hash=sha256:455e4ab35cb2a4a91a8404e08fd3c621bae433922e59bf1c494fe20a426b013b \ + --hash=sha256:4607ec7d93355fbc25b8dc5189153cf21d66063b9f9cd04dd2774e6e783f9b6a \ + --hash=sha256:470e1b1e4c42f1ead2189166a299691871a2df5056c976e7fb96feafaf5f9d44 \ + --hash=sha256:492f37230bbf9581ab2c17bcda862c249afb9ae2e3ab2dd6db59943bc4cc3153 \ + --hash=sha256:4dfbfe5a6e1e80a7082af559f66386405025ec278833e0c649f69cbc6e1004cc \ + --hash=sha256:4e3f052c671d5f425cca5ea5901cf11a831369fba4a55a3862cab93c323b4c3b \ + --hash=sha256:5078ab00664307fab2019b522a93aeb191122789f085daf5fd9e362154021d4a \ + --hash=sha256:51e1519d676a9f14dad9c2a411170d43b022ddb7989562df4e849b261ce127b2 \ + --hash=sha256:523c499235fb65add25d4bb01b1c4709ce695efdc7deb6c0a7bc515b5c44e0fb \ + --hash=sha256:545c36a0f3b2238c242cc9785439d3242a871b7bc39fe3f441bcaa07bf3aa83e \ + --hash=sha256:55d0e0e613a3f9ad600cf436e0e2b8057d1b52bcf1d91b2d36ac53451231e6a8 \ + --hash=sha256:5888fe5abc1ca2fa834a3e1b4c7ef0dcece286a7d7e95a609ef0934b777b9fc9 \ + --hash=sha256:58df29268a95e910f17db7ec9178eb7f15aa8619aaca3575275c4e6b3f4fe4c5 \ + --hash=sha256:59bddbe6f9ffecc68d641e1e2d619ce64cf8a9e9eeb74e5c518f74fc87abf1b0 \ + --hash=sha256:5a52a430d04225ffde633e6840bf2381d34c019ff98526b5929755b9052fb199 \ + --hash=sha256:5bf350452a43173e69e1fc74847c57a60e3d7515807287f29849baa2a85d8718 \ + --hash=sha256:5c23849235d2142ce444b2b8c6eceee9f82f4cc0bd5c9081602e4155c6197807 \ + --hash=sha256:61aed66ee042b3b49ef85fdf75714234d055d89d8496ac1c6e47f89e7a30d5e4 \ + --hash=sha256:6219adaf59711ba7063a52496e8ec6d3fa3e209d7827d83eee3b2abc780a1744 \ + --hash=sha256:64846211a2debe7c071d2146d2283d2b0c1c93dc8fd5fb7794faac2ca6061b5c \ + --hash=sha256:686c93d86f2b426c803024b805bd161a6cd10e9627c23e901640eab646c0ad8a \ + --hash=sha256:6871973bfbd4408f7f1c632b30bbb5bbd9671c1bc8650af6823e24b7be13709b \ + --hash=sha256:6af5b74073bd25bae695e6d00919f6a9be7ed5a9f8836d981eb1ffe84139e6fb \ + --hash=sha256:6b303d88e6a0bda789ec4b7801c7bad68e27230ba1fe4baffc756d1fbd32dc9d \ + --hash=sha256:6cb41cd1432f1dc19a231cf70b54d42b2c9f05085155859263fce06fa4d41388 \ + --hash=sha256:6cf564d43c4388149ca58ee571d0f5ccf875e20d1fd4662fd94cc0d1ea3b10ef \ + --hash=sha256:6eb6aedeb7352b8f3b6af9cbd67983840165c00428e63f1b420a85885128ea31 \ + --hash=sha256:70f19a2ca8429f91e82eeffb2f51cb87bc2d6e953b009b91a92d29c3a16ccb03 \ + --hash=sha256:71dbd74314c5df52a1bccf7b8bca46d14e943af7a2012e73b23f49977ef194c8 \ + --hash=sha256:73b64e69c4150748e020356d958af94bec33c70a0a93d665cfa8f6d580fe1a63 \ + --hash=sha256:746243a080b4ca790b8499af3d7cf9825d5f5987933950cd818e767ee353d826 \ + --hash=sha256:755079792868ce5d4938e83b91a0939b34fb858a1ca65a104f2d771bea57faa1 \ + --hash=sha256:7573e80232c5bcf80c24c038cf7e53a463f5c3b1dd1dd4109d66304f4dccc233 \ + --hash=sha256:76eb4a5c20e86f9f848286f167024890f2862258a965d254774deb7fc1545ca1 \ + --hash=sha256:77f6aac0137309b31448c1bdcda4c6c77077664a6d018ece8d94019c68a5a5b9 \ + --hash=sha256:785a216bbaf8f15fc974e964ced7322cd3d774bb0e86949edd78c6bffd6ba35b \ + --hash=sha256:7b68d3495d95da120651a5628c7ebadee84ed001a1b76e6afc325c42482f15b5 \ + --hash=sha256:8079849db9a1371bfd90bad088458a8fb836261879df2233cc9632464ecf64e1 \ + --hash=sha256:81c83c0abe614446a283d994d2c07c4f58632dea2cdf66ba9e2921bb8ccd593e \ + --hash=sha256:826871c42cebaae22f0a2b5673a4a1a75c851bb2d13b3c17764a630a6b298984 \ + --hash=sha256:84963d3f395ef5e9a32ce47155e08a7962fa292c159a10cb98b931cef1416925 \ + --hash=sha256:84ac78df457e1ee3f7e733bd114823302ae8c5ad5542d7e6647d92ffaa090a04 \ + --hash=sha256:86d703d9faa1ffc8ae4e9de0fa007712ed2171b5c0d93811a8e2e105ac729b0d \ + --hash=sha256:86f3f9343a288eb85a81ef20a752b2f84564296636db54a9fff0b5c8deaf1df2 \ + --hash=sha256:8adca2e793288e5f1bb29279bb439d0d3cfbb50eddca7e7e6ffd42ff4f482406 \ + --hash=sha256:8c21265b251d99bbb40080d178a8953e35601d3a1564e05c4de4c0d2ca616797 \ + --hash=sha256:8c286860abfe8b100cac1c02e225e5776eb9216edd71ba17cdb237da4af32bc9 \ + --hash=sha256:8f770b0c77e5fac482e1ba03ca1a7e18286bfb213d749932a00a7e4cd5de5e06 \ + --hash=sha256:93946d89fa04d5ba64dd323a8dd8d901676cb8a3c81d99ae4f6c051a9b4c3f2f \ + --hash=sha256:96b8b0c6dc5d78682f54a450785e075aa929cde768304cad363cd4efba5a82ac \ + --hash=sha256:9bd3caac219df476dd0cc3fe01d2f1581ed588906feac767abd9614c1c12f8b3 \ + --hash=sha256:a277f97eba7d66b1ee27eb5dab5b774ff46a10c78d89a1d3dcce04ce1357c8ca \ + --hash=sha256:a3cebb1fe4a1abb00465f3f8a17e09112603e8b7c59e5c3adbcd9f7815a64acd \ + --hash=sha256:ac3c6ee3264d6f5c44c617f90bc7e8b9e1587e7d6708c9d8f811cb65582ee312 \ + --hash=sha256:af2f7501580f274b63c4b2283bc425f5df7edf06ae5b171e5f87d912ff359a20 \ + --hash=sha256:b550585523339b71cb852b811aae49d08d7601ad8ffe9f5dc1562f4c3d22fd87 \ + --hash=sha256:b75f85660108965a94be77911a25a253429307294d9415b3c597118977a614de \ + --hash=sha256:b847b18d066c46b3b7ae49d6c94a7634c5e4a8983146ee25562a092000f5e3ad \ + --hash=sha256:bcc064f99183a9cbe7f26ed648c352031a74145cd61ed75d34632c73eb46a5a8 \ + --hash=sha256:c19b9357309b8cc6de8a48fca8e44a8c9c2feaaa2f5896d037fa505d48fcab80 \ + --hash=sha256:c4289293e5278d9314b00f15c37f2120fa51d3d68565292e715524c750e775a9 \ + --hash=sha256:cfafd7be8b16ceadd298db542cead37cddc211c4c49e04ad2596924df18625b1 \ + --hash=sha256:d0ce4feb52493e3513335b2accdcd75605652e4632772d3c8c2f7b86954d7f39 \ + --hash=sha256:d2c0bf24c72fd0491405dce5d40194f2070e9021ce648c1a1d46234b93d848ff \ + --hash=sha256:d47687806f9c54c84ea38733507081337922beca90ce819c7d852dd485bc0f23 \ + --hash=sha256:d85c558c9f8532bba287a990ac63767c7daf756f0d8c030219f62499b1fa228a \ + --hash=sha256:da139721f4b7cafdbff580a4f511ea24cb91f4909330c6b926a1ca53836c0a59 \ + --hash=sha256:dbbfe4e3c21c8166980cddc5bee1a315df082454f007947dfb6fb73800768165 \ + --hash=sha256:dc0288ce39190ee33fe6e4ec73161eed34e7e2da509b525546ca061778d62b64 \ + --hash=sha256:e088612ff90ebc9247e1a43074b72835804261c47e6a6c01cb3ddcb55360d688 \ + --hash=sha256:e654b6b04e39c9cb19cb8b04c6ddf1f2db07751fa14156413969fd78bad0e5cb \ + --hash=sha256:eaba834b72d573547b9d966465b3394b749d5e14208cc70acb63aca37619ab33 \ + --hash=sha256:eae86b1f027031e39db2e0e9c4842221edb7b8cd474d23f87a79b3bd4b651768 \ + --hash=sha256:eb2295da7c3769f6719b227a237aa6a5cfa6550e478bc838001b592c57e16575 \ + --hash=sha256:ebf918dfd6a74adc1b9ad71f63c4ab00902fcd3b7fd39f2e24d871db8d713b91 \ + --hash=sha256:ec89771f4272b989487a6364e519db6bbaba323e8bbf949ac89a45ea9c18b7a3 \ + --hash=sha256:ed1a24005daac667d577402d75a2922f9775a165b146b883ff1ad3602d8be689 \ + --hash=sha256:efe9f61bb30174d2f5c8396445c360c96c44e78164d0815dfe627ccf57849574 \ + --hash=sha256:f0bc7f684b65bcda9c20434267577db71bf9905ceddd32b60d1d93278d8c8d3a \ + --hash=sha256:f3d7f7b34114f7ddc6d72a8e882d49de636b35d9fd12b4d420d3c5729f6c9812 \ + --hash=sha256:f753eb70b1474a29e635e7542ff7312e6d6b951e0b25e8a2e8c34eeb1ddcd478 \ + --hash=sha256:fa13acf1046f95df808c64b1310705e143fab87aee73ae00cc42d640867fd2c1 \ + --hash=sha256:fd7790aa79c8b518e512ebcdfce9f11d8ef5f30efd43720c8a19a548b39fa489 \ + --hash=sha256:fe15ddf316f1f1f643347d3a474e74ce61880c79a11ec5dca53df20c071bd3e8 \ + --hash=sha256:ffa0380ad091de7d3fc33e17a97ff479851ee18a0a2a3ee56ff3215cdc886656 + # via anthropic jsonschema==4.26.0 \ --hash=sha256:0c26707e2efad8aa1bfc5b7ce170f3fccc2e4918ff85989ba9ffa9facb2be326 \ --hash=sha256:d489f15263b8d200f8387e64b4c3a75f06629559fb73deb8fdfb525f2dab50ce @@ -624,6 +756,7 @@ pydantic==2.13.5 \ --hash=sha256:346a034f080da3755d8e9cb5e00e8b07de1d39e4f6e2c87d8ab7cafa0b269a73 \ --hash=sha256:51a9c5f7b2f8e636f04c6cada605d9b6a3bf1348fdf945a3d8869b19bba0ee08 # via + # anthropic # mcp # mcp-types pydantic-core==2.46.5 \ @@ -1084,6 +1217,10 @@ shiboken6==6.11.2 \ # pyside6 # pyside6-addons # pyside6-essentials +sniffio==1.3.1 \ + --hash=sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2 \ + --hash=sha256:f4324edc670a0f49750a81b895f35c3adb843cca46f0530f79fc1babb23789dc + # via anthropic sse-starlette==3.4.11 \ --hash=sha256:1bae716c02f3e6f294be41ff333220692dae7c3cbab077c900f159676719dade \ --hash=sha256:c7b2244bdff016fe7f64e10075e89a3e6bbf899649cc89b0fe884b5545042453 @@ -1155,6 +1292,7 @@ typing-extensions==4.16.0 \ --hash=sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8 \ --hash=sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5 # via + # anthropic # anyio # cryptography # exceptiongroup diff --git a/CLAUDE.md b/CLAUDE.md index 46b9bf0..1ac6e6c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -58,7 +58,7 @@ je_api_testka/ ├── security/ # Auth helpers, header scan, fuzzing, pip-audit wrapper ├── runner/ # Parallel runner, tag filter, dependency ordering ├── integrations/ # cURL and HAR import, webhook notify, GitHub PR comment -├── ai/ # Pluggable text-completion backend (no-op by default) +├── ai/ # Pluggable text-completion backend (no-op by default; Anthropic reference backend) ├── cli/ # Subcommand CLI (`apitestka`) ├── mcp_server/ # MCP stdio server (`apitestka-mcp`) ├── pytest_plugin/ # pytest fixtures via the pytest11 entry point diff --git a/README.md b/README.md index 0195ae1..2c9dd7d 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,7 @@ server for Claude, and more. | **Security** | Auth helpers (Basic / Bearer / JWT / AWS SigV4), header / CORS / rate-limit / SSRF probes, pip-audit wrapper, fuzz inputs | | **Spec inference** | Test record → OpenAPI, JSON Schema inference, OpenAPI changelog | | **Contract testing** | Pact v2 consumer contracts from test runs, provider verification, bidirectional check against OpenAPI | -| **AI** | Pluggable backend with deterministic fallback for test generation, fake data, failure classification | +| **AI** | Pluggable backend with deterministic fallback for test generation, fake data, failure classification; Anthropic reference backend | | **MCP** | First-class Claude Code / MCP server exposing the framework as tools | | **GUI** | Optional PySide6 GUI (English / 繁中 / 简中 / 日本語) plus Swagger UI embed | | **Cross-platform** | Windows, macOS, Linux. Python 3.10–3.14 | @@ -356,6 +356,7 @@ apitestka mcp # MCP server over stdio apitestka openapi --report run_success.json -o openapi.json # infer a spec from a saved run apitestka openapi --run actions.json # run first, print the inferred spec apitestka contract verify pacts/web-shop.json --base-url http://localhost:8000 # see Contract Testing +apitestka generate-tests openapi.json -o actions.json [--ai anthropic] # see Pluggable AI Backend ``` ### Security Probes @@ -447,23 +448,40 @@ Locales: English, 繁體中文, 简体中文, 日本語. Switch via `LanguageWra ### Pluggable AI Backend -Default backend (`NoOpAIBackend`) never calls a network. Plug in your own to enable -LLM-driven test generation: +Three helpers can ask a language model: `generate_tests_from_openapi` (actions for every +operation), `generate_fake_payload` (a value for a JSON Schema) and `classify_failures` (labels for the +errors its rules cannot place). The default backend (`NoOpAIBackend`) never calls a network, and every +helper then uses its deterministic fallback, as it also does when a reply is empty or unusable +(generated actions must be a JSON list of `["AT_...", {...}]`). + +`AnthropicAIBackend` (`ai` extra) is the reference implementation on the Anthropic API. It takes +credentials from the environment the `anthropic` SDK reads (`ANTHROPIC_API_KEY`, or an `ant auth login` +profile) and never stores a key. The default model is `claude-opus-5-5`, with effort `medium` sent +explicitly. Server-side refusal fallbacks are on (`fallbacks="default"`); pass `fallbacks=None` to +turn them off. A refusal, a truncated reply, a rate limit, a server error or a network failure gives +the deterministic fallback, and a rejected request (bad key, unknown model) raises `APIAIBackendException`. ```python -from je_api_testka.ai import AIBackend, set_ai_backend, generate_tests_from_openapi +from je_api_testka.ai import AnthropicAIBackend, set_ai_backend, generate_tests_from_openapi + +set_ai_backend(AnthropicAIBackend()) # claude-opus-5-5, effort "medium" +set_ai_backend(AnthropicAIBackend(model="claude-sonnet-5-5", effort="low", timeout=60)) +actions = generate_tests_from_openapi(my_openapi_spec) +``` -class AnthropicBackend(AIBackend): - def complete(self, prompt, *, context=None): - ... # call your provider - return llm_response_text +Select a backend with `set_ai_backend(...)`, `select_ai_backend("anthropic", model=..., effort=...)`, +the `AT_select_ai_backend` action, or the environment: without a selection, the first use reads +`APITESTKA_AI_BACKEND` (`noop` or `anthropic`), `APITESTKA_AI_MODEL` and `APITESTKA_AI_EFFORT`. -set_ai_backend(AnthropicBackend()) -actions = generate_tests_from_openapi(my_openapi_spec) # falls back to a deterministic - # happy path if the LLM reply is - # not valid JSON +```bash +pip install 'je_api_testka[ai]' +export ANTHROPIC_API_KEY=... # or an `ant auth login` profile +apitestka generate-tests openapi.json -o actions.json --ai anthropic --effort high +APITESTKA_AI_BACKEND=anthropic apitestka mcp # any entry point, no code changes ``` +Any other provider plugs in the same way: subclass `AIBackend` and implement `complete(prompt, *, context)`. + --- ## MCP Server for Claude @@ -537,6 +555,7 @@ je_api_testka/ ```bash pip install 'je_api_testka[websocket]' pip install 'je_api_testka[grpc]' +pip install 'je_api_testka[ai]' pip install 'je_api_testka[schema]' pip install 'je_api_testka[security]' pip install 'je_api_testka[otel]' diff --git a/README/README_zh-CN.md b/README/README_zh-CN.md index 0843082..b48b9e4 100644 --- a/README/README_zh-CN.md +++ b/README/README_zh-CN.md @@ -62,7 +62,7 @@ record-replay 代理、安全检测、并行执行 runner,以及给 Claude 用 | **安全** | Auth helper(Basic / Bearer / JWT / AWS SigV4)、Header / CORS / Rate limit / SSRF probe、pip-audit、Fuzz | | **Spec 反推** | 测试记录 → OpenAPI、JSON Schema 推断、OpenAPI changelog | | **契约测试** | 从测试运行生成 Pact v2 消费者契约、提供端验证、与 OpenAPI 双向比较 | -| **AI** | 可插拔后端,LLM 不可用时自动回退到确定性 fallback | +| **AI** | 可插拔后端,LLM 不可用时自动回退到确定性 fallback;附 Anthropic 参考实现 | | **MCP** | 一级支持 Claude Code,将框架作为 MCP 工具暴露 | | **GUI** | 可选 PySide6 GUI(英 / 繁中 / 简中 / 日)+ 嵌入 Swagger UI | | **跨平台** | Windows、macOS、Linux,Python 3.10–3.14 | @@ -352,6 +352,7 @@ apitestka mcp # 走 stdio 启动 MCP server apitestka openapi --report run_success.json -o openapi.json # 从保存的运行结果反推规格 apitestka openapi --run actions.json # 先运行再打印反推的规格 apitestka contract verify pacts/web-shop.json --base-url http://localhost:8000 # 见契约测试 +apitestka generate-tests openapi.json -o actions.json [--ai anthropic] # 见可插拔 AI 后端 ``` ### 安全检测 @@ -439,21 +440,38 @@ headless 工具无需 PySide6 也能驱动面板。 ### 可插拔 AI 后端 -默认 `NoOpAIBackend` 不会触网。要启用 LLM 驱动的测试生成,自行挂载: +有三个工具会询问语言模型:`generate_tests_from_openapi`(为每个 operation 生成 action)、 +`generate_fake_payload`(生成符合 JSON Schema 的值)与 `classify_failures`(给规则分不出来的错误打标签)。 +默认 `NoOpAIBackend` 不会访问网络,各工具会改用确定性的 fallback;回复为空或无法使用时也一样 +(生成的 action 必须是 `["AT_...", {...}]` 组成的 JSON list)。 + +`AnthropicAIBackend`(`ai` extra)是基于 Anthropic API 的参考实现。凭据取自 `anthropic` SDK 读取的环境 +(`ANTHROPIC_API_KEY` 或 `ant auth login` 配置文件),不会保存密钥。默认模型是 `claude-opus-5-5`,并显式发送 +effort `medium`。服务器端的拒答 fallback 默认开启(`fallbacks="default"`),传 `fallbacks=None` 可关闭。 +遇到拒答、回复被截断、速率限制、服务器错误或网络失败时会改用确定性 fallback;请求被拒(密钥错误、 +未知模型)则抛出 `APIAIBackendException`。 ```python -from je_api_testka.ai import AIBackend, set_ai_backend, generate_tests_from_openapi +from je_api_testka.ai import AnthropicAIBackend, set_ai_backend, generate_tests_from_openapi + +set_ai_backend(AnthropicAIBackend()) # claude-opus-5-5, effort "medium" +set_ai_backend(AnthropicAIBackend(model="claude-sonnet-5-5", effort="low", timeout=60)) +actions = generate_tests_from_openapi(my_openapi_spec) +``` -class AnthropicBackend(AIBackend): - def complete(self, prompt, *, context=None): - ... # 调用你的 LLM provider - return llm_response_text +选择 backend 的方式:`set_ai_backend(...)`、`select_ai_backend("anthropic", model=..., effort=...)`、 +`AT_select_ai_backend` action,或环境变量:没有显式选择时,第一次使用会读取 `APITESTKA_AI_BACKEND` +(`noop` 或 `anthropic`)、`APITESTKA_AI_MODEL` 与 `APITESTKA_AI_EFFORT`。 -set_ai_backend(AnthropicBackend()) -actions = generate_tests_from_openapi(my_openapi_spec) # LLM 回应不是合法 JSON 时 - # 自动降级为确定性 happy path +```bash +pip install 'je_api_testka[ai]' +export ANTHROPIC_API_KEY=... # or an `ant auth login` profile +apitestka generate-tests openapi.json -o actions.json --ai anthropic --effort high +APITESTKA_AI_BACKEND=anthropic apitestka mcp # any entry point, no code changes ``` +其他 provider 也用同样方式接入:继承 `AIBackend` 并实现 `complete(prompt, *, context)`。 + --- ## Claude 的 MCP Server @@ -527,6 +545,7 @@ je_api_testka/ ```bash pip install 'je_api_testka[websocket]' pip install 'je_api_testka[grpc]' +pip install 'je_api_testka[ai]' pip install 'je_api_testka[schema]' pip install 'je_api_testka[security]' pip install 'je_api_testka[otel]' diff --git a/README/README_zh-TW.md b/README/README_zh-TW.md index c355e51..007a48e 100644 --- a/README/README_zh-TW.md +++ b/README/README_zh-TW.md @@ -62,7 +62,7 @@ record-replay proxy、安全性檢測、平行執行 runner,以及給 Claude 用 | **安全** | Auth helper(Basic / Bearer / JWT / AWS SigV4)、Header / CORS / Rate limit / SSRF probe、pip-audit、Fuzz | | **Spec 反推** | 測試紀錄 → OpenAPI、JSON Schema 推斷、OpenAPI changelog | | **契約測試** | 從測試執行產生 Pact v2 消費者契約、提供端驗證、與 OpenAPI 雙向比對 | -| **AI** | 可插拔後端,LLM 不可用時自動退回確定性 fallback | +| **AI** | 可插拔後端,LLM 不可用時自動退回確定性 fallback;附 Anthropic 參考實作 | | **MCP** | 一級支援 Claude Code,將框架曝露為 MCP 工具 | | **GUI** | 可選 PySide6 GUI(英 / 繁中 / 简中 / 日)+ 嵌入 Swagger UI | | **跨平台** | Windows、macOS、Linux,Python 3.10–3.14 | @@ -352,6 +352,7 @@ apitestka mcp # 走 stdio 啟動 MCP server apitestka openapi --report run_success.json -o openapi.json # 從存下的執行結果反推規格 apitestka openapi --run actions.json # 先執行再印出反推的規格 apitestka contract verify pacts/web-shop.json --base-url http://localhost:8000 # 見契約測試 +apitestka generate-tests openapi.json -o actions.json [--ai anthropic] # 見可插拔 AI 後端 ``` ### 安全檢測 @@ -439,21 +440,38 @@ headless 工具不需 PySide6 也能驅動面板。 ### 可插拔 AI 後端 -預設 `NoOpAIBackend` 不會碰網路。要啟用 LLM 驅動的測試生成,自行掛載: +有三個工具會詢問語言模型:`generate_tests_from_openapi`(為每個 operation 產生 action)、 +`generate_fake_payload`(產生符合 JSON Schema 的值)與 `classify_failures`(替規則分不出來的錯誤貼標籤)。 +預設 `NoOpAIBackend` 不會碰網路,各工具會改用確定性的 fallback;回覆是空的或無法使用時也一樣 +(產生的 action 必須是 `["AT_...", {...}]` 組成的 JSON list)。 + +`AnthropicAIBackend`(`ai` extra)是用 Anthropic API 的參考實作。憑證取自 `anthropic` SDK 讀的環境 +(`ANTHROPIC_API_KEY` 或 `ant auth login` 設定檔),不會保存金鑰。預設模型是 `claude-opus-5-5`,並明確送出 +effort `medium`。伺服器端的拒答 fallback 預設開啟(`fallbacks="default"`),傳 `fallbacks=None` 可關閉。 +遇到拒答、回覆被截斷、速率限制、伺服器錯誤或網路失敗時會改用確定性 fallback;請求被拒(金鑰錯誤、 +未知模型)則丟出 `APIAIBackendException`。 ```python -from je_api_testka.ai import AIBackend, set_ai_backend, generate_tests_from_openapi +from je_api_testka.ai import AnthropicAIBackend, set_ai_backend, generate_tests_from_openapi + +set_ai_backend(AnthropicAIBackend()) # claude-opus-5-5, effort "medium" +set_ai_backend(AnthropicAIBackend(model="claude-sonnet-5-5", effort="low", timeout=60)) +actions = generate_tests_from_openapi(my_openapi_spec) +``` -class AnthropicBackend(AIBackend): - def complete(self, prompt, *, context=None): - ... # 呼叫你的 LLM provider - return llm_response_text +選擇 backend 的方式:`set_ai_backend(...)`、`select_ai_backend("anthropic", model=..., effort=...)`、 +`AT_select_ai_backend` action,或環境變數:沒有明確選擇時,第一次使用會讀 `APITESTKA_AI_BACKEND` +(`noop` 或 `anthropic`)、`APITESTKA_AI_MODEL` 與 `APITESTKA_AI_EFFORT`。 -set_ai_backend(AnthropicBackend()) -actions = generate_tests_from_openapi(my_openapi_spec) # LLM 回應不是合法 JSON - # 時會自動降級為確定性 happy path +```bash +pip install 'je_api_testka[ai]' +export ANTHROPIC_API_KEY=... # or an `ant auth login` profile +apitestka generate-tests openapi.json -o actions.json --ai anthropic --effort high +APITESTKA_AI_BACKEND=anthropic apitestka mcp # any entry point, no code changes ``` +其他 provider 也用同樣方式接上:繼承 `AIBackend` 並實作 `complete(prompt, *, context)`。 + --- ## Claude 用的 MCP Server @@ -527,6 +545,7 @@ je_api_testka/ ```bash pip install 'je_api_testka[websocket]' pip install 'je_api_testka[grpc]' +pip install 'je_api_testka[ai]' pip install 'je_api_testka[schema]' pip install 'je_api_testka[security]' pip install 'je_api_testka[otel]' diff --git a/architecture.md b/architecture.md index 3be8f5c..aa23003 100644 --- a/architecture.md +++ b/architecture.md @@ -37,7 +37,7 @@ CLI, a TCP socket server, an MCP server, a pytest plugin and an optional PySide6 | `je_api_testka/security/` | Auth header helpers, CORS/SSRF/rate-limit probes, header scan, fuzzing, `pip-audit` wrapper | | `je_api_testka/runner/` | Parallel runner, tag filter, dependency ordering of actions | | `je_api_testka/integrations/` | cURL and HAR import, webhook notify, GitHub PR comment | -| `je_api_testka/ai/` | Pluggable text-completion backend (no-op by default) used by test generation, failure classification and fake payloads | +| `je_api_testka/ai/` | Pluggable text-completion backend (no-op by default) used by test generation, failure classification and fake payloads; `AnthropicAIBackend` (`ai` extra) is the reference implementation; selection by `set_ai_backend` / `select_ai_backend` / `AT_select_ai_backend` or `APITESTKA_AI_BACKEND` on first use | | `je_api_testka/cli/` | Subcommand CLI (`apitestka`): import, REPL, scaffold, completion, terminal summary | | `je_api_testka/mcp_server/` | MCP stdio server and its tool catalogue | | `je_api_testka/pytest_plugin/` | pytest fixtures, registered through the `pytest11` entry point | @@ -61,7 +61,8 @@ CLI, a TCP socket server, an MCP server, a pytest plugin and an optional PySide6 `-c/--create_project` and `--execute_str`. On `win32`/`cygwin`/`msys`, `--execute_str` is decoded with `json.loads` twice. Errors print `repr(error)` to stderr and exit with code 1. - **Subcommand CLI**: `apitestka` (`je_api_testka.cli.cli_main:main`) with `run`, `create`, `mock`, - `import`, `repl`, `summary`, `scaffold`, `completion`, `mcp`, `openapi` and `contract record|verify|compare` + `import`, `repl`, `summary`, `scaffold`, `completion`, `mcp`, `openapi`, `contract record|verify|compare` and + `generate-tests` (`cli/contract_cli.py`); shared helpers are in `cli/cli_common.py`. `cli/completion.py` `SUBCOMMANDS` lists the same names (a test compares them with the parser). - **MCP**: `apitestka-mcp`, `python -m je_api_testka.mcp_server` or `apitestka mcp` (stdio, needs the @@ -147,7 +148,7 @@ MCP host → apitestka-mcp → build_server → dispatch_tool(name, args) → AP justification comment. No `shell=True` (§ Security (Mandatory); § Static Analysis Compliance › Security). - Extend through `add_command_to_executor` or new `AT_` commands rather than reshaping the core map (§ Software Engineering Practices; § Common Development Workflows › Adding a New Executor Command). -- Import heavy optional dependencies lazily (PySide6, websockets, grpcio, jsonschema, mcp). +- Import heavy optional dependencies lazily (PySide6, websockets, grpcio, anthropic, jsonschema, mcp). `test_record_instance` must stay thread-safe (§ Performance Best Practices). - Limits: cognitive complexity ≤ 15, cyclomatic complexity ≤ 10, ≤ 7 parameters, functions ≤ 50 lines, files ≤ 500 lines, lines ≤ 120 characters (§ Static Analysis Compliance › Maintainability & Complexity). diff --git a/dev.toml b/dev.toml index 4842962..7fefb48 100644 --- a/dev.toml +++ b/dev.toml @@ -52,6 +52,8 @@ apitestka = "je_api_testka.pytest_plugin.plugin" gui = ["PySide6==6.11.2", "qt-material"] websocket = ["websockets>=12.0"] grpc = ["grpcio>=1.60"] +# ai/anthropic_backend.py (AnthropicAIBackend); anthropic 1.x is the httpx2-based SDK. +ai = ["anthropic>=1.0,<2"] schema = ["jsonschema>=4.0", "jsonpath-ng>=1.6"] security = ["pyjwt>=2.8", "botocore>=1.34"] otel = ["opentelemetry-api>=1.25", "opentelemetry-sdk>=1.25"] diff --git a/docs/source/Eng/Eng_index.rst b/docs/source/Eng/Eng_index.rst index 449f1d9..5460f5f 100644 --- a/docs/source/Eng/Eng_index.rst +++ b/docs/source/Eng/Eng_index.rst @@ -52,7 +52,7 @@ Features bidirectional check against OpenAPI * - **AI** - Pluggable backend with deterministic fallback for test generation, - fake data, failure classification + fake data, failure classification; Anthropic reference backend * - **MCP** - First-class Claude Code / MCP server exposing the framework as tools * - **GUI** @@ -85,3 +85,4 @@ Features doc/project/project_doc doc/test_record/test_record_doc doc/mcp/mcp_doc + doc/ai/ai_doc diff --git a/docs/source/Eng/doc/ai/ai_doc.rst b/docs/source/Eng/doc/ai/ai_doc.rst new file mode 100644 index 0000000..db52424 --- /dev/null +++ b/docs/source/Eng/doc/ai/ai_doc.rst @@ -0,0 +1,53 @@ +==================== +Pluggable AI Backend +==================== + +Three helpers in ``je_api_testka.ai`` can ask a language model: + +* ``generate_tests_from_openapi(spec)``: executor actions for every operation. A reply is used only when + it is a non-empty JSON list of ``["AT_...", {...}]`` actions. +* ``generate_fake_payload(schema)``: a JSON value that satisfies a JSON Schema. +* ``classify_failures(error_records)``: rules label each failure (``network``, ``auth``, + ``validation``, ``server``, ``other``); the ones left as ``other`` go to the model in one request. + +The default backend, ``NoOpAIBackend``, never calls a network, and every helper then uses its +deterministic fallback. It does the same when a reply is empty or unusable. A reply wrapped in one +Markdown code fence is accepted. + +Anthropic reference backend +--------------------------- + +``AnthropicAIBackend`` needs the ``ai`` extra (``pip install 'je_api_testka[ai]'``). It sends each request +to the Anthropic Messages API and returns the reply text. + +* Credentials come from the environment the ``anthropic`` SDK reads (``ANTHROPIC_API_KEY``, + ``ANTHROPIC_AUTH_TOKEN`` or an ``ant auth login`` profile). The backend never takes or stores a key. +* The default model is ``claude-opus-5-5``. Its thinking is always on, so depth is set with ``effort`` + (``low`` to ``max``); the backend sends ``medium`` explicitly. +* Server-side refusal fallbacks are on (``fallbacks="default"``): when a safety classifier declines, + the API re-runs the request on the fallback model recommended for that refusal category. Pass + ``fallbacks=None`` to turn it off. +* A refusal, a reply cut off at ``max_tokens``, a rate limit, a server error or a network failure + returns an empty reply (the SDK has already retried the transient ones), so the helper uses its + fallback. A rejected request (bad key, unknown model, invalid parameter) raises + ``APIAIBackendException``. + +.. code-block:: python + + from je_api_testka.ai import AnthropicAIBackend, set_ai_backend + + set_ai_backend(AnthropicAIBackend()) + set_ai_backend(AnthropicAIBackend(model="claude-sonnet-5-5", effort="low", timeout=60)) + +Selecting a backend +------------------- + +* ``set_ai_backend(backend)`` or ``select_ai_backend("anthropic", model=..., effort=...)``; +* the ``AT_select_ai_backend`` action: ``["AT_select_ai_backend", {"name": "anthropic"}]``; +* the environment: without a selection, the first use reads ``APITESTKA_AI_BACKEND`` (``noop`` or + ``anthropic``), ``APITESTKA_AI_MODEL`` and ``APITESTKA_AI_EFFORT``, so the CLI, the MCP server and the + socket server use a model without code changes; +* ``apitestka generate-tests openapi.json -o actions.json --ai anthropic [--model M] [--effort E]``. + +Other providers plug in by subclassing ``AIBackend`` and implementing ``complete(prompt, *, context)``; +return an empty string for "no answer". diff --git a/docs/source/Eng/doc/cli/cli_doc.rst b/docs/source/Eng/doc/cli/cli_doc.rst index b5f080f..3110aa4 100644 --- a/docs/source/Eng/doc/cli/cli_doc.rst +++ b/docs/source/Eng/doc/cli/cli_doc.rst @@ -82,6 +82,8 @@ The ``apitestka`` command (installed with the package) groups the everyday tasks - Infer an OpenAPI document from recorded traffic (below) * - ``contract record|verify|compare`` - Record, verify and compare Pact-style consumer contracts (see Contract Testing) + * - ``generate-tests SPEC [-o OUT] [--ai noop|anthropic] [--model] [--effort]`` + - Write test actions for an OpenAPI document (see Pluggable AI Backend) Inferring an OpenAPI document ----------------------------- diff --git a/docs/source/Zh/Zh_index.rst b/docs/source/Zh/Zh_index.rst index af41ae4..fc1a9c9 100644 --- a/docs/source/Zh/Zh_index.rst +++ b/docs/source/Zh/Zh_index.rst @@ -48,7 +48,7 @@ HAR / cURL 匯入器、record-replay proxy、安全檢測、平行 runner、給 * - **契約測試** - 從測試執行產生 Pact v2 消費者契約、提供端驗證、與 OpenAPI 雙向比對 * - **AI** - - 可插拔後端,LLM 不可用時自動退回確定性 fallback + - 可插拔後端,LLM 不可用時自動退回確定性 fallback;附 Anthropic 參考實作 * - **MCP** - 一級支援 Claude Code,將框架曝露為 MCP 工具 * - **GUI** @@ -81,3 +81,4 @@ HAR / cURL 匯入器、record-replay proxy、安全檢測、平行 runner、給 doc/project/project_doc doc/test_record/test_record_doc doc/mcp/mcp_doc + doc/ai/ai_doc diff --git a/docs/source/Zh/doc/ai/ai_doc.rst b/docs/source/Zh/doc/ai/ai_doc.rst new file mode 100644 index 0000000..e7fcefb --- /dev/null +++ b/docs/source/Zh/doc/ai/ai_doc.rst @@ -0,0 +1,47 @@ +================ +可插拔 AI 後端 +================ + +``je_api_testka.ai`` 裡有三個工具會詢問語言模型： + +* ``generate_tests_from_openapi(spec)``：為每個 operation 產生 executor action。只有回覆是非空的 + ``["AT_...", {...}]`` JSON list 時才採用。 +* ``generate_fake_payload(schema)``：產生符合 JSON Schema 的 JSON 值。 +* ``classify_failures(error_records)``：先用規則替失敗貼標籤（``network``、``auth``、``validation``、 + ``server``、``other``），留在 ``other`` 的再一次送給模型。 + +預設的 ``NoOpAIBackend`` 不會碰網路，各工具會改用確定性的 fallback；回覆是空的或無法使用時也一樣。 +回覆外面包了一層 Markdown code fence 也可以接受。 + +Anthropic 參考實作 +------------------ + +``AnthropicAIBackend`` 需要 ``ai`` extra（``pip install 'je_api_testka[ai]'``），把每個請求送到 +Anthropic Messages API 並回傳回覆文字。 + +* 憑證取自 ``anthropic`` SDK 讀的環境（``ANTHROPIC_API_KEY``、``ANTHROPIC_AUTH_TOKEN`` 或 + ``ant auth login`` 設定檔），backend 不接收也不保存金鑰。 +* 預設模型是 ``claude-opus-5-5``。它的 thinking 一律開啟，深度用 ``effort``（``low`` 到 ``max``）控制； + backend 會明確送出 ``medium``。 +* 伺服器端的拒答 fallback 預設開啟（``fallbacks="default"``）：安全分類器拒答時，API 會改用針對該拒答類別 + 建議的 fallback 模型重跑。傳 ``fallbacks=None`` 可關閉。 +* 拒答、回覆在 ``max_tokens`` 被截斷、速率限制、伺服器錯誤或網路失敗時回傳空字串（暫時性錯誤 SDK 已經重試過）， + 工具就改用 fallback。請求被拒（金鑰錯誤、未知模型、參數不合法）則丟出 ``APIAIBackendException``。 + +.. code-block:: python + + from je_api_testka.ai import AnthropicAIBackend, set_ai_backend + + set_ai_backend(AnthropicAIBackend()) + set_ai_backend(AnthropicAIBackend(model="claude-sonnet-5-5", effort="low", timeout=60)) + +選擇 backend +------------ + +* ``set_ai_backend(backend)`` 或 ``select_ai_backend("anthropic", model=..., effort=...)``； +* ``AT_select_ai_backend`` action：``["AT_select_ai_backend", {"name": "anthropic"}]``； +* 環境變數：沒有明確選擇時，第一次使用會讀 ``APITESTKA_AI_BACKEND``（``noop`` 或 ``anthropic``）、 + ``APITESTKA_AI_MODEL`` 與 ``APITESTKA_AI_EFFORT``，所以 CLI、MCP server 與 socket server 不用改程式就能用模型； +* ``apitestka generate-tests openapi.json -o actions.json --ai anthropic [--model M] [--effort E]``。 + +其他 provider 可繼承 ``AIBackend`` 並實作 ``complete(prompt, *, context)`` 接上；沒有答案時回傳空字串。 diff --git a/docs/source/Zh/doc/cli/cli_doc.rst b/docs/source/Zh/doc/cli/cli_doc.rst index 98f547e..4632562 100644 --- a/docs/source/Zh/doc/cli/cli_doc.rst +++ b/docs/source/Zh/doc/cli/cli_doc.rst @@ -82,6 +82,8 @@ CLI 參數 - 從錄下的流量反推 OpenAPI 文件（見下節） * - ``contract record|verify|compare`` - 產生、驗證與比對 Pact 風格消費者契約（見契約測試一章） + * - ``generate-tests SPEC [-o OUT] [--ai noop|anthropic] [--model] [--effort]`` + - 為 OpenAPI 文件產生測試 action（見可插拔 AI 後端一章） 反推 OpenAPI 文件 ---------------------- diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 52d66e7..5533011 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -50,3 +50,17 @@ Index and query commands: [README.md](README.md). New entries go at the end. - **Tests**: new `test/test_contract/` (pact, matching, schema and compatibility, verifier against a Flask provider on a free port, commands, CLI and executor). It includes an end-to-end flow: the consumer run becomes a contract, the provider run becomes its OpenAPI document, the two are compatible, and the contract verifies against the live provider. - **Result / numbers**: `pytest` 587 passed, 2 skipped (522 before); `sphinx -W` clean. - **Docs**: README ×3 (highlights, table of contents, new Contract Testing section, CLI, project tree; the Mock server highlight also gains the #8 WebSocket and gRPC endpoints), new `docs/source/{Eng,Zh}/doc/contract/contract_doc.rst`, CLI tables, `architecture.md` §2–§3, `CLAUDE.md` package tree. + +## U-20261001-05 · 2026-10-01 · Anthropic reference AI backend and backend selection · #done #ai + +- **What** (`progress.md` #10): `ai/anthropic_backend.py` `AnthropicAIBackend`, the reference `AIBackend` on the Anthropic Messages API (`client.beta.messages.create`, SDK `anthropic` 1.x). + - **Request**: the default model is `claude-opus-5-5` with `output_config.effort` `medium` sent explicitly (it is that model's default, and thinking cannot be turned off there, so `thinking` is not sent). `max_tokens` 16000, non-streaming. Server-side refusal fallbacks are on (`fallbacks="default"` with beta `server-side-fallback-2026-07-01`); `fallbacks=None` turns them off. The system prompt asks for the artefact only; the context is serialized with sorted keys. + - **Credentials**: whatever the SDK resolves from the environment (`ANTHROPIC_API_KEY`, `ANTHROPIC_AUTH_TOKEN`, an `ant auth login` profile). The backend has no key parameter, and a ready client can be injected. + - **Errors**: a refusal, a `max_tokens` cut-off, a rate limit, a 5xx or a connection/timeout error returns `""`, so the helper uses its deterministic fallback; the SDK has already retried the transient ones. 4xx rejections (auth, not found, bad request) raise the new `APIAIBackendException`. +- **Selection**: `build_ai_backend(name)` / `select_ai_backend(name, model, effort)` (`noop`, `anthropic`), the `AT_select_ai_backend` command, `apitestka generate-tests SPEC [-o OUT] [--ai] [--model] [--effort]`, and the environment: without a selection, the first `ai_backend()` call reads `APITESTKA_AI_BACKEND`, `APITESTKA_AI_MODEL` and `APITESTKA_AI_EFFORT`, so the CLI, the MCP server and the socket server use a model without code changes. +- **Callers**: `generate_tests_from_openapi` sends a prompt that describes the `["AT_test_api_method", {...}]` format; it used to say only "Generate APITestka action JSON", which a model cannot follow. It accepts only a non-empty list of `["AT_...", {...}]` actions. `generate_fake_payload` has a clearer prompt. `classify_failures` never used the backend, although `architecture.md` said it did; it now sends the errors its rules leave as `other` in one request and keeps them as `other` unless the reply is a JSON list of known labels, one per message. The new `ai/reply.parse_json_reply` accepts one Markdown code fence. +- **Packaging**: new `ai` extra `anthropic>=1.0,<2` in `pyproject.toml` and `dev.toml` (1.0.0 already accepts `fallbacks="default"`). `anthropic` 1.9.0 joins `.github/requirements/ci.{in,txt}` (plus `jiter`, `docstring-parser`, `sniffio`), so CI runs the backend tests, and it joins `uv.lock`. +- **Tests**: `test/test_ai/test_anthropic_backend.py` (a fake client: request shape, text extraction, refusal and `max_tokens`, the error chain built from real SDK exception classes, lazy client with timeout, missing SDK, test generation through the backend); `test/test_ai/test_backend_selection.py` (names, environment, executor, classifier and fake-payload paths, reply parsing, the `generate-tests` CLI). Also run on Python 3.10 with `anthropic` 1.0.0: `test/test_ai` 41 passed. +- **Not verified**: no call to the live API; this machine has no Anthropic credentials. +- **Result / numbers**: `pytest` 618 passed, 2 skipped (587 before); `sphinx -W` clean. +- **Docs**: README ×3 (the Pluggable AI Backend section rewritten, CLI, extras, highlights), new `docs/source/{Eng,Zh}/doc/ai/ai_doc.rst`, CLI tables, `architecture.md` §2, §3, §7, `CLAUDE.md` package tree. diff --git a/docs/updates/README.md b/docs/updates/README.md index 5115db7..ebae168 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261001-05 | 2026-10-01 | Anthropic reference AI backend and backend selection | #done #ai | [2026-10](2026-10.md) | | U-20261001-04 | 2026-10-01 | Pact-style bidirectional contract testing | #done #contract | [2026-10](2026-10.md) | | U-20261001-03 | 2026-10-01 | Mock server serves WebSocket and gRPC endpoints | #done #mock #grpc #websocket | [2026-10](2026-10.md) | | U-20261001-02 | 2026-10-01 | Imported, scaffolded and AI-generated actions run as they are | #bugfix #executor | [2026-10](2026-10.md) | @@ -90,5 +91,5 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 4 | +| [2026-10.md](2026-10.md) | 2026-10 | 5 | | [2026-09.md](2026-09.md) | 2026-09 | 23 | diff --git a/je_api_testka/ai/__init__.py b/je_api_testka/ai/__init__.py index 166f452..113e2ac 100644 --- a/je_api_testka/ai/__init__.py +++ b/je_api_testka/ai/__init__.py @@ -1,9 +1,12 @@ +from je_api_testka.ai.anthropic_backend import AnthropicAIBackend from je_api_testka.ai.backend import ( AIBackend, NoOpAIBackend, StaticAIBackend, - set_ai_backend, ai_backend, + build_ai_backend, + select_ai_backend, + set_ai_backend, ) from je_api_testka.ai.failure_classifier import classify_failures from je_api_testka.ai.fake_data_generator import generate_fake_payload @@ -11,11 +14,14 @@ __all__ = [ "AIBackend", + "AnthropicAIBackend", "NoOpAIBackend", "StaticAIBackend", "ai_backend", + "build_ai_backend", "classify_failures", "generate_fake_payload", "generate_tests_from_openapi", + "select_ai_backend", "set_ai_backend", ] diff --git a/je_api_testka/ai/anthropic_backend.py b/je_api_testka/ai/anthropic_backend.py new file mode 100644 index 0000000..e499d1c --- /dev/null +++ b/je_api_testka/ai/anthropic_backend.py @@ -0,0 +1,149 @@ +""" +Reference AI backend on the Anthropic API (``pip install je_api_testka[ai]``). + +:class:`AnthropicAIBackend` sends each :meth:`~AnthropicAIBackend.complete` +call as one Messages API request and returns the reply text. + +* Credentials come from the environment the ``anthropic`` SDK reads + (``ANTHROPIC_API_KEY``, ``ANTHROPIC_AUTH_TOKEN`` or an ``ant auth login`` + profile); the backend never takes or stores a key. +* The default model is ``claude-opus-5-5``. Its thinking is always on, so depth + is set with ``effort`` (``low`` to ``max``); the backend sends ``medium`` + explicitly unless told otherwise. +* Server-side refusal fallbacks are on (``fallbacks="default"``): when a safety + classifier declines, the API re-runs the request on the fallback model + Anthropic recommends for that refusal category. Pass ``fallbacks=None`` to + turn it off. +* A refusal, a truncated reply (``max_tokens``), a rate limit, a server error + or a network failure returns ``""``, so the caller uses its deterministic + fallback; the SDK already retried the transient ones. A rejected request + (bad key, unknown model, invalid parameter) raises + :class:`APIAIBackendException`, because retrying cannot fix it. +""" +from __future__ import annotations + +import json +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any, List, Optional + +from je_api_testka.ai.backend import AIBackend +from je_api_testka.utils.exception.exceptions import APIAIBackendException +from je_api_testka.utils.logging.loggin_instance import apitestka_logger + +if TYPE_CHECKING: # the SDK is optional; imported for annotations only + import anthropic + +DEFAULT_ANTHROPIC_MODEL: str = "claude-opus-5-5" +DEFAULT_ANTHROPIC_EFFORT: str = "medium" +DEFAULT_ANTHROPIC_MAX_TOKENS: int = 16000 +DEFAULT_FALLBACKS: str = "default" +FALLBACK_BETA: str = "server-side-fallback-2026-07-01" +ANTHROPIC_MISSING: str = "anthropic is not installed. Install with `pip install je_api_testka[ai]`." +_SERVER_ERROR_STATUS: int = 500 + +SYSTEM_PROMPT: str = ( + "You are the text-completion backend of APITestka, an API testing framework. " + "Each request asks for one artefact, such as executor actions, a JSON value that satisfies a schema, " + "or labels for failed requests; the request explains the format and any context follows in a " + " block. Reply with the artefact only. When JSON is asked for, reply with exactly one JSON " + "value, without explanations and without Markdown code fences." +) + + +def _import_anthropic(): + try: + import anthropic # type: ignore + except ImportError as error: + raise APIAIBackendException(ANTHROPIC_MISSING) from error + return anthropic + + +def _user_message(prompt: str, context: Optional[dict]) -> str: + if not context: + return prompt + # sort_keys keeps identical contexts byte-identical, which keeps them cacheable. + rendered = json.dumps(context, sort_keys=True, ensure_ascii=False, default=str) + return f"{prompt}\n\n\n{rendered}\n" + + +def _reply_text(response: Any) -> str: + return "".join(block.text for block in response.content if getattr(block, "type", "") == "text") + + +@dataclass +class AnthropicAIBackend(AIBackend): + """ + :class:`AIBackend` on the Anthropic Messages API. + + :param model: model ID (default ``claude-opus-5-5``). + :param effort: ``output_config.effort`` (``low``, ``medium``, ``high``, ``xhigh``, ``max``); ``None`` omits it. + :param max_tokens: output cap per reply, thinking included. + :param fallbacks: server-side refusal fallback mode; ``None`` turns it off. + :param timeout: request timeout in seconds; ``None`` keeps the SDK default. + :param client: a ready ``anthropic.Anthropic`` client (tests, proxies); built on first use when ``None``. + """ + + model: str = DEFAULT_ANTHROPIC_MODEL + effort: Optional[str] = DEFAULT_ANTHROPIC_EFFORT + max_tokens: int = DEFAULT_ANTHROPIC_MAX_TOKENS + fallbacks: Optional[str] = DEFAULT_FALLBACKS + timeout: Optional[float] = None + client: Optional["anthropic.Anthropic"] = None + + def _client(self) -> "anthropic.Anthropic": + if self.client is None: + anthropic = _import_anthropic() + options = {} if self.timeout is None else {"timeout": self.timeout} + self.client = anthropic.Anthropic(**options) + return self.client + + def request_options(self, prompt: str, context: Optional[dict] = None) -> dict: + """Return the keyword arguments :meth:`complete` passes to ``client.beta.messages.create``.""" + options: dict = { + "model": self.model, + "max_tokens": self.max_tokens, + "system": SYSTEM_PROMPT, + "messages": [{"role": "user", "content": _user_message(prompt, context)}], + } + if self.effort: + options["output_config"] = {"effort": self.effort} + betas: List[str] = [] + if self.fallbacks: + options["fallbacks"] = self.fallbacks + betas.append(FALLBACK_BETA) + if betas: + options["betas"] = betas + return options + + def complete(self, prompt: str, *, context: Optional[dict] = None) -> str: + """Return the model's reply text, or ``""`` when there is no usable reply (see the module notes).""" + anthropic = _import_anthropic() + try: + response = self._client().beta.messages.create(**self.request_options(prompt, context)) + except anthropic.RateLimitError as error: + apitestka_logger.error(f"AnthropicAIBackend rate limited after SDK retries: {error.message}") + return "" + except anthropic.APIStatusError as error: + if error.status_code >= _SERVER_ERROR_STATUS: + apitestka_logger.error(f"AnthropicAIBackend server error {error.status_code}: {error.message}") + return "" + raise APIAIBackendException( + f"Anthropic API rejected the request ({error.status_code}): {error.message}") from error + except anthropic.APIConnectionError as error: + apitestka_logger.error(f"AnthropicAIBackend could not reach the API: {error!r}") + return "" + return self._usable_text(response) + + def _usable_text(self, response: Any) -> str: + request_id = getattr(response, "_request_id", None) + apitestka_logger.info( + f"AnthropicAIBackend {getattr(response, 'model', self.model)} stop_reason={response.stop_reason} " + f"request_id={request_id}") + if response.stop_reason == "refusal": + category = getattr(getattr(response, "stop_details", None), "category", None) + apitestka_logger.error(f"AnthropicAIBackend: the request was declined (category {category})") + return "" + if response.stop_reason == "max_tokens": + apitestka_logger.error(f"AnthropicAIBackend: reply cut off at max_tokens={self.max_tokens}") + return "" + return _reply_text(response) diff --git a/je_api_testka/ai/backend.py b/je_api_testka/ai/backend.py index 7e649e5..2c031b1 100644 --- a/je_api_testka/ai/backend.py +++ b/je_api_testka/ai/backend.py @@ -1,16 +1,38 @@ """ Pluggable AI backend. -* :class:`NoOpAIBackend` is the default - it never calls a network or LLM. +* :class:`NoOpAIBackend` is the default - it never calls a network or LLM, and + every AI helper then uses its deterministic fallback. * :class:`StaticAIBackend` returns canned responses, useful for tests. -* Users plug in their own subclass via :func:`set_ai_backend`. The plumbing - for actual Anthropic / OpenAI / local LLM calls is intentionally outside - this package. +* :class:`~je_api_testka.ai.anthropic_backend.AnthropicAIBackend` is the + reference implementation on the Anthropic API (``ai`` extra). +* Users plug in their own subclass via :func:`set_ai_backend`. + +A backend answers with plain text; an empty string means "no answer", and the +callers then fall back to their deterministic behaviour. + +Selection: :func:`set_ai_backend` or :func:`select_ai_backend` (also the +``AT_select_ai_backend`` command). When neither was called, the first +:func:`ai_backend` call reads ``APITESTKA_AI_BACKEND`` (``noop`` or +``anthropic``; ``APITESTKA_AI_MODEL`` and ``APITESTKA_AI_EFFORT`` tune the +Anthropic backend), so the CLI, the MCP server and the socket server can use a +real model without code changes. """ from __future__ import annotations +import os +import threading from dataclasses import dataclass -from typing import Optional +from typing import Dict, Optional + +from je_api_testka.utils.exception.exceptions import APIAIBackendException + +AI_BACKEND_ENV: str = "APITESTKA_AI_BACKEND" +AI_MODEL_ENV: str = "APITESTKA_AI_MODEL" +AI_EFFORT_ENV: str = "APITESTKA_AI_EFFORT" +NOOP_BACKEND: str = "noop" +ANTHROPIC_BACKEND: str = "anthropic" +BACKEND_NAMES = (NOOP_BACKEND, ANTHROPIC_BACKEND) class AIBackend: @@ -37,17 +59,59 @@ def complete(self, prompt: str, *, context: Optional[dict] = None) -> str: # no return self.response -_active_backend: AIBackend = NoOpAIBackend() +# The active backend; empty until set or first read from the environment. +_selection: Dict[str, AIBackend] = {} +_lock = threading.Lock() +_ACTIVE: str = "active" + + +def build_ai_backend(name: str = NOOP_BACKEND, model: Optional[str] = None, + effort: Optional[str] = None) -> AIBackend: + """ + Return a new backend by name: ``noop`` or ``anthropic``. + + ``model`` and ``effort`` apply to the Anthropic backend (its defaults when ``None``). + + :raises APIAIBackendException: for an unknown name, or options given to ``noop``. + """ + if name == NOOP_BACKEND: + if model or effort: + raise APIAIBackendException("the noop AI backend takes no model or effort") + return NoOpAIBackend() + if name == ANTHROPIC_BACKEND: + from je_api_testka.ai.anthropic_backend import AnthropicAIBackend # the module imports this one + options = {key: value for key, value in (("model", model), ("effort", effort)) if value} + return AnthropicAIBackend(**options) + raise APIAIBackendException(f"unknown AI backend {name!r}; expected one of {BACKEND_NAMES}") + + +def _backend_from_environment() -> AIBackend: + name = os.environ.get(AI_BACKEND_ENV, "").strip().lower() or NOOP_BACKEND + if name == NOOP_BACKEND: + return NoOpAIBackend() + return build_ai_backend(name, model=os.environ.get(AI_MODEL_ENV) or None, + effort=os.environ.get(AI_EFFORT_ENV) or None) def ai_backend() -> AIBackend: - """Return the currently registered backend.""" - return _active_backend + """Return the active backend; on first use without a selection, build it from the environment.""" + with _lock: + if _ACTIVE not in _selection: + _selection[_ACTIVE] = _backend_from_environment() + return _selection[_ACTIVE] def set_ai_backend(backend: AIBackend) -> None: """Replace the active backend (e.g. install a real LLM-backed one).""" - global _active_backend if not isinstance(backend, AIBackend): raise TypeError("backend must be an AIBackend subclass") - _active_backend = backend + with _lock: + _selection[_ACTIVE] = backend + + +def select_ai_backend(name: str = NOOP_BACKEND, model: Optional[str] = None, + effort: Optional[str] = None) -> str: + """Build a backend with :func:`build_ai_backend`, make it active and return its class name.""" + backend = build_ai_backend(name, model=model, effort=effort) + set_ai_backend(backend) + return type(backend).__name__ diff --git a/je_api_testka/ai/failure_classifier.py b/je_api_testka/ai/failure_classifier.py index 610324a..6ce30d2 100644 --- a/je_api_testka/ai/failure_classifier.py +++ b/je_api_testka/ai/failure_classifier.py @@ -1,24 +1,41 @@ """ Lightweight failure classifier. -We keep this rule-based by default so it works without an LLM. Categories: +Rules classify each failure first, so this works without an LLM. Categories: * ``network``: ``ConnectError`` / ``Timeout`` / DNS issues. * ``auth``: 401 / 403 / "unauthorized" / "forbidden" in the message. * ``validation``: 400 / 422 / "invalid" / schema mismatches. * ``server``: 5xx. * ``other``: anything that did not match. + +When an AI backend other than :class:`NoOpAIBackend` is active, the messages +the rules leave as ``other`` go to it in one request; a reply that is not a +JSON list of known categories, one per message, leaves them as ``other``. """ from __future__ import annotations +import json from collections import Counter -from typing import Iterable +from typing import Iterable, List + +from je_api_testka.ai.backend import NoOpAIBackend, ai_backend +from je_api_testka.ai.reply import parse_json_reply +from je_api_testka.utils.logging.loggin_instance import apitestka_logger CATEGORY_NETWORK: str = "network" CATEGORY_AUTH: str = "auth" CATEGORY_VALIDATION: str = "validation" CATEGORY_SERVER: str = "server" CATEGORY_OTHER: str = "other" +CATEGORIES = (CATEGORY_NETWORK, CATEGORY_AUTH, CATEGORY_VALIDATION, CATEGORY_SERVER, CATEGORY_OTHER) + +CLASSIFICATION_PROMPT: str = ( + "Each context entry is the error message of a failed API test request. Label each one with exactly one " + f"of: {', '.join(CATEGORIES)} (network: connection, timeout or DNS; auth: authentication or permission; " + "validation: the request was rejected as malformed or invalid; server: the server failed; other: none of " + "these). Reply with a JSON array of labels in the same order as the messages." +) def _classify_one(error_text: str) -> str: @@ -34,10 +51,33 @@ def _classify_one(error_text: str) -> str: return CATEGORY_OTHER +def _ai_labels(messages: List[str]) -> List[str]: + backend = ai_backend() + if not messages or isinstance(backend, NoOpAIBackend): + return [CATEGORY_OTHER] * len(messages) + reply = backend.complete(CLASSIFICATION_PROMPT, context={"messages": messages}) + try: + labels = parse_json_reply(reply) if reply else None + except json.JSONDecodeError: + labels = None + if (isinstance(labels, list) and len(labels) == len(messages) + and all(label in CATEGORIES for label in labels)): + return labels + if reply: + apitestka_logger.error("AI backend returned unusable failure labels; keeping them as 'other'") + return [CATEGORY_OTHER] * len(messages) + + def classify_failures(error_records: Iterable[list]) -> Counter: """Bucket each error record into one of the categories above; return a counter.""" counter: Counter = Counter() + unmatched: List[str] = [] for entry in error_records: - message = entry[1] if len(entry) > 1 else "" - counter[_classify_one(str(message))] += 1 + message = str(entry[1] if len(entry) > 1 else "") + category = _classify_one(message) + if category == CATEGORY_OTHER: + unmatched.append(message) + else: + counter[category] += 1 + counter.update(_ai_labels(unmatched)) return counter diff --git a/je_api_testka/ai/fake_data_generator.py b/je_api_testka/ai/fake_data_generator.py index 504cc57..44e607d 100644 --- a/je_api_testka/ai/fake_data_generator.py +++ b/je_api_testka/ai/fake_data_generator.py @@ -10,6 +10,7 @@ from typing import Any from je_api_testka.ai.backend import NoOpAIBackend, ai_backend +from je_api_testka.ai.reply import parse_json_reply from je_api_testka.data.faker_helpers import fake_email, fake_uuid, fake_word from je_api_testka.utils.logging.loggin_instance import apitestka_logger @@ -42,13 +43,13 @@ def generate_fake_payload(schema: dict) -> Any: if isinstance(backend, NoOpAIBackend): return _deterministic(schema) response = backend.complete( - "Produce a JSON value that satisfies this schema.", + "Reply with one realistic JSON value that satisfies the JSON Schema in the context.", context={"schema": schema}, ) if not response: return _deterministic(schema) try: - return json.loads(response) + return parse_json_reply(response) except json.JSONDecodeError: apitestka_logger.error("AI backend returned non-JSON for fake payload; falling back") return _deterministic(schema) diff --git a/je_api_testka/ai/reply.py b/je_api_testka/ai/reply.py new file mode 100644 index 0000000..d453a4e --- /dev/null +++ b/je_api_testka/ai/reply.py @@ -0,0 +1,21 @@ +"""Read JSON out of a model's text reply.""" +from __future__ import annotations + +import json +import re +from typing import Any + +_FENCE = re.compile(r"^```[A-Za-z0-9_-]*\s*\n(.*?)\n?```$", re.DOTALL) + + +def parse_json_reply(reply: str) -> Any: + """ + Parse ``reply`` as JSON, ignoring surrounding whitespace and one Markdown code fence. + + :raises json.JSONDecodeError: when the reply is not JSON. + """ + text = reply.strip() + fenced = _FENCE.match(text) + if fenced: + text = fenced.group(1).strip() + return json.loads(text) diff --git a/je_api_testka/ai/test_generator.py b/je_api_testka/ai/test_generator.py index 3ff8546..efe8128 100644 --- a/je_api_testka/ai/test_generator.py +++ b/je_api_testka/ai/test_generator.py @@ -1,8 +1,9 @@ """ Generate executor action JSON from an OpenAPI spec via the active AI backend. -If the backend is :class:`NoOpAIBackend` we fall back to a deterministic -``records_to_openapi``-style approach: one happy-path action per operation. +If the backend is :class:`NoOpAIBackend`, or its reply is empty, not JSON or +not a list of well-formed ``AT_*`` actions, we fall back to a deterministic +approach: one happy-path action per operation. """ from __future__ import annotations @@ -10,10 +11,23 @@ from typing import Any, List from je_api_testka.ai.backend import NoOpAIBackend, ai_backend -from je_api_testka.utils.executor.request_action import build_request_action +from je_api_testka.ai.reply import parse_json_reply +from je_api_testka.utils.executor.request_action import REQUEST_COMMAND, build_request_action from je_api_testka.utils.logging.loggin_instance import apitestka_logger HTTP_VERBS = ("get", "put", "post", "patch", "delete", "options", "head") +COMMAND_PREFIX: str = "AT_" +_ACTION_LENGTH: int = 2 # [command, kwargs or args] + +TEST_GENERATION_PROMPT: str = ( + "Write APITestka test actions for the OpenAPI document in the context. Reply with a JSON array; " + f'each element is one action of the form ["{REQUEST_COMMAND}", {{...}}] whose object takes: ' + '"http_method" (lower-case verb), "test_url" (first server URL + path, with example values in place ' + 'of path parameters), optional "params" (query parameters), "headers", "json" (request body) and ' + '"result_check_dict" (e.g. {"status_code": 200}; APITestka only records 2xx and 3xx responses as ' + "successes). Cover every operation with a realistic happy-path call that uses the documented examples " + "and required fields." +) def _deterministic_actions(spec: dict) -> List[list]: @@ -30,27 +44,34 @@ def _deterministic_actions(spec: dict) -> List[list]: return actions +def _is_action(value: Any) -> bool: + if not isinstance(value, list) or not value or len(value) > _ACTION_LENGTH: + return False + name = value[0] + if not isinstance(name, str) or not name.startswith(COMMAND_PREFIX): + return False + return len(value) == 1 or isinstance(value[1], (dict, list)) + + def generate_tests_from_openapi(spec: dict) -> List[Any]: """ Return executor actions via the AI backend if configured, otherwise a deterministic fallback. The fallback is one ``["AT_test_api_method", {...}]`` happy-path action per operation. - A backend reply that is a JSON list is returned as it is. + A backend reply is used only when it is a non-empty JSON list of ``["AT_...", {...}]`` actions. """ backend = ai_backend() if isinstance(backend, NoOpAIBackend): return _deterministic_actions(spec) - response = backend.complete( - "Generate APITestka action JSON for the following OpenAPI document.", - context={"spec": spec}, - ) + response = backend.complete(TEST_GENERATION_PROMPT, context={"openapi": spec}) if not response: return _deterministic_actions(spec) try: - parsed: Any = json.loads(response) + parsed: Any = parse_json_reply(response) except json.JSONDecodeError: apitestka_logger.error("AI backend returned non-JSON; falling back") return _deterministic_actions(spec) - if isinstance(parsed, list): + if isinstance(parsed, list) and parsed and all(_is_action(action) for action in parsed): return parsed + apitestka_logger.error("AI backend reply is not a list of AT_* actions; falling back") return _deterministic_actions(spec) diff --git a/je_api_testka/cli/cli_main.py b/je_api_testka/cli/cli_main.py index 36c8d47..b96ea0f 100644 --- a/je_api_testka/cli/cli_main.py +++ b/je_api_testka/cli/cli_main.py @@ -13,6 +13,7 @@ mcp Run the MCP server over stdio. openapi Infer an OpenAPI document from recorded traffic. contract Record, verify and compare Pact-style consumer contracts. + generate-tests Write test actions for an OpenAPI document (AI backend or deterministic). """ from __future__ import annotations @@ -22,10 +23,14 @@ from pathlib import Path from typing import Optional, Sequence +from je_api_testka.ai.backend import BACKEND_NAMES, select_ai_backend +from je_api_testka.ai.test_generator import generate_tests_from_openapi from je_api_testka.cli.cli_common import run_action_path from je_api_testka.cli.contract_cli import configure_contract_parser +from je_api_testka.contract.commands import read_openapi from je_api_testka.spec.openapi_export import build_openapi from je_api_testka.spec.records_to_openapi import DEFAULT_SPEC_TITLE, DEFAULT_SPEC_VERSION +from je_api_testka.utils.exception.exceptions import APIAIBackendException, APIContractException from je_api_testka.utils.logging.loggin_instance import apitestka_logger from je_api_testka.utils.project.create_project_structure import create_project_dir @@ -122,6 +127,33 @@ def _cmd_openapi(args: argparse.Namespace) -> int: return 0 +def _cmd_generate_tests(args: argparse.Namespace) -> int: + try: + spec = read_openapi(args.spec) + if args.ai: + select_ai_backend(args.ai, model=args.model, effort=args.effort) + except (APIContractException, APIAIBackendException) as error: + apitestka_logger.error(f"cli generate-tests: {error}") + return 2 + text = json.dumps(generate_tests_from_openapi(spec), indent=2, ensure_ascii=False) + if args.output: + Path(args.output).write_text(text, encoding="utf-8") + else: + print(text) + return 0 + + +def _configure_generate_tests_parser(generate_parser: argparse.ArgumentParser) -> None: + generate_parser.add_argument("spec", help="OpenAPI document (JSON)") + generate_parser.add_argument("-o", "--output", help="Destination action JSON file (default: stdout)") + generate_parser.add_argument("--ai", choices=BACKEND_NAMES, + help="AI backend to use (default: APITESTKA_AI_BACKEND, else noop)") + generate_parser.add_argument("--model", help="Model for the anthropic backend") + generate_parser.add_argument("--effort", choices=("low", "medium", "high", "xhigh", "max"), + help="Effort for the anthropic backend") + generate_parser.set_defaults(func=_cmd_generate_tests) + + def _configure_openapi_parser(openapi_parser: argparse.ArgumentParser) -> None: openapi_parser.add_argument("--report", action="append", default=[], metavar="FILE", help="Saved JSON success report (_success.json); repeatable") @@ -183,6 +215,8 @@ def build_parser() -> argparse.ArgumentParser: _configure_openapi_parser(sub.add_parser("openapi", help="Infer an OpenAPI document from recorded traffic")) configure_contract_parser(sub.add_parser("contract", help="Record, verify and compare consumer contracts")) + _configure_generate_tests_parser( + sub.add_parser("generate-tests", help="Write test actions for an OpenAPI document")) return parser diff --git a/je_api_testka/cli/completion.py b/je_api_testka/cli/completion.py index 979ef94..ce4696c 100644 --- a/je_api_testka/cli/completion.py +++ b/je_api_testka/cli/completion.py @@ -7,7 +7,8 @@ from __future__ import annotations SUBCOMMANDS = ( - "run", "create", "mock", "import", "repl", "summary", "scaffold", "completion", "mcp", "openapi", "contract", + "run", "create", "mock", "import", "repl", "summary", "scaffold", "completion", "mcp", "openapi", + "contract", "generate-tests", ) SUPPORTED_SHELLS = ("bash", "zsh", "fish", "powershell") diff --git a/je_api_testka/utils/exception/exceptions.py b/je_api_testka/utils/exception/exceptions.py index f2d4bf5..16367c9 100644 --- a/je_api_testka/utils/exception/exceptions.py +++ b/je_api_testka/utils/exception/exceptions.py @@ -160,3 +160,7 @@ class MockServerException(APITesterException): class APIContractException(APITesterException): """A consumer contract is malformed, or it does not hold against a provider or its OpenAPI document.""" + + +class APIAIBackendException(APITesterException): + """An AI backend is misconfigured or the provider rejected the request.""" diff --git a/je_api_testka/utils/executor/action_executor.py b/je_api_testka/utils/executor/action_executor.py index 6f5b681..983894f 100644 --- a/je_api_testka/utils/executor/action_executor.py +++ b/je_api_testka/utils/executor/action_executor.py @@ -2,6 +2,7 @@ from typing import Dict, Callable, Any, List, Union from je_api_testka import test_api_method_httpx +from je_api_testka.ai.backend import select_ai_backend from je_api_testka.ai.failure_classifier import classify_failures from je_api_testka.ai.fake_data_generator import generate_fake_payload from je_api_testka.ai.test_generator import generate_tests_from_openapi @@ -162,6 +163,7 @@ def __init__(self): "AT_classify_failures": classify_failures, "AT_generate_fake_payload": generate_fake_payload, "AT_generate_tests_from_openapi": generate_tests_from_openapi, + "AT_select_ai_backend": select_ai_backend, # Extra protocol backends (sync entry points only; async variants # are intentionally not registered because they return coroutines # that the executor would not await). diff --git a/progress.md b/progress.md index 90561f9..6278486 100644 --- a/progress.md +++ b/progress.md @@ -8,7 +8,6 @@ Cross-repo and workspace items live in `D:\Codes\progress.md` (relevant here: X- - **#3** [DECIDE] The PyPI classifier still says `Development Status :: 2 - Pre-Alpha`. - **#6** GUI redesign: restructure and restyle. `gui/main_widget.py` is one 605-line widget (over the 500-line limit) with six flat tabs. It should become a sidebar-navigated window with a request/response split view, and history and environment panels docked (`gui/history_panel.py`, `gui/env_manager_model.py` exist but are not wired in). It needs a refreshed theme (`gui/main_window.py:21` hard-codes `dark_amber.xml`), pages for the newer features (#8–#13), and all four language wrappers (English, Traditional Chinese, Simplified Chinese, Japanese). `APITestkaWidget` stays the embeddable entry point (PyBreeze contract, `architecture.md` §6). -- **#10** Reference AI backend on the Anthropic API. `ai/backend.py:1` ships only `NoOpAIBackend` and `StaticAIBackend`. Missing: an optional `ai` extra, an `AnthropicAIBackend` (key from the environment, never stored), and a way to select it (CLI, executor). - **#11** Load-test bridge to LoadDensity. No way to turn APITestka actions into a LoadDensity run. It needs a converter to LoadDensity's action JSON (`LD_start_test` with `tasks`) and a runner that uses LoadDensity's `--execute_file` CLI. That is a new cross-project contract, so it also needs §6 in both repos and the root `AI_CONTEXT.md`. - **#12** Response-time trends and anomaly detection. `utils/generate_report/trend_store.py:13` stores only run totals (success, failure, average elapsed). Missing: per-endpoint latency history, a baseline (median/MAD or EWMA), anomaly flags, and a trend report. - **#13** Test-as-spec full loop. The pieces exist separately (`cli/import_specs.py` spec → actions, `spec/records_to_openapi.py` records → spec, `diff/contract_diff.py`, `spec/openapi_changelog.py`), but no single flow runs the tests, infers the spec, checks it against the committed spec, fails on drift, and generates actions for uncovered operations. Depends on #7. diff --git a/pyproject.toml b/pyproject.toml index 0993d27..ad169f3 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -52,6 +52,8 @@ apitestka = "je_api_testka.pytest_plugin.plugin" gui = ["PySide6==6.11.2", "qt-material"] websocket = ["websockets>=12.0"] grpc = ["grpcio>=1.60"] +# ai/anthropic_backend.py (AnthropicAIBackend); anthropic 1.x is the httpx2-based SDK. +ai = ["anthropic>=1.0,<2"] schema = ["jsonschema>=4.0", "jsonpath-ng>=1.6"] security = ["pyjwt>=2.8", "botocore>=1.34"] otel = ["opentelemetry-api>=1.25", "opentelemetry-sdk>=1.25"] diff --git a/test/test_ai/test_anthropic_backend.py b/test/test_ai/test_anthropic_backend.py new file mode 100644 index 0000000..05519e5 --- /dev/null +++ b/test/test_ai/test_anthropic_backend.py @@ -0,0 +1,147 @@ +"""Tests for the Anthropic reference backend (a fake client stands in for the API).""" +from __future__ import annotations + +import builtins +import json +from types import SimpleNamespace + +import pytest + +from je_api_testka.ai import anthropic_backend +from je_api_testka.ai.anthropic_backend import FALLBACK_BETA, SYSTEM_PROMPT, AnthropicAIBackend +from je_api_testka.ai.backend import NoOpAIBackend, set_ai_backend +from je_api_testka.ai.test_generator import generate_tests_from_openapi +from je_api_testka.utils.exception.exceptions import APIAIBackendException + + +class _FakeMessages: + def __init__(self, response=None, error=None): + self.response = response + self.error = error + self.calls = [] + + def create(self, **kwargs): + self.calls.append(kwargs) + if self.error is not None: + raise self.error + return self.response + + +def _response(text="ok", stop_reason="end_turn", category=None): + content = [SimpleNamespace(type="thinking", thinking=""), SimpleNamespace(type="text", text=text)] + return SimpleNamespace(content=content, stop_reason=stop_reason, model="claude-opus-5-5", + stop_details=SimpleNamespace(category=category), _request_id="req_1") + + +def _backend(response=None, error=None, **options): + messages = _FakeMessages(response, error) + return AnthropicAIBackend(client=SimpleNamespace(beta=SimpleNamespace(messages=messages)), **options), messages + + +@pytest.fixture(autouse=True) +def _reset_backend(): + yield + set_ai_backend(NoOpAIBackend()) + + +def test_default_request_uses_opus_medium_effort_and_fallbacks(): + options = AnthropicAIBackend().request_options("Do it.", {"b": 1, "a": "é"}) + assert options["model"] == "claude-opus-5-5" + assert options["max_tokens"] == 16000 + assert options["system"] == SYSTEM_PROMPT + assert options["output_config"] == {"effort": "medium"} + assert options["fallbacks"] == "default" + assert options["betas"] == [FALLBACK_BETA] + assert "thinking" not in options # thinking is always on for this model + assert options["messages"] == [{"role": "user", "content": 'Do it.\n\n\n{"a": "é", "b": 1}\n'}] + + +def test_options_can_turn_off_fallbacks_and_effort(): + options = AnthropicAIBackend(model="claude-sonnet-5-5", effort=None, fallbacks=None).request_options("p") + assert options["model"] == "claude-sonnet-5-5" + assert options["messages"][0]["content"] == "p" + assert not {"output_config", "fallbacks", "betas"} & set(options) + + +def test_complete_returns_text_blocks_only(): + pytest.importorskip("anthropic") + backend, messages = _backend(_response("hello")) + assert backend.complete("p", context={"x": 1}) == "hello" + assert messages.calls[0]["fallbacks"] == "default" + + +@pytest.mark.parametrize("stop_reason", ["refusal", "max_tokens"]) +def test_unusable_replies_become_empty(stop_reason): + pytest.importorskip("anthropic") + backend, _messages = _backend(_response("partial", stop_reason=stop_reason, category="cyber")) + assert backend.complete("p") == "" + + +def _status_error(anthropic, error_class, status): + httpx2 = pytest.importorskip("httpx2") + request = httpx2.Request("POST", "https://api.anthropic.com/v1/messages") + return error_class("boom", response=httpx2.Response(status, request=request), body=None) + + +def test_transient_errors_become_empty(): + anthropic = pytest.importorskip("anthropic") + httpx2 = pytest.importorskip("httpx2") + request = httpx2.Request("POST", "https://api.anthropic.com/v1/messages") + for error in (_status_error(anthropic, anthropic.RateLimitError, 429), + _status_error(anthropic, anthropic.InternalServerError, 529), + anthropic.APIConnectionError(request=request), + anthropic.APITimeoutError(request=request)): + backend, _messages = _backend(error=error) + assert backend.complete("p") == "" + + +def test_rejected_requests_raise(): + anthropic = pytest.importorskip("anthropic") + for error_class, status in ((anthropic.AuthenticationError, 401), (anthropic.BadRequestError, 400), + (anthropic.NotFoundError, 404)): + backend, _messages = _backend(error=_status_error(anthropic, error_class, status)) + with pytest.raises(APIAIBackendException, match=str(status)): + backend.complete("p") + + +def test_client_is_built_lazily_with_timeout(monkeypatch): + anthropic = pytest.importorskip("anthropic") + built = [] + monkeypatch.setattr(anthropic, "Anthropic", lambda **kwargs: built.append(kwargs) or "client") + backend = AnthropicAIBackend(timeout=30.0) + assert built == [] + assert backend._client() == "client" and backend._client() == "client" + assert built == [{"timeout": 30.0}] + + +def test_missing_sdk_raises(monkeypatch): + real_import = builtins.__import__ + + def _fake_import(name, *args, **kwargs): + if name == "anthropic": + raise ImportError("nope") + return real_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", _fake_import) + with pytest.raises(APIAIBackendException, match="je_api_testka\\[ai\\]"): + anthropic_backend._import_anthropic() + + +SPEC = {"servers": [{"url": "https://api.invalid"}], "paths": {"/items": {"get": {"responses": {"200": {}}}}}} + + +def test_test_generation_uses_a_valid_fenced_reply(): + pytest.importorskip("anthropic") + actions = [["AT_test_api_method", {"http_method": "get", "test_url": "https://api.invalid/items"}]] + backend, messages = _backend(_response("```json\n" + json.dumps(actions) + "\n```")) + set_ai_backend(backend) + assert generate_tests_from_openapi(SPEC) == actions + assert '"openapi"' in messages.calls[0]["messages"][0]["content"] + + +@pytest.mark.parametrize("reply", ["[]", '[{"AT_test_api_method": {}}]', '[["print", {}]]', '[["AT_x", 1]]', "nope"]) +def test_test_generation_falls_back_on_unusable_replies(reply): + pytest.importorskip("anthropic") + backend, _messages = _backend(_response(reply)) + set_ai_backend(backend) + assert generate_tests_from_openapi(SPEC)[0][1]["result_check_dict"] == {"status_code": 200} diff --git a/test/test_ai/test_backend_selection.py b/test/test_ai/test_backend_selection.py new file mode 100644 index 0000000..e052ccc --- /dev/null +++ b/test/test_ai/test_backend_selection.py @@ -0,0 +1,123 @@ +"""Tests for choosing the AI backend by name, from the environment, the executor and the CLI.""" +from __future__ import annotations + +import json + +import pytest + +from je_api_testka.ai import backend as backend_module +from je_api_testka.ai.anthropic_backend import AnthropicAIBackend +from je_api_testka.ai.backend import ( + NoOpAIBackend, + StaticAIBackend, + ai_backend, + build_ai_backend, + select_ai_backend, + set_ai_backend, +) +from je_api_testka.ai.failure_classifier import classify_failures +from je_api_testka.ai.fake_data_generator import generate_fake_payload +from je_api_testka.ai.reply import parse_json_reply +from je_api_testka.cli.cli_main import main +from je_api_testka.utils.exception.exceptions import APIAIBackendException +from je_api_testka.utils.executor.action_executor import execute_action + + +@pytest.fixture(autouse=True) +def _reset_backend(): + yield + set_ai_backend(NoOpAIBackend()) + + +def test_build_by_name(): + assert isinstance(build_ai_backend("noop"), NoOpAIBackend) + backend = build_ai_backend("anthropic", model="claude-sonnet-5-5", effort="low") + assert (backend.model, backend.effort) == ("claude-sonnet-5-5", "low") + assert build_ai_backend("anthropic").model == "claude-opus-5-5" + + +def test_build_rejects_unknown_name_and_noop_options(): + with pytest.raises(APIAIBackendException): + build_ai_backend("gpt") + with pytest.raises(APIAIBackendException): + build_ai_backend("noop", model="x") + + +def test_environment_selects_backend_on_first_use(monkeypatch): + monkeypatch.setattr(backend_module, "_selection", {}) + monkeypatch.setenv("APITESTKA_AI_BACKEND", "Anthropic") + monkeypatch.setenv("APITESTKA_AI_MODEL", "claude-haiku-4-5") + monkeypatch.setenv("APITESTKA_AI_EFFORT", "") + selected = ai_backend() + assert isinstance(selected, AnthropicAIBackend) + assert (selected.model, selected.effort) == ("claude-haiku-4-5", "medium") + assert ai_backend() is selected + + +def test_environment_defaults_to_noop(monkeypatch): + monkeypatch.setattr(backend_module, "_selection", {}) + monkeypatch.delenv("APITESTKA_AI_BACKEND", raising=False) + assert isinstance(ai_backend(), NoOpAIBackend) + + +def test_select_through_the_executor(): + record = execute_action([["AT_select_ai_backend", {"name": "anthropic", "effort": "high"}]]) + assert list(record.values()) == ["AnthropicAIBackend"] + assert ai_backend().effort == "high" + assert select_ai_backend() == "NoOpAIBackend" + + +def test_failure_classifier_asks_the_backend_about_unmatched_errors(): + set_ai_backend(StaticAIBackend(response='```json\n["network", "server"]\n```')) + counts = classify_failures([[{}, "401 Unauthorized"], [{}, "socket reset by peer"], [{}, "upstream said no"]]) + assert counts == {"auth": 1, "network": 1, "server": 1} + + +@pytest.mark.parametrize("reply", ["", "nope", '["network"]', '["network", "cosmic"]']) +def test_failure_classifier_keeps_other_on_unusable_labels(reply): + set_ai_backend(StaticAIBackend(response=reply)) + assert classify_failures([[{}, "a"], [{}, "b"]]) == {"other": 2} + + +def test_fake_payload_accepts_fenced_json(): + set_ai_backend(StaticAIBackend(response='```\n{"id": 7}\n```')) + assert generate_fake_payload({"type": "object"}) == {"id": 7} + + +def test_parse_json_reply(): + assert parse_json_reply(' [1, 2] ') == [1, 2] + assert parse_json_reply('```json\n{"a": 1}\n```') == {"a": 1} + with pytest.raises(json.JSONDecodeError): + parse_json_reply("```\nnot json\n```") + + +def _spec(tmp_path): + path = tmp_path / "openapi.json" + path.write_text(json.dumps({"servers": [{"url": "https://api.invalid"}], + "paths": {"/items": {"get": {"responses": {"200": {}}}}}}), encoding="utf-8") + return str(path) + + +def test_cli_generate_tests_deterministic(tmp_path): + out = tmp_path / "actions.json" + assert main(["generate-tests", _spec(tmp_path), "-o", str(out), "--ai", "noop"]) == 0 + assert json.loads(out.read_text(encoding="utf-8"))[0][1]["test_url"] == "https://api.invalid/items" + + +def test_cli_generate_tests_with_anthropic(tmp_path, monkeypatch, capsys): + actions = [["AT_test_api_method", {"http_method": "get", "test_url": "https://api.invalid/items?page=1"}]] + seen = {} + + def _fake_complete(self, prompt, *, context=None): + seen.update(model=self.model, effort=self.effort) + return json.dumps(actions) + + monkeypatch.setattr(AnthropicAIBackend, "complete", _fake_complete) + assert main(["generate-tests", _spec(tmp_path), "--ai", "anthropic", "--effort", "high"]) == 0 + assert json.loads(capsys.readouterr().out) == actions + assert seen == {"model": "claude-opus-5-5", "effort": "high"} + + +def test_cli_generate_tests_usage_errors(tmp_path): + assert main(["generate-tests", str(tmp_path / "missing.json")]) == 2 + assert main(["generate-tests", _spec(tmp_path), "--ai", "noop", "--model", "x"]) == 2 diff --git a/test/test_executor_integration.py b/test/test_executor_integration.py index 0eaeb9c..1dcf6c5 100644 --- a/test/test_executor_integration.py +++ b/test/test_executor_integration.py @@ -45,7 +45,7 @@ "AT_write_contract", "AT_verify_contract", "AT_check_contract_against_openapi", # AI "AT_classify_failures", "AT_generate_fake_payload", - "AT_generate_tests_from_openapi", + "AT_generate_tests_from_openapi", "AT_select_ai_backend", # Extra protocols "AT_test_api_method_websocket", "AT_test_api_method_sse", diff --git a/uv.lock b/uv.lock index 7a76e19..09c0f7a 100644 --- a/uv.lock +++ b/uv.lock @@ -20,6 +20,24 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/99/91/8acff4f5e50511b911bbccb72b8628a49c68ce14148cd9f6431094859a90/annotated_types-0.8.0-py3-none-any.whl", hash = "sha256:f072f4d804ea359e4eaf198b1af7a8b0943881a87f31bb764f8bf219bb9419e0", size = 13427, upload-time = "2026-07-23T20:16:12.938Z" }, ] +[[package]] +name = "anthropic" +version = "1.9.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "docstring-parser" }, + { name = "httpx2" }, + { name = "jiter" }, + { name = "pydantic" }, + { name = "sniffio" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/18/0a/1ab3bf53a672fd9dd42d3208e335940049431b79781b2a9b324203d30c38/anthropic-1.9.0.tar.gz", hash = "sha256:da5191a0af0b853a797fc0c6b09fc5b05cb5d9da063ba80fdd03c736abe81b6c", size = 1243092, upload-time = "2026-09-28T18:04:03.005Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/15/2a/67c1b2ddf91be3112856ed74b96d1576f00dd423e4f20bb251aa144fbd6b/anthropic-1.9.0-py3-none-any.whl", hash = "sha256:a7fe05a9e4b72113b934501698df5d56484ac89f05677f0f793f8bfddb15cfff", size = 1360651, upload-time = "2026-09-28T18:04:01.452Z" }, +] + [[package]] name = "anyio" version = "4.15.1" @@ -423,6 +441,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/07/6c/aa3f2f849e01cb6a001cd8554a88d4c77c5c1a31c95bdf1cf9301e6d9ef4/defusedxml-0.7.1-py2.py3-none-any.whl", hash = "sha256:a352e7e428770286cc899e2542b6cdaedb2b4953ff269a210103ec58f6198a61", size = 25604, upload-time = "2021-03-08T10:59:24.45Z" }, ] +[[package]] +name = "docstring-parser" +version = "0.18.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e0/4d/f332313098c1de1b2d2ff91cf2674415cc7cddab2ca1b01ae29774bd5fdf/docstring_parser-0.18.0.tar.gz", hash = "sha256:292510982205c12b1248696f44959db3cdd1740237a968ea1e2e7a900eeb2015", size = 29341, upload-time = "2026-04-14T04:09:19.867Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a7/5f/ed01f9a3cdffbd5a008556fc7b2a08ddb1cc6ace7effa7340604b1d16699/docstring_parser-0.18.0-py3-none-any.whl", hash = "sha256:b3fcbed555c47d8479be0796ef7e19c2670d428d72e96da63f3a40122860374b", size = 22484, upload-time = "2026-04-14T04:09:18.638Z" }, +] + [[package]] name = "exceptiongroup" version = "1.3.1" @@ -629,6 +656,9 @@ dependencies = [ ] [package.optional-dependencies] +ai = [ + { name = "anthropic" }, +] grpc = [ { name = "grpcio" }, ] @@ -658,6 +688,7 @@ websocket = [ [package.metadata] requires-dist = [ + { name = "anthropic", marker = "extra == 'ai'", specifier = ">=1.0,<2" }, { name = "botocore", marker = "extra == 'security'", specifier = ">=1.34" }, { name = "defusedxml" }, { name = "flask" }, @@ -674,7 +705,7 @@ requires-dist = [ { name = "requests" }, { name = "websockets", marker = "extra == 'websocket'", specifier = ">=12.0" }, ] -provides-extras = ["gui", "websocket", "grpc", "schema", "security", "otel", "mcp"] +provides-extras = ["gui", "websocket", "grpc", "ai", "schema", "security", "otel", "mcp"] [[package]] name = "jinja2" @@ -688,6 +719,132 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" }, ] +[[package]] +name = "jiter" +version = "0.17.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9c/1f/8176d92e001f86505424b41664032ae26a882bc9ca41a32c803f373f9195/jiter-0.17.0.tar.gz", hash = "sha256:03e432f226a453851079fb84cd17c6da9991eab723e28d716f14ae3d906e0c12", size = 229037, upload-time = "2026-09-12T15:14:14.253Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e8/fb/0b68a8666a203ab349334e502285153ac97e84a1a3e2f80de8385ed0899c/jiter-0.17.0-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:ed1a24005daac667d577402d75a2922f9775a165b146b883ff1ad3602d8be689", size = 290968, upload-time = "2026-09-12T15:11:10.44Z" }, + { url = "https://files.pythonhosted.org/packages/ca/e9/5993079500530586655c6f2823e2926b3f5ceddd49918c5a86757eadba3a/jiter-0.17.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:b847b18d066c46b3b7ae49d6c94a7634c5e4a8983146ee25562a092000f5e3ad", size = 324072, upload-time = "2026-09-12T15:11:12.03Z" }, + { url = "https://files.pythonhosted.org/packages/2a/56/6f5dbffaa5b0e647386e3c3376aa150987538f1b13f6da1a9582787957af/jiter-0.17.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7b68d3495d95da120651a5628c7ebadee84ed001a1b76e6afc325c42482f15b5", size = 344891, upload-time = "2026-09-12T15:11:13.387Z" }, + { url = "https://files.pythonhosted.org/packages/9a/9d/8dd5719add6c77be95b23c129682a1484c5339ffd66736d66bfa6831a2f3/jiter-0.17.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:3c1a5336c04a41b1f1cf9572e294aec27cc569767ff73de7bf87a91f0bea7cb9", size = 328213, upload-time = "2026-09-12T15:11:14.655Z" }, + { url = "https://files.pythonhosted.org/packages/6e/e5/82f885863afaa79efac066efdb7b70bd242ef83d3e548453d61b3737e54a/jiter-0.17.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b75f85660108965a94be77911a25a253429307294d9415b3c597118977a614de", size = 343033, upload-time = "2026-09-12T15:11:16.046Z" }, + { url = "https://files.pythonhosted.org/packages/a0/41/ee3cd7db1704ec9c274ae549803ef1369c801490f7f6d48d0aa3b9ac1ae0/jiter-0.17.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:32aaaa764604496610a3ad2d98503ae88ccb2fbe769e892ff4533e778e85f708", size = 351701, upload-time = "2026-09-12T15:11:19.188Z" }, + { url = "https://files.pythonhosted.org/packages/93/2c/af9ca503b38683a82b6732352a861515f28e2d46fe445aea57e07fc86e06/jiter-0.17.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:826871c42cebaae22f0a2b5673a4a1a75c851bb2d13b3c17764a630a6b298984", size = 348754, upload-time = "2026-09-12T15:11:20.486Z" }, + { url = "https://files.pythonhosted.org/packages/d0/ba/692b071b194270b019b7d1790adb8658193379b951029e5ad15a14a2809d/jiter-0.17.0-cp310-cp310-manylinux_2_31_riscv64.whl", hash = "sha256:00b5a98df3e3a3e8cf7b619f4ac2f8bf975bbf3d95d02c5d17b8dbfe5c8b8245", size = 329263, upload-time = "2026-09-12T15:11:21.701Z" }, + { url = "https://files.pythonhosted.org/packages/f1/8f/35a22beb6b9eb025b7e2f88a38ca3b9c87017aac237feec2583eabe243ad/jiter-0.17.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:6af5b74073bd25bae695e6d00919f6a9be7ed5a9f8836d981eb1ffe84139e6fb", size = 337572, upload-time = "2026-09-12T15:11:23.145Z" }, + { url = "https://files.pythonhosted.org/packages/7a/0f/4341ceb3ce08199b4e8a798acc0bb0c79b3e861b6cecdb9d3b5138965c5d/jiter-0.17.0-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:16dd0c1baf098ae70b8f3616574eb3fedf34e26670b89e16a7e67561f737ed2d", size = 485174, upload-time = "2026-09-12T15:11:24.394Z" }, + { url = "https://files.pythonhosted.org/packages/a2/6d/92fa138ac82db9001b68563df204c32f2203bee84301df083260039ca23b/jiter-0.17.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:545c36a0f3b2238c242cc9785439d3242a871b7bc39fe3f441bcaa07bf3aa83e", size = 522115, upload-time = "2026-09-12T15:11:25.645Z" }, + { url = "https://files.pythonhosted.org/packages/52/92/9b11557ddf898e84c253d1c42cc1bddd8ad1a754b46b01ae35a6a253526f/jiter-0.17.0-cp310-cp310-win32.whl", hash = "sha256:155be7355bdb7ca76ab0961be8982c225f964a5c073a83984183f22391cc29fc", size = 188163, upload-time = "2026-09-12T15:11:26.793Z" }, + { url = "https://files.pythonhosted.org/packages/a5/f3/b4381cc962fed7d505913d5920e9f51f1fd5cf340a563134e8cadf3604d2/jiter-0.17.0-cp310-cp310-win_amd64.whl", hash = "sha256:37150a9e02e869475854fa20b7d0d5e26d18d0f8bc17293999973ff27e99ae7a", size = 231679, upload-time = "2026-09-12T15:11:28.911Z" }, + { url = "https://files.pythonhosted.org/packages/a1/50/17afdaffcc8af4bf4fddf2b6c26d066553aa2221983f2affcde435fc2532/jiter-0.17.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:cfafd7be8b16ceadd298db542cead37cddc211c4c49e04ad2596924df18625b1", size = 290699, upload-time = "2026-09-12T15:11:30.085Z" }, + { url = "https://files.pythonhosted.org/packages/c9/e4/c185d32d5b3657ad84da26c84a9eb15f00aa1b39d6882fcc0052dba2d7c2/jiter-0.17.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:8adca2e793288e5f1bb29279bb439d0d3cfbb50eddca7e7e6ffd42ff4f482406", size = 324246, upload-time = "2026-09-12T15:11:31.3Z" }, + { url = "https://files.pythonhosted.org/packages/24/7a/8b8903bfe91a90a8fa1ec9b45d9fda5b6287a386693b69d720a882d73f3c/jiter-0.17.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:30c692d567ba206c7cca38c9d1d0ccc70c9786290173c184d871ca12e9981ed7", size = 343275, upload-time = "2026-09-12T15:11:32.758Z" }, + { url = "https://files.pythonhosted.org/packages/a2/5d/6821fae2abc71a3c3a84bef8598d31fc4f27d9edfb55bd8f6c08afb8ef93/jiter-0.17.0-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:81c83c0abe614446a283d994d2c07c4f58632dea2cdf66ba9e2921bb8ccd593e", size = 328034, upload-time = "2026-09-12T15:11:33.9Z" }, + { url = "https://files.pythonhosted.org/packages/f3/51/8e7a963b1c2dfdc01d6228b004f50a2a3d7c46f0549d7b096a5d15ef81d5/jiter-0.17.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:073dc68c1a700c8fc480e877864a6b6ffc887533e261f4380c08c16bf09d057a", size = 342457, upload-time = "2026-09-12T15:11:36.414Z" }, + { url = "https://files.pythonhosted.org/packages/73/27/8b2a267e3bda45d9298331cacfe3521e761f0a2b05ad10a23c6548d08358/jiter-0.17.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:492f37230bbf9581ab2c17bcda862c249afb9ae2e3ab2dd6db59943bc4cc3153", size = 351271, upload-time = "2026-09-12T15:11:37.692Z" }, + { url = "https://files.pythonhosted.org/packages/4a/8f/5c74e5e142a6736833d7a991ab04d5c0738038dc44db05af0bb3cd2559e8/jiter-0.17.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5888fe5abc1ca2fa834a3e1b4c7ef0dcece286a7d7e95a609ef0934b777b9fc9", size = 346052, upload-time = "2026-09-12T15:11:39.722Z" }, + { url = "https://files.pythonhosted.org/packages/98/9c/f54920f06d1696e80b1be841d56412871c6856b1b1e3b541b1ed35346554/jiter-0.17.0-cp311-cp311-manylinux_2_31_riscv64.whl", hash = "sha256:84ac78df457e1ee3f7e733bd114823302ae8c5ad5542d7e6647d92ffaa090a04", size = 328915, upload-time = "2026-09-12T15:11:41.065Z" }, + { url = "https://files.pythonhosted.org/packages/21/53/080f126863bceb055db9f1fd5431485eb493b35545a3c35e961ac18cd924/jiter-0.17.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:7573e80232c5bcf80c24c038cf7e53a463f5c3b1dd1dd4109d66304f4dccc233", size = 337240, upload-time = "2026-09-12T15:11:42.356Z" }, + { url = "https://files.pythonhosted.org/packages/f0/76/3ab742823a0e0e70e143c6c90a482d9d90396ac2445e7b9483eb7245d3b5/jiter-0.17.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:11902505d401691720f5785c15b02204248526edee11b635cd6c40cd52b81599", size = 484917, upload-time = "2026-09-12T15:11:43.549Z" }, + { url = "https://files.pythonhosted.org/packages/14/e0/8ca71bc8b9cc9ed96c9da565863e00f3bb875a8fb82abcf03e975e067902/jiter-0.17.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:64846211a2debe7c071d2146d2283d2b0c1c93dc8fd5fb7794faac2ca6061b5c", size = 521868, upload-time = "2026-09-12T15:11:44.704Z" }, + { url = "https://files.pythonhosted.org/packages/c1/02/81f8719dcedb75713082a2048405376c81f6546b75af8167f3bba01a1ed0/jiter-0.17.0-cp311-cp311-win32.whl", hash = "sha256:c19b9357309b8cc6de8a48fca8e44a8c9c2feaaa2f5896d037fa505d48fcab80", size = 187844, upload-time = "2026-09-12T15:11:45.874Z" }, + { url = "https://files.pythonhosted.org/packages/3e/8c/59693f348488f01ed12d862e99ab8da14961152d3e9c39b9b1ef363f3572/jiter-0.17.0-cp311-cp311-win_amd64.whl", hash = "sha256:e654b6b04e39c9cb19cb8b04c6ddf1f2db07751fa14156413969fd78bad0e5cb", size = 231402, upload-time = "2026-09-12T15:11:47.083Z" }, + { url = "https://files.pythonhosted.org/packages/fe/89/fb35e286463cb9f01edc2c4e47df6e5477ee36bca5096414e9ea87985588/jiter-0.17.0-cp311-cp311-win_arm64.whl", hash = "sha256:3ad556afc289f15d2b181b941982d01f06190863c07440185b9f354e1bd2def3", size = 187006, upload-time = "2026-09-12T15:11:48.245Z" }, + { url = "https://files.pythonhosted.org/packages/aa/f8/07bd8c3a23f7a8a6875e6a820bbffe1483a18f18f9398a91b5495123176e/jiter-0.17.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:ebf918dfd6a74adc1b9ad71f63c4ab00902fcd3b7fd39f2e24d871db8d713b91", size = 291633, upload-time = "2026-09-12T15:11:49.431Z" }, + { url = "https://files.pythonhosted.org/packages/0e/5e/0de4c6f84ffefa6809ffc2d550b9a314365acf7e7ec9b6c7375d49047900/jiter-0.17.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:61aed66ee042b3b49ef85fdf75714234d055d89d8496ac1c6e47f89e7a30d5e4", size = 321695, upload-time = "2026-09-12T15:11:52.727Z" }, + { url = "https://files.pythonhosted.org/packages/20/ac/befe2e82065bee37a0252081666ed2f48c1ac5f5c6c318c2de8168ba393d/jiter-0.17.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:76eb4a5c20e86f9f848286f167024890f2862258a965d254774deb7fc1545ca1", size = 341967, upload-time = "2026-09-12T15:11:54.231Z" }, + { url = "https://files.pythonhosted.org/packages/9f/cd/9797c1e529746750ae589da7c1a8c24373f00d88e11a989f9e5eb1959079/jiter-0.17.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:bcc064f99183a9cbe7f26ed648c352031a74145cd61ed75d34632c73eb46a5a8", size = 326546, upload-time = "2026-09-12T15:11:55.41Z" }, + { url = "https://files.pythonhosted.org/packages/d9/fd/e6914c38d6347bab4ebff2b1f0c0f191db276e7a1d5c376176757da42fe3/jiter-0.17.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:73b64e69c4150748e020356d958af94bec33c70a0a93d665cfa8f6d580fe1a63", size = 340995, upload-time = "2026-09-12T15:11:58.211Z" }, + { url = "https://files.pythonhosted.org/packages/9d/7d/611b3abf6f88945b5474da5cdc6d1a185e805ac9bf446bb7766dcda6ea87/jiter-0.17.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:f0bc7f684b65bcda9c20434267577db71bf9905ceddd32b60d1d93278d8c8d3a", size = 352188, upload-time = "2026-09-12T15:11:59.414Z" }, + { url = "https://files.pythonhosted.org/packages/52/f8/b6e513ecbdf3b3cebe587c2279281ecf775b729a58cf4cc7bdf898ded029/jiter-0.17.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8c21265b251d99bbb40080d178a8953e35601d3a1564e05c4de4c0d2ca616797", size = 345025, upload-time = "2026-09-12T15:12:00.697Z" }, + { url = "https://files.pythonhosted.org/packages/28/a8/fe26d06c5a6c5a4cfe703c5154c8a140da1305671eb3681aba9422d4f393/jiter-0.17.0-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:f3d7f7b34114f7ddc6d72a8e882d49de636b35d9fd12b4d420d3c5729f6c9812", size = 329180, upload-time = "2026-09-12T15:12:01.831Z" }, + { url = "https://files.pythonhosted.org/packages/e1/58/e6d66a26af40a20e62486feb7e222fd50f6e7aaa4f107abd89675dcc835b/jiter-0.17.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:5078ab00664307fab2019b522a93aeb191122789f085daf5fd9e362154021d4a", size = 335805, upload-time = "2026-09-12T15:12:03.056Z" }, + { url = "https://files.pythonhosted.org/packages/ef/3e/96520aa2fef5ef831d95483a902140bfab83dcac9eaa74f7df61b5e50a1b/jiter-0.17.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:470e1b1e4c42f1ead2189166a299691871a2df5056c976e7fb96feafaf5f9d44", size = 484121, upload-time = "2026-09-12T15:12:04.414Z" }, + { url = "https://files.pythonhosted.org/packages/6a/8f/5d9d92fe538bf36ff481a2278c48147e59c1cf8eb2f7be665260665febe5/jiter-0.17.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:6eb6aedeb7352b8f3b6af9cbd67983840165c00428e63f1b420a85885128ea31", size = 521310, upload-time = "2026-09-12T15:12:05.612Z" }, + { url = "https://files.pythonhosted.org/packages/50/06/a09f979b22e652afbc3de66c709b2ba92edcef555f7535ab937c86b4f21a/jiter-0.17.0-cp312-cp312-win32.whl", hash = "sha256:362bb47423886d45a9f705d2d9d4008c6eedd4e41eb1bab4e96fb6daa06b33fd", size = 185029, upload-time = "2026-09-12T15:12:06.994Z" }, + { url = "https://files.pythonhosted.org/packages/6c/d9/98265a005b2473ec2be5a84e2b64c2f65382c673879f1574845cd4bcd77c/jiter-0.17.0-cp312-cp312-win_amd64.whl", hash = "sha256:9bd3caac219df476dd0cc3fe01d2f1581ed588906feac767abd9614c1c12f8b3", size = 227381, upload-time = "2026-09-12T15:12:08.823Z" }, + { url = "https://files.pythonhosted.org/packages/a8/11/2e05bf5a56e57a543ebb8f585074adf09383e99d7b062dac92eab1f4d57f/jiter-0.17.0-cp312-cp312-win_arm64.whl", hash = "sha256:36ee6e69027396664e59995b9a635a947a5304ee9837279584a0bb8145c8f6b8", size = 183610, upload-time = "2026-09-12T15:12:10.374Z" }, + { url = "https://files.pythonhosted.org/packages/40/eb/2c4a8075ed5ea02b56911e9375d4c8d7784572ff4af32e5a99ae0d071044/jiter-0.17.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:1b18434638228c0c184281609bf3d9459026a0f1ea48fb76c205e3ef72069caa", size = 290991, upload-time = "2026-09-12T15:12:11.641Z" }, + { url = "https://files.pythonhosted.org/packages/ca/b1/34bfa29599d420423baac6ff7cada6674fe63d5a7a2ccb3900b904678783/jiter-0.17.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:ec89771f4272b989487a6364e519db6bbaba323e8bbf949ac89a45ea9c18b7a3", size = 321425, upload-time = "2026-09-12T15:12:13.855Z" }, + { url = "https://files.pythonhosted.org/packages/11/71/a5ac64a62a04aebd556afadab14a6b730001e16df87266ded943a100a1d9/jiter-0.17.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4e3f052c671d5f425cca5ea5901cf11a831369fba4a55a3862cab93c323b4c3b", size = 343138, upload-time = "2026-09-12T15:12:15.046Z" }, + { url = "https://files.pythonhosted.org/packages/01/dd/f761e320ea473314cb68612bc6a435393464dbd198051399b36848b4ebf3/jiter-0.17.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:785a216bbaf8f15fc974e964ced7322cd3d774bb0e86949edd78c6bffd6ba35b", size = 325805, upload-time = "2026-09-12T15:12:16.506Z" }, + { url = "https://files.pythonhosted.org/packages/19/1a/27d8e40f0fb29bbc7a5adf30907144396a115dbe93d5d8976c054a6dfe96/jiter-0.17.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:d85c558c9f8532bba287a990ac63767c7daf756f0d8c030219f62499b1fa228a", size = 340230, upload-time = "2026-09-12T15:12:17.682Z" }, + { url = "https://files.pythonhosted.org/packages/ac/c0/30bcde78a28155461f965d16b7aca4ffca6d17494d905f7a0bb072e6c64e/jiter-0.17.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:5c23849235d2142ce444b2b8c6eceee9f82f4cc0bd5c9081602e4155c6197807", size = 351343, upload-time = "2026-09-12T15:12:19.337Z" }, + { url = "https://files.pythonhosted.org/packages/27/17/91420b156315ae22732f5ee1a7b5725a030aab9dc8fd7dcdacfb4aa588d3/jiter-0.17.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:58df29268a95e910f17db7ec9178eb7f15aa8619aaca3575275c4e6b3f4fe4c5", size = 344990, upload-time = "2026-09-12T15:12:20.705Z" }, + { url = "https://files.pythonhosted.org/packages/6d/a2/ae6d5672644cc11127970277c9aeb0fa6fae376845587f5b0a8e8828167c/jiter-0.17.0-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:a277f97eba7d66b1ee27eb5dab5b774ff46a10c78d89a1d3dcce04ce1357c8ca", size = 328624, upload-time = "2026-09-12T15:12:23.859Z" }, + { url = "https://files.pythonhosted.org/packages/04/62/45cb1162f6aa586536e4a973fc339d72dc6b08cca030d70a838a307aa778/jiter-0.17.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:fe15ddf316f1f1f643347d3a474e74ce61880c79a11ec5dca53df20c071bd3e8", size = 334731, upload-time = "2026-09-12T15:12:25.229Z" }, + { url = "https://files.pythonhosted.org/packages/d9/5f/45c1574b644da7deda0b7591c349520dcf83ce45b24d7ca19922dab1fc27/jiter-0.17.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:02adebb7ce6413c44d40af9ad59d1c1cd79630ccdcb6f7bdd2d461e48c03d8f9", size = 483649, upload-time = "2026-09-12T15:12:27.557Z" }, + { url = "https://files.pythonhosted.org/packages/c1/d3/ebea1ecb5b241c519f192b30215c79a8e47f42f1621acbcd6f8830728416/jiter-0.17.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:55d0e0e613a3f9ad600cf436e0e2b8057d1b52bcf1d91b2d36ac53451231e6a8", size = 520758, upload-time = "2026-09-12T15:12:28.99Z" }, + { url = "https://files.pythonhosted.org/packages/64/e6/682b641ff0765ea9bdc349dbc7d223de5c8af8ec1abda0db3406992f92fe/jiter-0.17.0-cp313-cp313-win32.whl", hash = "sha256:2c45ad7c973ef33fe5114a953377b35a95240f4542c0724d9f781e47dc24bac7", size = 184334, upload-time = "2026-09-12T15:12:30.813Z" }, + { url = "https://files.pythonhosted.org/packages/b8/d2/9a49aac2b27af4cc5015e368c0cc3588491a532f717a668ffce1f1ac57da/jiter-0.17.0-cp313-cp313-win_amd64.whl", hash = "sha256:a3cebb1fe4a1abb00465f3f8a17e09112603e8b7c59e5c3adbcd9f7815a64acd", size = 226601, upload-time = "2026-09-12T15:12:32.096Z" }, + { url = "https://files.pythonhosted.org/packages/b4/ce/9a43e9f614608eafa78de22aedcff54cd21324467b5d442d5c9b00244145/jiter-0.17.0-cp313-cp313-win_arm64.whl", hash = "sha256:96b8b0c6dc5d78682f54a450785e075aa929cde768304cad363cd4efba5a82ac", size = 183103, upload-time = "2026-09-12T15:12:34.396Z" }, + { url = "https://files.pythonhosted.org/packages/01/9e/23065f8e2c7a4c372c1b6f6622e4cfab4dc786cb5150052b1527e6a6a840/jiter-0.17.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:00d783a779c5664e16dbad5e3a3c3a75e128b07dd5f4765159658d9210a50ca5", size = 292210, upload-time = "2026-09-12T15:12:35.613Z" }, + { url = "https://files.pythonhosted.org/packages/ea/81/67b58647560bc82a4490d722caa8561d7a86a9f45d4fa620b7e5fe282c7a/jiter-0.17.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:0619d806e260ecf0c2a64521942c94af5d547c9ec99b55ae4f51b538b5576a76", size = 321512, upload-time = "2026-09-12T15:12:36.907Z" }, + { url = "https://files.pythonhosted.org/packages/c7/07/6658359a25f55927f7f8bf0e16465dee2ccd0b2a1a5208acc0df8972e074/jiter-0.17.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:dc0288ce39190ee33fe6e4ec73161eed34e7e2da509b525546ca061778d62b64", size = 343897, upload-time = "2026-09-12T15:12:38.189Z" }, + { url = "https://files.pythonhosted.org/packages/46/04/5d50a9f0319cbdc37fd53c27f8c313d46afc34f1b048219ae6d8ea068da4/jiter-0.17.0-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:5a52a430d04225ffde633e6840bf2381d34c019ff98526b5929755b9052fb199", size = 326519, upload-time = "2026-09-12T15:12:39.532Z" }, + { url = "https://files.pythonhosted.org/packages/bb/c7/d02517832b29eb8275fdd0f4ce0f17b80f58cc4c3ebecd4d9ace990d633d/jiter-0.17.0-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:37f33d327900bf2879613b3363fd48df97b4232d0c41f54bcf2e790c2fc40a71", size = 341369, upload-time = "2026-09-12T15:12:41.486Z" }, + { url = "https://files.pythonhosted.org/packages/3b/07/499b5f5603501cdd93a73a6a176dfad9c96555a3ae58ca9f8e3acba63dc9/jiter-0.17.0-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:6cf564d43c4388149ca58ee571d0f5ccf875e20d1fd4662fd94cc0d1ea3b10ef", size = 352160, upload-time = "2026-09-12T15:12:42.721Z" }, + { url = "https://files.pythonhosted.org/packages/f5/75/b04013c7743269d4533ef4e746fc0ed678a143968dd7448658e3f51daad2/jiter-0.17.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:523c499235fb65add25d4bb01b1c4709ce695efdc7deb6c0a7bc515b5c44e0fb", size = 345018, upload-time = "2026-09-12T15:12:44.192Z" }, + { url = "https://files.pythonhosted.org/packages/1d/96/cbb6fd1e42a77c8412ec4643db95059b30cdfc635e387cc9193e098ce268/jiter-0.17.0-cp314-cp314-manylinux_2_31_riscv64.whl", hash = "sha256:455e4ab35cb2a4a91a8404e08fd3c621bae433922e59bf1c494fe20a426b013b", size = 329244, upload-time = "2026-09-12T15:12:45.491Z" }, + { url = "https://files.pythonhosted.org/packages/15/67/d3be402f398566a379bf40ae65be5c3505b14d9e95e0802a597ddde7ddee/jiter-0.17.0-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:6871973bfbd4408f7f1c632b30bbb5bbd9671c1bc8650af6823e24b7be13709b", size = 335693, upload-time = "2026-09-12T15:12:46.935Z" }, + { url = "https://files.pythonhosted.org/packages/7f/8d/98e2c4130b93d64f1d67c89060b928d04102549bf05e64451c9e6024f9ca/jiter-0.17.0-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:77f6aac0137309b31448c1bdcda4c6c77077664a6d018ece8d94019c68a5a5b9", size = 484329, upload-time = "2026-09-12T15:12:48.361Z" }, + { url = "https://files.pythonhosted.org/packages/78/5e/8da91e49f0fbca37c3489fb4cf3ad6676d4965f00ae5468bca3a2513737a/jiter-0.17.0-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:93946d89fa04d5ba64dd323a8dd8d901676cb8a3c81d99ae4f6c051a9b4c3f2f", size = 521358, upload-time = "2026-09-12T15:12:49.856Z" }, + { url = "https://files.pythonhosted.org/packages/be/21/5388684a5a38af3557cd9c2424b9827c71809cff24373c75ef9d0d3dfba9/jiter-0.17.0-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:70f19a2ca8429f91e82eeffb2f51cb87bc2d6e953b009b91a92d29c3a16ccb03", size = 110459, upload-time = "2026-09-12T15:12:51.747Z" }, + { url = "https://files.pythonhosted.org/packages/b1/ad/58b3a93525d2ffca7f54d9dee441381990082bd1172fbeb8d6a3f72a4dc3/jiter-0.17.0-cp314-cp314-win32.whl", hash = "sha256:71dbd74314c5df52a1bccf7b8bca46d14e943af7a2012e73b23f49977ef194c8", size = 185043, upload-time = "2026-09-12T15:12:54.477Z" }, + { url = "https://files.pythonhosted.org/packages/7a/4a/1aa520eb6c359b262c14ff995ca7283837208ddfb1202082ce9d73cf214d/jiter-0.17.0-cp314-cp314-win_amd64.whl", hash = "sha256:ac3c6ee3264d6f5c44c617f90bc7e8b9e1587e7d6708c9d8f811cb65582ee312", size = 227163, upload-time = "2026-09-12T15:12:55.931Z" }, + { url = "https://files.pythonhosted.org/packages/cf/e4/5997f648794bd9b499491d0ff480b096cc9a9c65bdba29f57568e6aa1705/jiter-0.17.0-cp314-cp314-win_arm64.whl", hash = "sha256:6219adaf59711ba7063a52496e8ec6d3fa3e209d7827d83eee3b2abc780a1744", size = 183505, upload-time = "2026-09-12T15:12:58.196Z" }, + { url = "https://files.pythonhosted.org/packages/ac/4a/84a5ec271d09f7590b6073af5ee4abb44eab4ccace453b7e2c5ce45234ca/jiter-0.17.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:59bddbe6f9ffecc68d641e1e2d619ce64cf8a9e9eeb74e5c518f74fc87abf1b0", size = 321527, upload-time = "2026-09-12T15:12:59.394Z" }, + { url = "https://files.pythonhosted.org/packages/39/71/9e1fd0045f5920b4c36be35c3f0f0dfd123668684f8ad352619d7aa44183/jiter-0.17.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6cb41cd1432f1dc19a231cf70b54d42b2c9f05085155859263fce06fa4d41388", size = 340865, upload-time = "2026-09-12T15:13:00.756Z" }, + { url = "https://files.pythonhosted.org/packages/b7/2b/14627fd2bc377f3dd09491bcace6b90e34b4d7fea2f1f3295031ff91f528/jiter-0.17.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:fd7790aa79c8b518e512ebcdfce9f11d8ef5f30efd43720c8a19a548b39fa489", size = 325412, upload-time = "2026-09-12T15:13:02.152Z" }, + { url = "https://files.pythonhosted.org/packages/4c/f3/8d5808f7bf0f456bde79e6393587183a0cee5f83d179fe1f7f1eff2ba067/jiter-0.17.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:dbbfe4e3c21c8166980cddc5bee1a315df082454f007947dfb6fb73800768165", size = 340473, upload-time = "2026-09-12T15:13:03.485Z" }, + { url = "https://files.pythonhosted.org/packages/4f/da/1d8c7c6c4ae6b2423b94a81b6b907d37b28f87664e077427b531bf1b5313/jiter-0.17.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:8c286860abfe8b100cac1c02e225e5776eb9216edd71ba17cdb237da4af32bc9", size = 350757, upload-time = "2026-09-12T15:13:04.828Z" }, + { url = "https://files.pythonhosted.org/packages/eb/96/c1813dcca15c5a370145a448aaea7d1f83f6f0228a5f1130e79340ee385f/jiter-0.17.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f753eb70b1474a29e635e7542ff7312e6d6b951e0b25e8a2e8c34eeb1ddcd478", size = 345203, upload-time = "2026-09-12T15:13:06.131Z" }, + { url = "https://files.pythonhosted.org/packages/d7/f7/fc61cbcf2992d169ede13648fc3fd8e2d3171a3669dde43cd4db556549ac/jiter-0.17.0-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:eae86b1f027031e39db2e0e9c4842221edb7b8cd474d23f87a79b3bd4b651768", size = 328322, upload-time = "2026-09-12T15:13:07.392Z" }, + { url = "https://files.pythonhosted.org/packages/8f/88/46418a3abbdffb7dc41b314200360f24f75faaeb35573e81c92de322cce9/jiter-0.17.0-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:5bf350452a43173e69e1fc74847c57a60e3d7515807287f29849baa2a85d8718", size = 336570, upload-time = "2026-09-12T15:13:08.666Z" }, + { url = "https://files.pythonhosted.org/packages/f0/28/b8a55b949be6306df8888e365a8df05441de8a7b11289f6957004302e41e/jiter-0.17.0-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:da139721f4b7cafdbff580a4f511ea24cb91f4909330c6b926a1ca53836c0a59", size = 482879, upload-time = "2026-09-12T15:13:10.037Z" }, + { url = "https://files.pythonhosted.org/packages/75/3b/21d0afa53ba0680962c39f3eb95ed2946f8793369ed44b0c82b490723081/jiter-0.17.0-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:8079849db9a1371bfd90bad088458a8fb836261879df2233cc9632464ecf64e1", size = 520406, upload-time = "2026-09-12T15:13:11.456Z" }, + { url = "https://files.pythonhosted.org/packages/ef/03/bcbaf8b6b9ea23c2c074411f8ecfbb02d820abac5d0cb8f4e280209174a2/jiter-0.17.0-cp314-cp314t-win32.whl", hash = "sha256:8f770b0c77e5fac482e1ba03ca1a7e18286bfb213d749932a00a7e4cd5de5e06", size = 184434, upload-time = "2026-09-12T15:13:13.037Z" }, + { url = "https://files.pythonhosted.org/packages/7a/b5/5d6ce2c93ef6fe1241b37a9005547f9b6d58db1f07f39fe95807d4b98f51/jiter-0.17.0-cp314-cp314t-win_amd64.whl", hash = "sha256:c4289293e5278d9314b00f15c37f2120fa51d3d68565292e715524c750e775a9", size = 227392, upload-time = "2026-09-12T15:13:14.933Z" }, + { url = "https://files.pythonhosted.org/packages/f5/4b/1e52baf90187606e33a7b8cfa8f96f5829acd7f01870077eb01059ab76d0/jiter-0.17.0-cp314-cp314t-win_arm64.whl", hash = "sha256:4dfbfe5a6e1e80a7082af559f66386405025ec278833e0c649f69cbc6e1004cc", size = 182776, upload-time = "2026-09-12T15:13:16.239Z" }, + { url = "https://files.pythonhosted.org/packages/05/fc/efe3ac75564ab10f53517958f5ccdc231fc7334af66c76776cb554a88967/jiter-0.17.0-cp315-cp315-macosx_10_12_x86_64.whl", hash = "sha256:84963d3f395ef5e9a32ce47155e08a7962fa292c159a10cb98b931cef1416925", size = 292143, upload-time = "2026-09-12T15:13:17.502Z" }, + { url = "https://files.pythonhosted.org/packages/d1/4c/46982118d91f9ffe9714319d21ec4f98d9b7e0cfd9062826c524a54de24e/jiter-0.17.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:ffa0380ad091de7d3fc33e17a97ff479851ee18a0a2a3ee56ff3215cdc886656", size = 321341, upload-time = "2026-09-12T15:13:19.133Z" }, + { url = "https://files.pythonhosted.org/packages/e7/12/9b1ac6ecc6307049913db54839ddba1c11c1ef72c5a8bbb5514bc3b50d1b/jiter-0.17.0-cp315-cp315-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:755079792868ce5d4938e83b91a0939b34fb858a1ca65a104f2d771bea57faa1", size = 344383, upload-time = "2026-09-12T15:13:20.508Z" }, + { url = "https://files.pythonhosted.org/packages/a9/b6/527cc72af836d824e9d4d666e64f0a1ca7eafd662a8da9657b78592172ba/jiter-0.17.0-cp315-cp315-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:3bf4dc2b84a464117fb097d15a25c58d100d2692888e3b0d92df5b48ed16b7c0", size = 326841, upload-time = "2026-09-12T15:13:21.83Z" }, + { url = "https://files.pythonhosted.org/packages/d1/41/567f98617e88005b249503b933803f633ec6ba2d427cf4cc35e5c832125c/jiter-0.17.0-cp315-cp315-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:02a360707033d8cef53f7f3480817a1489177a259ec6ec01e98c37e0b922ddca", size = 341354, upload-time = "2026-09-12T15:13:23.323Z" }, + { url = "https://files.pythonhosted.org/packages/40/da/b29cda895b785f7d426e224638a885b6145a08ce853b381f34afe3e88c5d/jiter-0.17.0-cp315-cp315-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:300ce01ab0215e3dea4d00090143c909aedc65c0f809b3c07983e1d038f291b9", size = 351985, upload-time = "2026-09-12T15:13:26.526Z" }, + { url = "https://files.pythonhosted.org/packages/f7/5c/8a73829e7389e72ea298a450f2b3cb58e71a3e464b45f6d8753740f1c4f5/jiter-0.17.0-cp315-cp315-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:746243a080b4ca790b8499af3d7cf9825d5f5987933950cd818e767ee353d826", size = 346052, upload-time = "2026-09-12T15:13:27.887Z" }, + { url = "https://files.pythonhosted.org/packages/1d/2f/98d6001026932c095ba440925570123043bed29f5ff56158dfe729a9e81b/jiter-0.17.0-cp315-cp315-manylinux_2_31_riscv64.whl", hash = "sha256:b550585523339b71cb852b811aae49d08d7601ad8ffe9f5dc1562f4c3d22fd87", size = 329159, upload-time = "2026-09-12T15:13:31.569Z" }, + { url = "https://files.pythonhosted.org/packages/94/2e/708dc1d2678f092c31c12754e860cd8353e6a85ecbdb1010157edca0da9e/jiter-0.17.0-cp315-cp315-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:0239520085cac678e77a606fd7e3f1c60c371d719790c5e3807388d3da4354c2", size = 336001, upload-time = "2026-09-12T15:13:32.846Z" }, + { url = "https://files.pythonhosted.org/packages/f4/f0/75a5ae38862f4eaf0fe2f8a9fbf6484c4890df04c06dcdffc45e36bca61a/jiter-0.17.0-cp315-cp315-musllinux_1_1_aarch64.whl", hash = "sha256:eb2295da7c3769f6719b227a237aa6a5cfa6550e478bc838001b592c57e16575", size = 484281, upload-time = "2026-09-12T15:13:35.333Z" }, + { url = "https://files.pythonhosted.org/packages/a0/32/6636fae811c27c7f93e1b11fb5800de6a5c9e4269a27cf718e0b31218ad1/jiter-0.17.0-cp315-cp315-musllinux_1_1_x86_64.whl", hash = "sha256:e088612ff90ebc9247e1a43074b72835804261c47e6a6c01cb3ddcb55360d688", size = 521300, upload-time = "2026-09-12T15:13:37.101Z" }, + { url = "https://files.pythonhosted.org/packages/61/aa/12df7e0b0b1a2602e3d5a5a7104d7d9700f254b400f134a9b50955c4d231/jiter-0.17.0-cp315-cp315-win32.whl", hash = "sha256:0b52d52035b3907c5b1f6277857b29c1cbfc965e24e0f27330dbed83edb591ec", size = 185138, upload-time = "2026-09-12T15:13:38.901Z" }, + { url = "https://files.pythonhosted.org/packages/ba/ec/3dd2e495032cddde05723c1f4c743b67a23e55d2af244692a7f58f0cdae3/jiter-0.17.0-cp315-cp315-win_amd64.whl", hash = "sha256:10f5558eed511b830488003449d942bd75829ad6257dc58cb9a03e596a7777b1", size = 226950, upload-time = "2026-09-12T15:13:40.17Z" }, + { url = "https://files.pythonhosted.org/packages/c3/c7/ef85704e0a57e9cadb2babc05f6d7c5df4a1c75da1a6ee31e1986b0099a5/jiter-0.17.0-cp315-cp315-win_arm64.whl", hash = "sha256:fa13acf1046f95df808c64b1310705e143fab87aee73ae00cc42d640867fd2c1", size = 183618, upload-time = "2026-09-12T15:13:41.432Z" }, + { url = "https://files.pythonhosted.org/packages/0e/9a/a4b348349de68762b58d6713973d363ad80a1c741d0bf8def7975f0ecb26/jiter-0.17.0-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:af2f7501580f274b63c4b2283bc425f5df7edf06ae5b171e5f87d912ff359a20", size = 321155, upload-time = "2026-09-12T15:13:42.716Z" }, + { url = "https://files.pythonhosted.org/packages/c1/70/aebd6d0b5f0677de3a3d0bdc4a05fac949b97c4ede454c8809f180ac7b17/jiter-0.17.0-cp315-cp315t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:10c5349312e5cb02b7a21e123a57665afa895953f05bf252a9dd4c13a572b7ab", size = 340985, upload-time = "2026-09-12T15:13:44.115Z" }, + { url = "https://files.pythonhosted.org/packages/a7/82/4c3b49796b5eb62f3f5046f957683f4ba0135fe1a60957c11180512460df/jiter-0.17.0-cp315-cp315t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:86f3f9343a288eb85a81ef20a752b2f84564296636db54a9fff0b5c8deaf1df2", size = 325670, upload-time = "2026-09-12T15:13:45.901Z" }, + { url = "https://files.pythonhosted.org/packages/bc/43/f6341ecb4872202a4ef150486fcee0e1ace4aa3da39b71b82061452cdd3a/jiter-0.17.0-cp315-cp315t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:4607ec7d93355fbc25b8dc5189153cf21d66063b9f9cd04dd2774e6e783f9b6a", size = 340339, upload-time = "2026-09-12T15:13:47.442Z" }, + { url = "https://files.pythonhosted.org/packages/f9/c4/bc2c86e08fa065e03cb2fbc53b367c3640a7d257ef9d877b29118ea636b7/jiter-0.17.0-cp315-cp315t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:10cd64a5720ad7f809ac5466ff1705813f1b6b510f195a73acafba0ac0e1f675", size = 350705, upload-time = "2026-09-12T15:13:48.848Z" }, + { url = "https://files.pythonhosted.org/packages/9d/67/91f12aa111cca6e3a197c3e36bf60a034bf9f122f6d41112a639e44217d8/jiter-0.17.0-cp315-cp315t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:efe9f61bb30174d2f5c8396445c360c96c44e78164d0815dfe627ccf57849574", size = 345011, upload-time = "2026-09-12T15:13:50.215Z" }, + { url = "https://files.pythonhosted.org/packages/f5/cb/9f5556e8f6ec89755fb5a709d8eb8270c9a324e31079eda0dfbeca451b6e/jiter-0.17.0-cp315-cp315t-manylinux_2_31_riscv64.whl", hash = "sha256:370d8fe5bf201dc6925e8a84c81ac7291f74d9fd1778234fc79d517064a5c76b", size = 328268, upload-time = "2026-09-12T15:13:51.809Z" }, + { url = "https://files.pythonhosted.org/packages/22/98/153f20680fb75781a490fb849940e2b00f95035c7aa054df592f36ed33fc/jiter-0.17.0-cp315-cp315t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:6b303d88e6a0bda789ec4b7801c7bad68e27230ba1fe4baffc756d1fbd32dc9d", size = 337024, upload-time = "2026-09-12T15:13:53.095Z" }, + { url = "https://files.pythonhosted.org/packages/af/59/b16c9be3a5035df4466cc72e888188c027562de90a723d290ab6814cb9d4/jiter-0.17.0-cp315-cp315t-musllinux_1_1_aarch64.whl", hash = "sha256:30793a24a31e968969757c9e08d830cbb15a2cd3c4959b4498b38f4b1c2258eb", size = 482766, upload-time = "2026-09-12T15:13:55.713Z" }, + { url = "https://files.pythonhosted.org/packages/d0/55/667dea313094024bef082175d6bfe8976f90d1c00c926af9df1d8e0eab48/jiter-0.17.0-cp315-cp315t-musllinux_1_1_x86_64.whl", hash = "sha256:686c93d86f2b426c803024b805bd161a6cd10e9627c23e901640eab646c0ad8a", size = 520367, upload-time = "2026-09-12T15:13:57.674Z" }, + { url = "https://files.pythonhosted.org/packages/21/e3/4b1a43501fb9ed17b01d137e380cb0e8fdcb39a254ce31aa2ab95bc861ac/jiter-0.17.0-cp315-cp315t-win32.whl", hash = "sha256:86d703d9faa1ffc8ae4e9de0fa007712ed2171b5c0d93811a8e2e105ac729b0d", size = 184603, upload-time = "2026-09-12T15:13:59.27Z" }, + { url = "https://files.pythonhosted.org/packages/f9/f2/b8ee0372b6ebdf1bde5cc44495d5291d17f961065f5b48f8616cc67cac2e/jiter-0.17.0-cp315-cp315t-win_amd64.whl", hash = "sha256:42b0260445251b1bc520a63baa94a32d88e0f931fba234f1764db7feb7c72174", size = 227936, upload-time = "2026-09-12T15:14:00.472Z" }, + { url = "https://files.pythonhosted.org/packages/a4/b4/923a1215daba959aed8355973315cb3f81f53e0d01c5b211870a27b41f45/jiter-0.17.0-cp315-cp315t-win_arm64.whl", hash = "sha256:d47687806f9c54c84ea38733507081337922beca90ce819c7d852dd485bc0f23", size = 182977, upload-time = "2026-09-12T15:14:01.799Z" }, + { url = "https://files.pythonhosted.org/packages/b9/3b/05a917204413e2e09906dfa35240c1021227aeb56c7305abea9562c598b4/jiter-0.17.0-graalpy311-graalpy242_311_native-macosx_10_12_x86_64.whl", hash = "sha256:eaba834b72d573547b9d966465b3394b749d5e14208cc70acb63aca37619ab33", size = 288143, upload-time = "2026-09-12T15:14:02.998Z" }, + { url = "https://files.pythonhosted.org/packages/d9/e1/a1cd3c0cf8f79945939e4f8caae9990529f67d7f77f671769f73956329b9/jiter-0.17.0-graalpy311-graalpy242_311_native-macosx_11_0_arm64.whl", hash = "sha256:51e1519d676a9f14dad9c2a411170d43b022ddb7989562df4e849b261ce127b2", size = 281847, upload-time = "2026-09-12T15:14:04.414Z" }, + { url = "https://files.pythonhosted.org/packages/72/b4/9b797679e09f4a46c32986aeb3670bd9bc562fc0b373c7a0ee5c5dce1206/jiter-0.17.0-graalpy311-graalpy242_311_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d0ce4feb52493e3513335b2accdcd75605652e4632772d3c8c2f7b86954d7f39", size = 304797, upload-time = "2026-09-12T15:14:05.733Z" }, + { url = "https://files.pythonhosted.org/packages/25/4a/0d77415b27a00d970e4e710f7c1de62e96a11c4cab3ed1add0015af04626/jiter-0.17.0-graalpy311-graalpy242_311_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:29f49b325e0234e4ad9ecca5b861ffbd09b95ccac9bd46fa55841b6e56eea5fe", size = 307815, upload-time = "2026-09-12T15:14:07.105Z" }, + { url = "https://files.pythonhosted.org/packages/17/31/4bb27f54333d3b9ef1e5bd3312dc0b4bbe59c68bb0885fdb40583a6b1567/jiter-0.17.0-graalpy312-graalpy250_312_native-macosx_10_12_x86_64.whl", hash = "sha256:454c4997d73cc466c71fd565d91e603b0274e48ea0c6b0b7a7aee6967e4ceb7c", size = 288415, upload-time = "2026-09-12T15:14:08.455Z" }, + { url = "https://files.pythonhosted.org/packages/28/30/879570ecf82574eaea77c5eb10309f4b630dece5f2a556e9814a90ba3f2d/jiter-0.17.0-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:40d2c240f8f80b5b0f201b29f0ae129c81448c60c772227a41747b5e0026f6a2", size = 279113, upload-time = "2026-09-12T15:14:10.117Z" }, + { url = "https://files.pythonhosted.org/packages/77/7a/1f0b8a35fbd079a4f1752c31a15dc99cf277f863747c459be0af39e900e5/jiter-0.17.0-graalpy312-graalpy250_312_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3e05f5adbf68c4bd11e1610f394034d984152988e84be6f8314235ce6f2139e5", size = 303708, upload-time = "2026-09-12T15:14:11.445Z" }, + { url = "https://files.pythonhosted.org/packages/e1/8b/d76219ebdbcf3d4209d9d21a0810db4c8d0a6f88e3ee87d30bdea4e90d30/jiter-0.17.0-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d2c0bf24c72fd0491405dce5d40194f2070e9021ce648c1a1d46234b93d848ff", size = 307147, upload-time = "2026-09-12T15:14:12.897Z" }, +] + [[package]] name = "jmespath" version = "1.1.0" @@ -1467,6 +1624,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, ] +[[package]] +name = "sniffio" +version = "1.3.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a2/87/a6771e1546d97e7e041b6ae58d80074f81b7d5121207425c964ddf5cfdbd/sniffio-1.3.1.tar.gz", hash = "sha256:f4324edc670a0f49750a81b895f35c3adb843cca46f0530f79fc1babb23789dc", size = 20372, upload-time = "2024-02-25T23:20:04.057Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e9/44/75a9c9421471a6c4805dbf2356f7c181a29c1879239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2", size = 10235, upload-time = "2024-02-25T23:20:01.196Z" }, +] + [[package]] name = "sse-starlette" version = "3.4.11" From 288483087e5d434bdb75f0721b1b8c19582b8193 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Thu, 1 Oct 2026 01:33:55 +0800 Subject: [PATCH 13/43] feat: run APITestka requests as a LoadDensity load test Request actions or recorded traffic become LoadDensity HTTP tasks (URL, method, request kwargs, status assertion). The runner starts `python -m je_load_density --execute_file` out of process and judges the run from LoadDensity's summary report, since LoadDensity exits 0 even when an action fails; failure-rate and p95 limits are optional. Adds `apitestka load convert|run`, AT_write_load_test and AT_run_load_test, and records the LoadDensity contract in section 6. --- CLAUDE.md | 2 +- README.md | 28 ++- README/README_zh-CN.md | 26 ++- README/README_zh-TW.md | 26 ++- architecture.md | 18 +- docs/source/Eng/Eng_index.rst | 2 +- docs/source/Eng/doc/cli/cli_doc.rst | 2 + .../Eng/doc/integrations/integrations_doc.rst | 38 ++++ docs/source/Zh/Zh_index.rst | 2 +- docs/source/Zh/doc/cli/cli_doc.rst | 2 + .../Zh/doc/integrations/integrations_doc.rst | 37 +++ docs/updates/2026-10.md | 12 + docs/updates/README.md | 3 +- je_api_testka/cli/cli_main.py | 3 + je_api_testka/cli/completion.py | 2 +- je_api_testka/cli/load_cli.py | 100 +++++++++ je_api_testka/integrations/__init__.py | 16 ++ je_api_testka/integrations/load_density.py | 139 ++++++++++++ .../integrations/load_density_commands.py | 70 ++++++ .../integrations/load_density_runner.py | 143 ++++++++++++ .../utils/executor/action_executor.py | 4 + progress.md | 1 - test/test_executor_integration.py | 1 + test/test_integrations/test_load_density.py | 211 ++++++++++++++++++ 24 files changed, 876 insertions(+), 12 deletions(-) create mode 100644 je_api_testka/cli/load_cli.py create mode 100644 je_api_testka/integrations/load_density.py create mode 100644 je_api_testka/integrations/load_density_commands.py create mode 100644 je_api_testka/integrations/load_density_runner.py create mode 100644 test/test_integrations/test_load_density.py diff --git a/CLAUDE.md b/CLAUDE.md index 1ac6e6c..a16b49e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -57,7 +57,7 @@ je_api_testka/ ├── contract/ # Pact-style consumer contracts, provider verification, OpenAPI check ├── security/ # Auth helpers, header scan, fuzzing, pip-audit wrapper ├── runner/ # Parallel runner, tag filter, dependency ordering -├── integrations/ # cURL and HAR import, webhook notify, GitHub PR comment +├── integrations/ # cURL and HAR import, webhook notify, GitHub PR comment, LoadDensity bridge ├── ai/ # Pluggable text-completion backend (no-op by default; Anthropic reference backend) ├── cli/ # Subcommand CLI (`apitestka`) ├── mcp_server/ # MCP stdio server (`apitestka-mcp`) diff --git a/README.md b/README.md index 2c9dd7d..f2da3c3 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ server for Claude, and more. | **Mock server** | Static, dynamic, stateful, fault injection, OpenAPI-driven, Jinja templating, webhook receiver, record-replay proxy, WebSocket and gRPC endpoints | | **Runner** | Sequential & parallel execution, tag filters, dependency-aware ordering, retry policies | | **Reports** | HTML / JSON / XML / **JUnit / Allure / Markdown** / shields.io badge / SQLite trend store / run diff | -| **Integrations** | Slack / Teams / Discord webhook, GitHub PR comment, cURL & HAR importers, OpenAPI / Postman importer | +| **Integrations** | Slack / Teams / Discord webhook, GitHub PR comment, cURL & HAR importers, OpenAPI / Postman importer, LoadDensity load-test bridge | | **CLI / DX** | Subcommand CLI, REPL, terminal summary, shell completion, scaffold | | **Security** | Auth helpers (Basic / Bearer / JWT / AWS SigV4), header / CORS / rate-limit / SSRF probes, pip-audit wrapper, fuzz inputs | | **Spec inference** | Test record → OpenAPI, JSON Schema inference, OpenAPI changelog | @@ -341,6 +341,31 @@ Each importer returns `["AT_test_api_method", {...}]` actions that `execute_acti as they are. Runner metadata (`id`, `depends_on`, `tags`) may stay in the kwargs; the executor strips it before the call. +**Load testing with LoadDensity.** The same requests can run as a load test on +[LoadDensity](https://github.com/Integration-Automation/LoadDensity) (Locust). `AT_test_api_method` and +`AT_test_api_method_httpx` actions, or recorded traffic from saved reports, become LoadDensity HTTP tasks: +URL, method, `params`, `headers`, `cookies`, `json`, `data`, `timeout`, `allow_redirects` and `verify` carry +over, and `result_check_dict["status_code"]` becomes a status assertion. Other actions, and relative URLs, +are left out and listed. `load run` starts LoadDensity in its own process +(`python -m je_load_density --execute_file`) and judges the run from its summary: no requests, a +failure rate above `--max-failure-rate` or a p95 above `--max-p95-ms` exits with 1. + +```bash +pip install je_load_density # in this interpreter, or point --python at another one +apitestka load convert --actions smoke.json -o load.json --users 20 --time 30 +apitestka load run --actions smoke.json --users 20 --spawn-rate 5 --time 30 \ + --max-failure-rate 0.01 --max-p95-ms 500 +apitestka load run --report run_success.json --time 60 --python /opt/ld/bin/python +``` + +JSON actions: `AT_write_load_test` writes the LoadDensity file; `AT_run_load_test` runs it and fails the +action when a threshold is broken. + +```json +["AT_run_load_test", {"action_file": "smoke.json", "profile": {"user_count": 20, "test_time": 30}, + "thresholds": {"max_failure_rate": 0.01, "max_p95_ms": 500}}] +``` + ### CLI / Developer Experience ```bash @@ -357,6 +382,7 @@ apitestka openapi --report run_success.json -o openapi.json # infer a spec fro apitestka openapi --run actions.json # run first, print the inferred spec apitestka contract verify pacts/web-shop.json --base-url http://localhost:8000 # see Contract Testing apitestka generate-tests openapi.json -o actions.json [--ai anthropic] # see Pluggable AI Backend +apitestka load run --actions smoke.json --users 20 --time 30 # see Integrations ``` ### Security Probes diff --git a/README/README_zh-CN.md b/README/README_zh-CN.md index b48b9e4..e737997 100644 --- a/README/README_zh-CN.md +++ b/README/README_zh-CN.md @@ -57,7 +57,7 @@ record-replay 代理、安全检测、并行执行 runner,以及给 Claude 用 | **模拟服务器** | 静态、动态、stateful、故障注入、OpenAPI 驱动、Jinja 模板、Webhook 接收、record-replay 代理、WebSocket 与 gRPC 端点 | | **Runner** | 顺序与并行、Tag 过滤、Dependency-aware 排序、Retry 策略 | | **报告** | HTML / JSON / XML / **JUnit / Allure / Markdown** / shields.io badge / SQLite 趋势库 / Run diff | -| **生态集成** | Slack / Teams / Discord webhook、GitHub PR comment、cURL & HAR 导入、OpenAPI / Postman 导入 | +| **生态集成** | Slack / Teams / Discord webhook、GitHub PR comment、cURL & HAR 导入、OpenAPI / Postman 导入、LoadDensity 负载测试桥接 | | **CLI / DX** | 子命令式 CLI、REPL、终端摘要、Shell completion、Scaffold | | **安全** | Auth helper(Basic / Bearer / JWT / AWS SigV4)、Header / CORS / Rate limit / SSRF probe、pip-audit、Fuzz | | **Spec 反推** | 测试记录 → OpenAPI、JSON Schema 推断、OpenAPI changelog | @@ -337,6 +337,29 @@ actions = convert_spec_file("openapi.json", spec_format="openapi") 可以直接执行。runner 的元数据(`id`、`depends_on`、`tags`)可以留在 kwargs 里, executor 调用前会先去掉。 +**用 LoadDensity 做负载测试。** 同一批请求可以在 +[LoadDensity](https://github.com/Integration-Automation/LoadDensity)(Locust)上运行负载测试。 +`AT_test_api_method` 与 `AT_test_api_method_httpx` action,或保存报告里录到的流量,会变成 LoadDensity 的 +HTTP task:URL、method、`params`、`headers`、`cookies`、`json`、`data`、`timeout`、`allow_redirects` 与 +`verify` 原样带过去,`result_check_dict["status_code"]` 变成状态码断言。其他 action 与相对网址会被跳过并列出。 +`load run` 在独立进程启动 LoadDensity(`python -m je_load_density --execute_file`),并根据它的摘要判断结果: +没有任何请求、失败率超过 `--max-failure-rate` 或 p95 超过 `--max-p95-ms` 时退出码为 1。 + +```bash +pip install je_load_density # 装在这个解释器,或用 --python 指定另一个 +apitestka load convert --actions smoke.json -o load.json --users 20 --time 30 +apitestka load run --actions smoke.json --users 20 --spawn-rate 5 --time 30 \ + --max-failure-rate 0.01 --max-p95-ms 500 +apitestka load run --report run_success.json --time 60 --python /opt/ld/bin/python +``` + +JSON action:`AT_write_load_test` 写出 LoadDensity 文件;`AT_run_load_test` 执行它,超过阈值时该 action 失败。 + +```json +["AT_run_load_test", {"action_file": "smoke.json", "profile": {"user_count": 20, "test_time": 30}, + "thresholds": {"max_failure_rate": 0.01, "max_p95_ms": 500}}] +``` + ### CLI / 开发体验 ```bash @@ -353,6 +376,7 @@ apitestka openapi --report run_success.json -o openapi.json # 从保存的运 apitestka openapi --run actions.json # 先运行再打印反推的规格 apitestka contract verify pacts/web-shop.json --base-url http://localhost:8000 # 见契约测试 apitestka generate-tests openapi.json -o actions.json [--ai anthropic] # 见可插拔 AI 后端 +apitestka load run --actions smoke.json --users 20 --time 30 # 见生态集成 ``` ### 安全检测 diff --git a/README/README_zh-TW.md b/README/README_zh-TW.md index 007a48e..490fc3f 100644 --- a/README/README_zh-TW.md +++ b/README/README_zh-TW.md @@ -57,7 +57,7 @@ record-replay proxy、安全性檢測、平行執行 runner,以及給 Claude 用 | **模擬伺服器** | 靜態、動態、stateful、故障注入、OpenAPI 驅動、Jinja 模板、Webhook 接收、record-replay proxy、WebSocket 與 gRPC 端點 | | **Runner** | 順序與平行執行、Tag 過濾、Dependency-aware 排序、Retry 策略 | | **報告** | HTML / JSON / XML / **JUnit / Allure / Markdown** / shields.io badge / SQLite 趨勢資料庫 / Run diff | -| **生態整合** | Slack / Teams / Discord webhook、GitHub PR comment、cURL & HAR 匯入、OpenAPI / Postman 匯入 | +| **生態整合** | Slack / Teams / Discord webhook、GitHub PR comment、cURL & HAR 匯入、OpenAPI / Postman 匯入、LoadDensity 負載測試橋接 | | **CLI / DX** | 子命令式 CLI、REPL、終端摘要、Shell completion、Scaffold | | **安全** | Auth helper(Basic / Bearer / JWT / AWS SigV4)、Header / CORS / Rate limit / SSRF probe、pip-audit、Fuzz | | **Spec 反推** | 測試紀錄 → OpenAPI、JSON Schema 推斷、OpenAPI changelog | @@ -337,6 +337,29 @@ actions = convert_spec_file("openapi.json", spec_format="openapi") 可以直接執行。runner 的中繼資料(`id`、`depends_on`、`tags`)可以留在 kwargs 裡, executor 呼叫前會先拿掉。 +**用 LoadDensity 做負載測試。** 同一批請求可以在 +[LoadDensity](https://github.com/Integration-Automation/LoadDensity)(Locust)上跑負載測試。 +`AT_test_api_method` 與 `AT_test_api_method_httpx` action,或存下報告裡錄到的流量,會變成 LoadDensity 的 +HTTP task:URL、method、`params`、`headers`、`cookies`、`json`、`data`、`timeout`、`allow_redirects` 與 +`verify` 原樣帶過去,`result_check_dict["status_code"]` 變成狀態碼斷言。其他 action 與相對網址會被略過並列出。 +`load run` 在獨立行程啟動 LoadDensity(`python -m je_load_density --execute_file`),並依它的摘要判斷結果: +沒有任何請求、失敗率超過 `--max-failure-rate` 或 p95 超過 `--max-p95-ms` 時結束碼為 1。 + +```bash +pip install je_load_density # 裝在這個直譯器,或用 --python 指定另一個 +apitestka load convert --actions smoke.json -o load.json --users 20 --time 30 +apitestka load run --actions smoke.json --users 20 --spawn-rate 5 --time 30 \ + --max-failure-rate 0.01 --max-p95-ms 500 +apitestka load run --report run_success.json --time 60 --python /opt/ld/bin/python +``` + +JSON action:`AT_write_load_test` 寫出 LoadDensity 檔案;`AT_run_load_test` 執行它,超過門檻時該 action 失敗。 + +```json +["AT_run_load_test", {"action_file": "smoke.json", "profile": {"user_count": 20, "test_time": 30}, + "thresholds": {"max_failure_rate": 0.01, "max_p95_ms": 500}}] +``` + ### CLI / 開發體驗 ```bash @@ -353,6 +376,7 @@ apitestka openapi --report run_success.json -o openapi.json # 從存下的執 apitestka openapi --run actions.json # 先執行再印出反推的規格 apitestka contract verify pacts/web-shop.json --base-url http://localhost:8000 # 見契約測試 apitestka generate-tests openapi.json -o actions.json [--ai anthropic] # 見可插拔 AI 後端 +apitestka load run --actions smoke.json --users 20 --time 30 # 見生態整合 ``` ### 安全檢測 diff --git a/architecture.md b/architecture.md index aa23003..85618f5 100644 --- a/architecture.md +++ b/architecture.md @@ -36,7 +36,7 @@ CLI, a TCP socket server, an MCP server, a pytest plugin and an optional PySide6 | `je_api_testka/contract/` | Pact v2 consumer contracts from records (`pact.py`), matching rules (`matching.py`), provider verification (`verifier.py`), bidirectional check against an OpenAPI document (`openapi_compat.py`, `openapi_schema.py`), file-level steps for the executor and CLI (`commands.py`) | | `je_api_testka/security/` | Auth header helpers, CORS/SSRF/rate-limit probes, header scan, fuzzing, `pip-audit` wrapper | | `je_api_testka/runner/` | Parallel runner, tag filter, dependency ordering of actions | -| `je_api_testka/integrations/` | cURL and HAR import, webhook notify, GitHub PR comment | +| `je_api_testka/integrations/` | cURL and HAR import, webhook notify, GitHub PR comment; LoadDensity bridge (`load_density.py` converts requests to LoadDensity tasks, `load_density_runner.py` runs them out of process and judges the summary, `load_density_commands.py` for the executor and CLI) | | `je_api_testka/ai/` | Pluggable text-completion backend (no-op by default) used by test generation, failure classification and fake payloads; `AnthropicAIBackend` (`ai` extra) is the reference implementation; selection by `set_ai_backend` / `select_ai_backend` / `AT_select_ai_backend` or `APITESTKA_AI_BACKEND` on first use | | `je_api_testka/cli/` | Subcommand CLI (`apitestka`): import, REPL, scaffold, completion, terminal summary | | `je_api_testka/mcp_server/` | MCP stdio server and its tool catalogue | @@ -61,8 +61,8 @@ CLI, a TCP socket server, an MCP server, a pytest plugin and an optional PySide6 `-c/--create_project` and `--execute_str`. On `win32`/`cygwin`/`msys`, `--execute_str` is decoded with `json.loads` twice. Errors print `repr(error)` to stderr and exit with code 1. - **Subcommand CLI**: `apitestka` (`je_api_testka.cli.cli_main:main`) with `run`, `create`, `mock`, - `import`, `repl`, `summary`, `scaffold`, `completion`, `mcp`, `openapi`, `contract record|verify|compare` and - `generate-tests` + `import`, `repl`, `summary`, `scaffold`, `completion`, `mcp`, `openapi`, `contract record|verify|compare`, + `generate-tests` and `load convert|run` (`cli/load_cli.py`) (`cli/contract_cli.py`); shared helpers are in `cli/cli_common.py`. `cli/completion.py` `SUBCOMMANDS` lists the same names (a test compares them with the parser). - **MCP**: `apitestka-mcp`, `python -m je_api_testka.mcp_server` or `apitestka mcp` (stdio, needs the @@ -134,6 +134,18 @@ MCP host → apitestka-mcp → build_server → dispatch_tool(name, args) → AP - **TestPioneer**: `with: api-runner` imports `je_api_testka.execute_action` in-process (`test_pioneer/executor/run/utils.py`). `parallel_run` spawns `python -m je_api_testka --execute_file