feat: decoupled parser adapter architecture for continuous Dart SDK version support

[caverav]

Problem

Dart AOT does not maintain a stable ABI. Across minor Dart and Flutter releases, the VM changes snapshot magic hashes, Class IDs (CIDs), struct field offsets, and occasionally the underlying clustered serialization algorithms and reference encoding formats.

Attempting to address this by releasing a full flutterdec binary per Flutter/Dart version is unmaintainable:

1. Release explosion: Rebuilding and maintaining 100+ full decompiler binaries across release matrices.
2. Maintenance overhead: Bug fixes in ARM64 disassembly, IR generation, CFG structuring, or pseudocode lifting would require backporting to dozens of version branches.
3. Misalignment with tool architecture: >90% of flutterdec (instruction decoding, LLIR, CFG structuring, expression lifters, quality gates, and export script emitters) is Dart-version-agnostic.

Conversely, relying exclusively on static layout tables (data/dart-profiles.json) only solves identification and static field offsets; it cannot deserialize snapshots when cluster encoding formats or serializer logic evolve in new Dart VM generations.

---

Proposed Solution

Maintain a single, unified flutterdec core binary and decouple snapshot parsing into independently versioned parser adapters keyed by snapshot hash.

[Target Binary (libapp.so)] 
        │
        ├──> flutterdec Loader (Extracts 32-byte Snapshot Hash)
        │
        ├──> Exact Adapter Resolution (`dart_adapter_<hash>`)
        │       ├── Exact Hash Match  --> Executes dedicated parser backend (Serwalker / r2flutter / compiled parser)
        │       └── Unmapped / Beta   --> Verified fallback or graceful degradation to instruction carving
        │
        └──> ProgramModel JSON (Schema v3)
                │
                └──> Unified flutterdec Core Engine (Disasm -> IR -> CFG -> Decompiler)

Key Components

1. Strict Core / Adapter Contract (ProgramModel Schema v3):

   • Keep the adapter boundary as a strictly validated JSON contract (schemas/adapter.schema.json).
   • The core consumes ProgramModel (libraries, classes, functions, object_pool, pool_geometry) without embedding Dart VM version-specific C++ headers.

2. CI-Driven Adapter & Profile Harvesting:

   • Implement automated CI jobs tracking the official Dart SDK repository.
   • For each new Dart release/tag:
     • Extract the 32-byte snapshot hash, CIDs, and struct layout constants into data/dart-profiles.json.
     • Build and package the corresponding lightweight parser adapter artifact.
   • Publish signed adapter artifacts that can be installed via CLI:

     flutterdec adapter install --dart-hash <HASH>

3. Safe Fallback & Graceful Degradation Policy:

   • Exact match by default: Require exact snapshot hash matching for high-confidence deserialization.
   • Opt-in family compatibility: Allow experimental fallback to a tested family profile only when strict invariant checks pass (e.g., valid entry VAs, bounded pool indices, non-corrupted string tables).
   • Graceful fallback: If a snapshot format is unknown or unsupported, degrade cleanly to instruction stream analysis (isolate_instructions), function prologue recovery, and dispatch table call detection (sel<offset>) rather than failing fatally or emitting plausible-but-hallucinated metadata.

---

Alternatives Considered

1. One full flutterdec binary per Flutter version:
   • Trade-off: High release maintenance burden, broken UX, duplicated build artifacts for components that never change between Dart versions.
2. Monolithic in-core Rust deserializer:
   • Trade-off: Heavy maintenance burden to mirror Dart VM C++ changes in Rust; high risk of silent deserialization breakage when VM serialization internals shift.

---

Scope and Risks

• Affected crates/modules:
  • crates/flutterdec-adapter: Adapter registry management, artifact verification, and manifest resolution.
  • adapters/: Reference adapter templates and native backend integrations (Serwalker / r2flutter / blutter).
  • data/dart-profiles.json: Continued profile updates for static identification.
  • CI / GitHub Actions: Automated scraping and build pipelines for Dart SDK tags.
• Expected impact on performance/quality:
  • Zero performance regression on the core disassembly/decompilation pipeline.
  • Faster onboarding of new Flutter/Dart versions without binary rebuilds.
  • Eliminates false-positive metadata by enforcing strict hash matching and schema validation.
• Test strategy:
  • Acceptance tests verifying that fixtures from at least two distinct Dart snapshot families produce valid Schema v3 ProgramModel outputs.
  • Test verifying unknown snapshot hashes take the safe fallback path without fabricating metadata.
  • Negative test suite ensuring corrupt or mismatched adapter artifacts fail validation gracefully on checksum, snapshot hash, and schema checks.
  • Determinism & regression test ensuring existing decompilation golden outputs remain unchanged.

---

Proposed Next Steps

1. Formalize the adapter distribution manifest schema (version, snapshot_hash, schema_version, os/arch, checksum).
2. Set up automated CI workflows to track new Dart SDK releases and produce layout profiles / adapters.
3. Validate Serwalker integration as a primary native snapshot parser backend emitting Schema v3 ProgramModel.

[caverav]

cc @kalixtez

[caverav]

Recomendation

This issue identifies the correct architectural boundary: keep one version-independent flutterdec core and isolate Dart snapshot parsing behind an adapter boundary.

One important refinement: the exact snapshot hash must remain the compatibility key, but it should not automatically define a separate parser implementation or executable. Parser code should be versioned by snapshot grammar family. Layout constants and other data should be versioned by profile. Exact hash records should provide the compatibility evidence connecting them.

The target architecture should be:

Target binary
    |
    v
Loader extracts SnapshotIdentity
    |
    v
Exact compatibility registry lookup
    |
    +-- parser family
    +-- data profile
    +-- verified adapter artifact
    |
    v
Adapter protocol
    |
    v
ProgramModel
    |
    v
Strict semantic validator
    |
    v
Version-independent core

Unknown identities must use an explicit machine-code-only fallback. They must never select a parser by proximity, Dart semantic version, filename, or heuristic hash matching.

1. Compatibility identity

The registry should match an exact identity containing:

• the 32-character lowercase hexadecimal snapshot hash;
• snapshot kind, such as FullAOT;
• target architecture;
• normalized layout-affecting feature tuple, including compressed pointers;
• optionally, a digest of the normalized feature set.

The Dart snapshot hash is an MD5 digest represented as 32 hexadecimal characters. It is distinct from the snapshot magic number and from the Dart semantic version.

dart_version and flutter_version should remain informational metadata. They must not determine parser compatibility.

Dart generates this hash from VM snapshot serializer sources, and the VM performs exact snapshot-version equality checks when loading a snapshot:

• https://github.com/dart-lang/sdk/blob/3.11.1/tools/make_version.py
• https://github.com/dart-lang/sdk/blob/3.11.1/runtime/vm/app_snapshot.cc#L9337-L9366

A changed hash means that compatibility cannot be assumed. It does not prove that the serialization grammar changed. Conversely, an unchanged or reused profile is not enough to authorize parsing without an exact registry record.

2. Separate parser families, profiles, and compatibility records

These concepts should remain independent.

Parser family

A parser family owns behavior that depends on serialized snapshot grammar:

• clustered serialization;
• reference encoding;
• cluster ordering;
• ReadAlloc and ReadFill;
• object roots;
• dispatch tables;
• post-load behavior;
• version-specific parser logic.

One parser family may support multiple exact snapshot identities, but only after real fixtures and invariant checks prove that the implementation remains valid.

Data profile

A profile contains data-only layout information:

• CIDs;
• object-header rules;
• field offsets;
• alignment;
• pointer width;
• pool geometry;
• instruction-table layout;
• feature-dependent constants.

Profiles should be independently updateable and content-addressed. They should not be compiled into the core with include_str!; adding or correcting a profile should not require rebuilding the main decompiler.

Compatibility record

Every supported hash should have an exact registry record containing:

snapshot_hash
snapshot_kind
target_arch
features_digest
dart_version                 # informational
flutter_version              # informational
parser_family_id
profile_id
profile_sha256
protocol_major
model_major
compatibility_evidence
trust_tier
host artifact variants:
  host_os
  host_arch
  artifact_sha256
  artifact_size
  provenance

Multiple exact records may point to the same parser artifact and profile. A new hash therefore does not automatically require a new executable.

The registry is the authority. Adapter output must not be allowed to declare itself authoritative merely by returning a matching hash or an adapter_kind string.

3. ProgramModel contract

The current Schema v3 contract is too weak for this boundary. Provenance, capabilities, unavailable values, and semantic validation change the meaning of the model, not just its JSON shape.

Because this project is still pre-1.0, a clean ProgramModel v4 cutover is preferable to permanently accepting v2, v3, and v4.

ProgramModel v4 should include:

• explicit schema name and major/minor version;
• producer ID and producer version;
• adapter artifact digest and trust tier;
• observed snapshot identity;
• snapshot kind and normalized features;
• input region names, sizes, virtual addresses, and SHA-256 digests;
• selected compatibility-record ID and digest;
• parser-family ID and digest;
• profile ID and digest;
• per-domain capability state:
  • complete;
  • partial;
  • unavailable;
• provenance for facts that are exact, derived, or heuristic;
• stable IDs and referential integrity;
• code ranges separate from semantic function records;
• diagnostics and parser limitations;
• one controlled extensions object.

Placeholder strings must not masquerade as semantic values. For example, an unresolved class, library, owner, or name should be represented as unavailable with a diagnostic, not as an apparent exact name such as <unresolved ...>.

Likewise, heuristic pool information must not be assigned confidence: 1.0. Use typed provenance such as exact, derived, and heuristic. Numeric confidence should only be added if it has a defined and calibrated meaning.

JSON Schema should be generated from canonical Rust DTOs. The schema validates shape; Rust validation must enforce semantic invariants.

Required invariants include:

• input identity equals adapter identity;
• adapter identity equals the selected registry record;
• parser family and profile match the selected record;
• all IDs are unique;
• all references are valid;
• virtual-address arithmetic is checked;
• code ranges stay within executable regions;
• function ranges are valid and non-overlapping where required;
• pool indices and pool geometry are bounded;
• capability states agree with populated fields;
• authoritative names cannot originate from heuristic sources;
• ordering is deterministic;
• unavailable data is not represented as fabricated semantic metadata.

4. Adapter protocol

A full JSON-RPC protocol is unnecessary for the initial one-process-per-analysis design.

Start with protocol v1:

• one JSON request on stdin;
• request includes protocol/model versions;
• request includes the selected compatibility-record digest;
• snapshot data is passed through immutable files, not base64-encoded JSON;
• request includes input paths, sizes, SHA-256 digests, and virtual addresses;
• adapter writes its model to an assigned output path;
• adapter returns one structured status result on stdout;
• stderr is diagnostic-only and bounded;
• the host validates output size, digest, schema, semantics, and identity before consuming it.

The host should create private input/output directories for every invocation.

The protocol should not allow an adapter to choose its own compatibility record. The host resolves the exact registry entry first and passes that decision to the adapter.

Add JSON-RPC later only if long-lived adapters, streaming, remote workers, or concurrent calls become real requirements.

Native subprocesses should be the first execution format. In-process shared libraries reduce startup overhead but allow parser bugs to crash or compromise the main process. WASM may be useful later for distributing third-party adapters, but it does not solve grammar classification or compatibility admission and should not block the native design.

5. Adapter containment and artifact verification

Before executing an adapter, the host should verify:

• registry record exists for the exact identity;
• artifact SHA-256 matches;
• host OS and architecture match;
• target architecture matches;
• parser family and profile digests match;
• artifact path cannot escape the installation directory;
• artifact is installed atomically;
• protocol and model versions are supported.

The execution boundary should provide:

• wall-clock timeout;
• bounded stdout and stderr;
• bounded output file size;
• memory and process limits where available;
• private working directory;
• cleared environment except for an allow-list;
• no inherited unrelated file descriptors;
• no network access where sandbox support exists;
• process-tree termination on timeout;
• cleanup on success and failure.

Checksums provide integrity, not publisher authenticity. Automatic remote installation should wait until the registry and artifacts support signing, rollback protection, and freeze protection through TUF, Sigstore, or an equivalent mechanism.

Initial distribution should be local/bundled and install-only. Do not add automatic network installation before the trust model is complete.

6. Honest fallback behavior

Unknown or unsupported snapshot identities should not require a fake hash-named adapter.

The fallback may provide:

• executable regions;
• architecture-derived facts;
• recovered function candidates;
• direct branch and call destinations;
• dispatch-table call detection;
• explicit heuristic function IDs;
• clear capability limitations.

The fallback must mark these unavailable unless independently recovered:

• Dart libraries;
• class hierarchy;
• original semantic names;
• authoritative ObjectPool meaning;
• arbitrary object identities;
• semantic entry functions.

It must not synthesize values such as:

package:app/main.dart
Global
main

merely to satisfy a non-empty model check.

This fallback is a useful product behavior, not an adapter failure. It lets unknown versions remain analyzable without emitting plausible but untrusted metadata.

7. Serwalker integration

The current Serwalker work is a good first parser candidate. It already demonstrates several useful properties:

• exact Dart 3.11.1 snapshot-hash checking;
• rejection of unsupported hashes;
• FullAOT-only validation;
• compressed-pointer validation;
• separation of snapshot, cluster, root, instruction-table, raw-object, and model-producer code.

However, it should not be merged with its current core integration shape.

Required changes:

1. Package Serwalker as an adapter executable implementing protocol v1.
2. Remove the direct flutterdec-core dependency on the Serwalker crate.
3. Do not route Serwalker through AdapterBackend::Internal.
4. Preserve Internal for machine-code-only fallback.
5. Select Serwalker through an exact registry record.
6. Emit ProgramModel v4 rather than hardcoded Schema v3.
7. Move data-only layout constants into a referenced profile where possible.
8. Preserve exact hash and feature checks inside the parser as defense in depth.
9. Replace unresolved placeholder strings with typed unavailable values and diagnostics.
10. Do not mark unresolved names as exact.
11. Do not assign full confidence to unresolved or heuristic pool entries.
12. Add a real Dart 3.11.1 snapshot fixture and an end-to-end adapter test.
13. Compare libraries, classes, functions, code ranges, and pool facts against an SDK-matched reference.
14. Prove deterministic output for the same input and adapter artifact.
15. Fix all workspace checks before merge.

The parser must also preserve stream alignment across every snapshot section. In particular, root parsing must consume the initial-field-table and shared-initial-field-table counts and references before continuing to the dispatch table, even if those tables are discarded by the current model.

The object-store layer should remain format-oriented and return references or domain-neutral values. Conversion into LibraryInfo, ClassInfo, and other ProgramModel types belongs in the model producer layer. A resolver that returns Result<()> while claiming to resolve object-store state should be replaced with a real state-producing API.

Serwalker should become the first external adapter artifact, not define the generic host contract.

8. Other parser backends

r2flutter can be useful as:

• an independent parser;
• a reference oracle;
• a source of exact object-pool information;
• a guarded adapter for identities it can prove exactly.

Its fingerprint mode must never provide authoritative metadata for an unknown hash.

Blutter can be useful as:

• an SDK-matched reference parser;
• an expensive CI oracle;
• an explicit fallback where installation and build costs are acceptable.

Neither backend should be selected from a substring in adapter_kind. Backend authority must come from the verified registry record and provenance fields.

9. Landing order

The work should land as stacked changes.

PR 1: Semantic contract

Start from main:

• add typed SnapshotIdentity;
• define protocol v1 request/result types;
• introduce ProgramModel v4;
• add capability and provenance states;
• add strict relational validation;
• migrate the current core consumer;
• migrate the checked-in reference adapter;
• remove synthetic semantic fallback records;
• stop inferring authority from adapter_kind.

PR 2: Adapter host and registry

Add:

• exact identity-to-family/profile/artifact resolution;
• runtime-loaded profiles;
• subprocess execution;
• SHA-256 verification;
• host/target validation;
• schema and semantic validation;
• resource containment;
• explicit machine-code fallback;
• deterministic error reporting.

Do not add automatic network installation yet.

PR 3: Serwalker adapter

After the first two contracts stabilize:

• rebase or transplant PR Flutterdec dart serwalker #74;
• package Serwalker as protocol v1;
• emit ProgramModel v4;
• remove direct core linkage;
• stop overloading Internal;
• add real fixtures and reference comparisons;
• preserve exact identity checks;
• fix parser alignment and model-provenance issues;
• complete all CI checks.

PR #74 can temporarily target the foundation branch as a stacked PR, but the merge order should remain contract, host/registry, then Serwalker. Building the foundation directly on the current Serwalker branch would mix generic architecture with parser-specific implementation and make the core contract depend on one parser.

PR 4: Grammar-family proof

Add a second exact snapshot identity.

For that identity:

• run the existing parser family unchanged;
• compare output against an SDK-matched reference;
• run semantic invariants;
• add a profile-only change if only layout data differs;
• split or update the parser family if serialization behavior differs.

Never infer family compatibility from Dart version proximity.

PR 5: CI and distribution

For every Dart stable or beta candidate:

1. acquire the exact SDK/source;
2. generate or obtain real snapshot fixtures;
3. extract the exact identity;
4. generate a candidate profile;
5. test existing parser families;
6. run semantic invariants;
7. compare overlapping facts with an SDK-matched reference;
8. run wrong-hash and wrong-feature tests;
9. run corruption, truncation, timeout, and oversized-output tests;
10. verify artifact digests and host variants;
11. publish a compatibility record only after all checks pass;
12. otherwise publish a compatibility-break artifact containing the SDK diff, failed fixtures, and failed invariants.

CI should automate SDK acquisition, identity extraction, profile generation, comparison, and publication. It should not attempt general-purpose C++-to-Rust parser translation; Dart serialization logic is procedural and conditional, not a stable declarative schema.

10. Acceptance gates

Issue #96 should not be considered complete until all of these behaviors are covered:

1. A known exact identity selects the documented parser family, profile, and artifact.
2. A wrong hash fails before adapter output reaches the core.
3. A wrong snapshot kind, architecture, or feature tuple fails.
4. A heuristically discovered hash cannot select an authoritative parser.
5. An unknown identity uses machine-code fallback without fabricated semantic metadata.
6. Corrupt or truncated adapter output fails relational validation.
7. Adapter crashes, timeouts, malformed JSON, and oversized output remain contained.
8. Artifact digest, host variant, target architecture, and registry mismatches fail before execution.
9. The same input and adapter produce deterministic canonical output.
10. Existing supported inputs preserve valid decompilation goldens.
11. A newly admitted hash/profile can be added without rebuilding the core binary.
12. Multiple exact registry records can reference one parser artifact.
13. At least two materially different snapshot families have real fixtures.
14. Failed compatibility candidates never enter the production registry.
15. Parser limitations and unavailable fields are visible to callers.
16. Adapter provenance is preserved through the complete pipeline.
17. Machine-code fallback remains separate from snapshot deserialization.
18. Security limits apply to every adapter invocation.

Final decision

The recommended architecture is:

Exact snapshot identity
    + verified compatibility registry
    + grammar-family parser
    + independently versioned data profile
    + external adapter protocol
    + ProgramModel v4
    + strict semantic validation
    + honest machine-code fallback

Reject:

• one complete flutterdec binary per Dart or Flutter version;
• one copied parser executable per snapshot hash;
• one parser implementation per Dart major/minor version;
• a monolithic in-core deserializer as the primary architecture;
• heuristic hash matching as authoritative compatibility;
• synthetic metadata for unknown snapshots;
• automatic remote installation before artifact trust is implemented;
• automatic general-purpose C++-to-Rust parser translation.

Recommended first implementation priority:

1. semantic contract and ProgramModel v4;
2. adapter host and exact registry;
3. Serwalker as the first external adapter;
4. second-family compatibility proof;
5. CI-driven profile and registry updates;
6. signed remote distribution only after the local trust model is complete.

[kalixtez]

Small things to consider:

• There might never be a chance for us to implement a JSON-RPC IPC model for adapters, given the task is not complex enough. Subprocess spawning works and I think will work fine for as long as the project lives.

• For now, we don't really need to use the snapshot kind to key a future profile rwgistry. We assume from the beginning that we are gonna be receiving a FullAOT binary. The other proposed components of the composite key, snapshot and target arch, are very valid and indeed should be implemented in my opinion.

• It is possible don't need to take into consideration alignment details when it comes to specifying a data profile. Dart doesn't insert padding or forces alignment constraints during serialization with the exception of the boundaries between snapshot section such as the clustered stream and the ROData image, where the output stream is aligned to image header-sized values.