feat(thoughtspot): bidirectional ThoughtSpot TML <-> Ossie converter

[djwaldo]

Adds converters/thoughtspot/ — a bidirectional converter between ThoughtSpot's semantic model format (Model TML plus its Table and SQL View documents) and the Ossie semantic model.

Proposed in #285 (consolidating #269); THOUGHTSPOT was added to the Dialect enum in #351.

Scope

Ossie	ThoughtSpot
datasets	Model model_tables[] + a Table or SQL View document
fields	Model columns[] with column_type: ATTRIBUTE
metrics	Model columns[] with column_type: MEASURE, always via formulas[]
relationships	Model inline joins and Table joins_with[]
custom_extensions[THOUGHTSPOT]	everything with no Ossie home

One structural note that shapes the whole design: one Ossie semantic model corresponds to 1 + N TML documents, not one file. The converter reads and writes the set.

Expression handling — the design decision most likely to draw questions

ThoughtSpot's formula language is not SQL. It has its own syntax (concat ( [a] , [b] ), [TABLE::Column] references, { } grouping) which no SQL dialect parses.

A ThoughtSpot formula is carried verbatim under a THOUGHTSPOT dialect entry. A portable ANSI_SQL sibling is added only where the whole expression is a bare column reference — the one shape where portability is certain. No SQL dialect is ever re-rendered into another, in either direction.

That is deliberate, and the reasoning is in converters/thoughtspot/README.md:

• The specification's own stated default is to pass unknown values through, and dialects[] exists precisely so an untranslatable expression can still travel tagged with the dialect it is valid in.
• No converter in this repository re-renders an expression across dialects. The Databricks converter carries expressions verbatim under its own vendor dialect and reads that first.
• Round-trip fidelity is therefore exact rather than best-effort: the return leg reads the verbatim entry.

expressions/catalog.py maps all 146 constructs the Ossie expression language defines to a ThoughtSpot rendering (108 direct, 37 via sql_* passthrough, 1 unmappable) and is used for Ossie → TML. expressions/reverse.py records which ThoughtSpot-native functions compose into a portable Ossie expression; it is present for testing and future use and is not yet wired into the shipped TML → Ossie path — the README says so explicitly rather than leaving it to be inferred.

Reference documentation is generated from the code

converters/thoughtspot/docs/ is produced by tools/generate_reference_docs.py from CATALOG, the reverse inventory, the datatype map and the vendor-payload key classification. A test compares the committed documents byte-for-byte against fresh generator output, so they cannot drift from the implementation.

Testing

804 tests, offline, on Python 3.10–3.14.

• Both directions, unit-level.
• Round-trip both ways over two fixture sets, with preservation and translation asserted separately — the return leg reads the verbatim dialect entry, so a round trip alone proves preservation and says nothing about translation quality.
• The issue log is asserted as a first-class output: every declared loss must be reported before it happens, and nothing may be reported that did not occur.
• Property-based round trips (hypothesis, test-only) over deliberately awkward identifiers — YAML 1.1 boolean tokens, names colliding only after normalisation, non-ASCII, names containing ::. This found two crashes the example fixtures never produced.
• Fixtures mirror examples/tpcds_semantic_model.yaml so this converter is comparable to its siblings; both expected documents validate against core-spec/ossie-schema.json and validation/validate.py.

Dependencies

PyYAML>=6.0 and nothing else at runtime. hypothesis and jsonschema are test-only.

Provenance

All TML fixtures and all code are original, authored for this contribution. Nothing derives from ThoughtSpot's thoughtspot_tml library or any other existing converter.

A note on relationship orientation

Ossie's Relationship has no cardinality field — direction carries it, with from documented
as "the logical dataset on the many side" and to as "the one side". ThoughtSpot's joins carry
an explicit cardinality, including ONE_TO_MANY, which has no Ossie representation in the
direction TML declares it.

The converter therefore swaps the endpoints for a ONE_TO_MANY join, so the emitted
relationship is always many-side-to-one-side as the specification requires; MANY_TO_ONE and
ONE_TO_ONE pass through unchanged. Both TML spellings of the same relationship consequently
produce the same Ossie relationship, and the vendor extension records the swap so the return
trip reproduces the original TML join — same direction, same target, same cardinality.

Worth flagging for reviewers: validation/validate.py does not catch an inverted
relationship, because its key-coverage check skips a dataset that declares no keys. An earlier
revision of this branch emitted the un-swapped form and validated clean.

Note on history

81 commits, conventional-commit style. Happy to squash to a smaller set of logical commits if that is preferred for review.

[djwaldo]

Rebased onto current main, the branch is up to date and the tests are green (816, offline,
Python 3.10–3.14).

One thing I cannot do from my side: no CI has run on this PR since it was opened. I believe
it needs a maintainer to approve the workflow run, since this is a fork PR and I have not had
one approved on this repo before. converter-thoughtspot-ci.yml is in the branch and its
pull_request path filter covers converters/thoughtspot/**, so it should pick up once
approved.

[djwaldo]

i've Rebased onto current main and migrated to the single-model document format from #383, the model's fields now sit at the document root, and a document still carrying the semantic_model wrapper is rejected by name rather than unwrapped, matching #396.

823 tests pass on 3.10–3.14, offline. Both fixtures and the emitted output validate against the current core-spec/ossie-schema.json, and to-ossie → to-tml round-trips the TPC-DS fixture end to end.

Two things I can't do from this side:

• CI has never run. Every workflow on the branch sits at action_required, so it needs a
  committer to approve the run.
• No reviewer yet. Open since 7 September.

Could a committer approve CI and take a look?

[djwaldo]

Your two reviews prompted a much wider pass. Seventeen commits since, and the most useful
thing in them did not come from more code review — it came from running converted models
against a live ThoughtSpot cluster.

Round-tripped 30 real customer models (161 tables, 1,471 columns, 202 formulas, 146 joins),
export → Ossie → TML. All 30 pass validation/validate.py, and no model loses a table, column
or join without saying so.

Then imported one back into the cluster and compared the numbers. A three-table joined
model, converted and re-imported, answers identically to its source — totals, five regions, a
formula-derived attribute, three distinct counts, to the last decimal.

That exercise found four defects that five rounds of adversarial review and 900+ tests had all
missed, because every one of them is invisible to document-level validation:

• with was omitted from a referencing join. Mandatory on every model_tables[].joins[]
  entry. The document passes the Ossie schema, passes validation/validate.py, and ThoughtSpot
  refuses it: "Compulsory Field worksheet->model_tables(2nd)->joins(1st)->with is not
  populated." Nothing caught it because the hand-authored fixtures omitted with too — code
  and tests shared one misunderstanding and agreed with each other.
• A bare group_aggregate(...) lost its column aggregation. For sum(...) and the
  group_sum shorthands that property is genuinely inert, and the converter discarded it for
  all of them. A bare group_aggregate is the exception: like a raw column, ThoughtSpot applies
  it. A changed answer, silently.
• Formulas that no columns[] entry surfaces were dropped. These are hidden helpers other
  formulas reference — a date-parameter model is the usual case. 32 of 41 formulas in one real
  model, and the visible ones referencing them came back with dangling references.
• obj_id was discarded along with guid and fqn. It is not the same kind of thing: it
  is the handle that survives import and lets a re-import update the object it came from instead
  of duplicating it.

Also fixed since your review: a formula cross-reference rebinding to a different formula;
duplicate relationship names producing a document validation/validate.py rejects; a renamed
dataset dangling every reference to itself; a Table merged with a SQL View; display-name
collisions invented between names ThoughtSpot treats as distinct. 824 tests then, 936 now, every
fix pinned by a test that fails when it is reverted.

Two things worth saying plainly. Several of these defects were asserted by our own tests —
one sweep derived each row's expected arity from the row's own template, so nothing could ever
disagree with itself; another pinned its defect and wrote the misreading into a comment as fact.
And four of the defects fixed in later rounds were introduced by earlier rounds' fixes. The rate
is dropping and the suite is much harder to fool than it was, but I would not describe this as
finished, and I would rather say so than present seventeen commits as convergence.

CI needs approving again on the new head.

[jbonofre]

@djwaldo amazing work! Thanks for the complete pass! I don't have ThoughtSpot cluster and I'm not ThoughtSpot expert as you are 😄 I just checked the code.

Let me do another pass, but I propose to merge quickly (we can always improve the converter later).

Thanks again!

[jbonofre]

Could this resolve the physical column to its Ossie field identifier before emitting the ANSI_SQL sibling?

For example, if the Ossie field is total_revenue but its warehouse column is NET_AMOUNT, this emits orders.NET_AMOUNT. Ossie's portable expression language resolves dataset.field against logical fields, so consumers may not be able to evaluate this reference.

We could emit the logical field name, or omit the sibling when that mapping cannot be made safely.

[jbonofre]

This only checks whether targets already exists: it doesn't detect duplicate paths among the generated files and --issues.

If --issues is equal to a new output path, the conversion writes the document and then overwrites it with issue JSON.

I think (as a follow-up) we should reject colliding paths before writing. The same problem exists in to-ossie.

[djwaldo]

Thanks for this one — I checked it against current main and it's already covered. That's my fault for not saying so when the fix landed.

Your comment on the same thing a day earlier (cli.py:207) prompted fce4b92, which added a collision check across every path a run will write — the converted output, each generated to-tml file, and --issues — evaluated before anything is written. Both shapes you describe refuse:

$ ossie-thoughtspot to-ossie tests/fixtures/minimal/*.tml -o x.yaml --issues x.yaml
Error: refusing to write the same path twice in one run: .../x.yaml.
--issues must name a different file from the converted output.
exit=1

$ ossie-thoughtspot to-tml m.yaml -o t/ --issues t/customers.table.tml
Error: refusing to write the same path twice in one run: .../t/customers.table.tml.
--issues must name a different file from the converted output.
exit=1

Nothing is written in either case — the output directory is left empty rather than half-populated — and to-ossie carries the same guard, so the symmetry you asked about holds.

I suspect the line you were reading is the separate pre-existing-target check, which does only what you describe; the collision check is a different guard that runs first. If those read as one thing, I'm glad to add a comment saying so.

Your other follow-up was real, though: the ANSI_SQL sibling naming warehouse columns is #459, with a fix in #460. It turned out to be 211 of 612 references across 20 of 31 real models — the qualifier was the Ossie dataset name while the column half was the warehouse column, so the pair resolved in neither namespace. Thanks for catching it, and for the review and merge.

[jbonofre]

I left a couple of comments as follow-up, but this converter is a great work.

Thanks again for your contribution! Great one!