fix(thoughtspot): portable expressions — unqualify a field's, quote what needs quoting

[djwaldo]

Fixes #459.

Two commits on the portable-expression surface. #462 is no longer in scope — see the comment below for why the metric half was withdrawn.

Commit 1 — a field's portable expression must not be dataset-qualified (#459)

A field's ANSI_SQL expression was <dataset>.<warehouse column> — Dim_Customer.Customer_Name. Both halves are real but come from different namespaces, so the pair resolves in neither: the qualifier is the Ossie dataset name (ThoughtSpot's Table-object name, which need not be the warehouse table — 44 of 164 datasets differ), while the column half is the warehouse column rather than the field identifier.

211 of 612 references named nothing the document declares, across 20 of 31 models.

Every sibling converter writes a field's expression as the bare warehouse column — databricks l_linenumber for line_number, nvidia name for customer_name, likewise gooddata, omni, orionbelt.

Commit 2 — quote an identifier that is not a regular one

SUM(cargo.Custom Clearance Time (min)) and SUM(HV: STORES.LATITUDE) are rejected by sqlglot, the parser Apache's validator uses. Per core-spec/expression_language.md identifiers follow ANSI SQL naming and the Ossie quote character is the double quote, so a non-regular identifier is quoted, embedded quotes doubled — and only when needed, since regular identifiers compare case-insensitively while quoted ones compare verbatim.

With the validator's SQL checks actually running:

	passed	failed	[SQL] findings
before	29	2	7
after	31	0	0

sqlglot joins the dev group. validation/validate.py prints sqlglot not installed, skipping SQL validation and then Validation PASSED — a skipped check reported as a pass, which is why those 7 findings sat unseen.

Test plan

• 950 tests pass, nothing skipped
• 31-model corpus: unresolvable field references 211 → 0, no structure lost
• Real validator (sqlglot present): 31/31, 0 SQL findings
• Returned TML byte-identical for all 195 documents at commit 1 — the reverse leg reads the THOUGHTSPOT dialect
• Mutation-checked: qualifying fields again fails 6 tests; removing quoting fails 4

🤖 Generated with Claude Code

[djwaldo]

@jbonofre both of your follow-up comments on #364 are addressed here, and the first one
turned out to be bigger than either of us thought.

Your ANSI_SQL comment was right. A field's portable expression named the warehouse
column, qualified by the Ossie dataset name -- two different namespaces, so it resolved in
neither. Measured across 31 real ThoughtSpot Models: 211 of 612 references named nothing
the document declares, in 20 of 31 models. What settled the fix was checking what the
other converters do: databricks writes l_linenumber for a field named line_number,
nvidia writes name for customer_name. The physical column was never the problem, the
qualifier was.

Verifying that turned up a second one. Metrics carried a THOUGHTSPOT dialect only --
476 of 476 -- and every vendor converter skips a metric with no portable dialect, so a
converted model lost all of its business logic on the way to any vendor. The machinery was
already there and unit-tested; it never fired because the resolver indexed ATTRIBUTE
columns and a metric aggregates a MEASURE column.

	before	after
references naming no declared field	211 / 612	0
metrics with a portable dialect	0 / 476	348 / 476

The remaining 128 metrics are formula-shaped and need real expression translation --
expressions/reverse.py is the inventory for that, and it stays a follow-up rather than
something I'd rush into this.

Your other comment, on cli.py colliding output paths, was already fixed the day before
you wrote it -- I replied inline with the repro rather than changing anything.

One ask: the workflow run needs approving. No CI has run on this PR. 951 tests pass
locally and the 31-model corpus is clean, but none of that is visible to you without it.

---

One thing worth knowing beyond this PR, because it is not converter-specific.
validation/validate.py prints

[SQL] Warning: sqlglot not installed, skipping SQL validation
Validation PASSED: model.ossie.yaml

A skipped check reported as a pass. Every "validator passed" figure I measured during this
work was a validator with its SQL half switched off -- and with sqlglot installed, this
converter's output was failing 2 of 31 documents before these commits, which I had not
seen. I have added sqlglot to this converter's dev group so its own tests parse what they
emit. Happy to raise it as a separate issue if you think the validator should exit
non-zero, or at least not say PASSED, when it has skipped a whole class of check.

[djwaldo]

Withdrawing the metric half of this PR

An independent adversarial review found that the metric change was wrong, and I have removed it. This PR is now the field fix (#459) plus identifier quoting. #462 stays open; I no longer think the fix I proposed there is the right one.

What was wrong. The metric commit emitted SUM(<dataset>.<warehouse column>) for a metric over a MEASURE column. A MEASURE column never becomes an Ossie field, so 348 of 348 emitted references named a column nothing in the document declares — which is the same namespace defect this PR's first commit fixes for fields. I reproduced it independently: 340 of 340 by my own count.

Why it matters more than it sounds. The reviewer ran every Ossie-to-vendor converter in the tree against the output. They split three ways:

How orders.amount is read	Converters
As the field name	Honeydew, Power BI, OrionBelt, Sigma, Snowflake
As the warehouse column (intended)	Databricks, dbt, NVIDIA
Field name first, then column	Omni

On a name collision the first group sums the wrong column. Power BI emits BLANK() for all 476 measures; OrionBelt marks 348 as not queryable. Before the change these metrics had no portable expression at all — a loud, known loss. I replaced it with a quiet wrong answer, which is worse.

I had justified the dataset qualifier by pointing at nvidia's SUM(orders.subtotal). That was reasoning from one example where I should have run the consumers.

What I missed, and why. My own round-trip harness measures unresolvable references by iterating a dataset's fields only — it never looks at metrics. It reported 0/0 while the true figure was 348/348. Four separate instruments I built during this work turned out to be measuring nothing; this was the fourth.

Where #462 goes now. The right fix is a design question rather than a patch: either MEASURE columns are declared as Ossie fields and metrics reference them by field name, or metrics keep no portable expression. That is the same namespace question the spec leaves open — spec.md:104 says metrics are "aggregate expressions on fields", expression_language.md:98 gives the shape dataset.field — and it affects every converter, not just this one. I will raise it on dev@ with the consumer evidence rather than decide it here.

The review also found several pre-existing defects in the merged converter, which I am filing separately.

[jbonofre]

_REGULAR_IDENTIFIER doesn't check for SQL reserved keywords. Per ANSI SQL (and core-spec/expression_language.md), regular identifiers cannot be reserved words.

When a bare field column happens to be a SQL keyword like on (which is present in the TPC-DS fixture below), _sql_identifier leaves it unquoted as on. A query like SELECT on then fails SQL parsing.

Could we check name.upper() against a set of SQL reserved keywords here?

   _SQL_RESERVED = frozenset({
       "ALL", "AND", "ANY", "AS", "ASC", "BETWEEN", "BY", "CASE", "CHECK",
       "CREATE", "CROSS", "DEFAULT", "DESC", "DISTINCT", "DROP", "ELSE",
       "END", "EXCEPT", "EXISTS", "FALSE", "FOR", "FROM", "FULL", "GROUP",
       "HAVING", "IN", "INNER", "INSERT", "INTERSECT", "INTO", "IS", "JOIN",
       "LEFT", "LIKE", "LIMIT", "NOT", "NULL", "OFFSET", "ON", "OR", "ORDER",
       "OUTER", "PRIMARY", "RIGHT", "SELECT", "SOME", "TABLE", "THEN", "TRUE",
       "UNION", "UNIQUE", "USER", "USING", "WHEN", "WHERE", "WITH",
   })

And in _sql_identifier:

   if _REGULAR_IDENTIFIER.match(name) and name.upper() not in _SQL_RESERVED:
       return name
   return '"' + name.replace('"', '""') + '"'

[jbonofre]

This fixture currently fails validation/validate.py when sqlglot is installed:

python validation/validate.py converters/thoughtspot/tests/fixtures/tpcds/expected.ossie.yaml

Output:

Validation FAILED with 1 error(s):
     [SQL] Field 'store.on' in model 'tpcds_retail_model' (ANSI_SQL): Invalid expression / Unexpected token. Line 1, Col: 9.

Once _sql_identifier quotes reserved keywords, this should be expression: '"on"'.

[jbonofre]

Adding sqlglot to the dev group is a great improvement.

As a follow-up in tests/test_fixtures.py, test_expected_output_validates_against_the_upstream_schema currently only validates against ossie-schema.json and doesn't run SQL validation. If we also invoked validate_sql (or validation/validate.py) there when sqlglot is available, pytest would catch SQL parse errors in fixtures (like the store.on issue) directly during test runs.

[jbonofre]

Could we add a test case here for a field whose warehouse column is a SQL reserved keyword (e.g. orders.on or orders.where) asserting that:

1. It is quoted as "on".
2. sqlglot.parse_on(f"SELECT {portable}") parses successfully.

[jbonofre]

Great catch and handling here. Declining to compose sum(...) around cross-referencing formulas and warning via TS-METRIC-AGGREGATION-DEFAULT-UNRESOLVED avoids silent double-aggregation bugs (sum([formula_a] / [formula_b])).