fix(thoughtspot): a malformed column or model name should not crash the whole conversion - #482
Conversation
…he whole conversion A columns[] entry with no name key, or one whose name is not a string (an int, null, or a bool), was read straight into identifiers.normalise() and raised a bare KeyError or TypeError instead of the converter's own "never a bare traceback" contract. A model name of the same wrongly typed shape hit the same TypeError. Add a shared _column_display_name() check ahead of the read in convert_field and convert_metric, and the same type check before convert()'s model name normalise() call. Both now log a WARNING and skip the malformed object (TS-FIELD-NO-NAME, TS-FIELD-NAME-INVALID, TS-METRIC-NO-NAME, TS-METRIC-NAME-INVALID, TS-MODEL-NAME-INVALID), matching the existing TS-KIND-* degrade and continue pattern rather than aborting the whole document. Refs apache#469: the issue also reports wrongly typed values in aggregation, data_type, formulas[].expr, join on, and db_column_name, plus the to-tml stash direction (custom_extensions, schema invalid documents). None of those are touched here.
kayemkim
left a comment
There was a problem hiding this comment.
Ran this merged onto current main (e2d94d3): the thoughtspot workflow passes on all five Pythons, 965 against 947 on main. Reproduced the issue through the CLI on the minimal fixture: a column with no name and one with name: 42 both give the bare KeyError/TypeError on main, and on this branch come back as TS-FIELD-NO-NAME/TS-FIELD-NAME-INVALID warnings with the rest of the model converted and the output passing validate.py. A model name: 7 falls back to model the same way.
One thing I noticed while trying it, not something this PR introduces: once a column can be skipped, a formula elsewhere that still references it by name comes out as TS-EXPR-PARAM, "references the ThoughtSpot runtime parameter total_order_amount". A reference to a name that never existed gets the same code on main, so it is the existing dangling-reference path, but now that a dropped column is a possible cause a reader could take it for a parameter. Might be worth a line when the rest of #469's table gets its follow-up.
For sequencing, #463 touches the same four files and is currently conflicting with main, so whichever of the two lands second will need a rebase. LGTM.
jbonofre
left a comment
There was a problem hiding this comment.
Thanks @AmirF194! The graceful degradation with _column_display_name() is very clean, and the column-level test coverage across various types is great.
I left two comments regarding model name handling: right now model_body.get("name") or "" causes falsy non-strings (0, False, None) to bypass the TS-MODEL-NAME-INVALID warning check while 42 and True are warned on. Once we check the raw value before falsy coercion so both truthy and falsy bad types behave consistently, this should be good to go.
| @@ -2048,6 +2087,21 @@ def convert(document_set: DocumentSet) -> OssieConversion: | |||
| model_body = document_set.model.body | |||
|
|
|||
| model_display_name = model_body.get("name") or "" | |||
There was a problem hiding this comment.
I think there is a bug here with model_body.get("name") or "". Because falsy values (0, False, None, [], {}) are coarced to "" before this check, they bypass if model_display_name and not instance(...):
name: TruelogsTS-MODEL-NAME-INVALID, butname: Falselogs nothing.name: 42logsTS-MODEL-NAME-INVALID, butname: 0logs nothing.name: nulllogs nothing for a model, but logsTS-FIELD-NAME-INVALIDfor a column.
In YAML/TML, name: 0 or name: false is not "no name at all" (missing key). It is an explicit non-string value of the wrong type and should be warned about consistently with truthy values and column names.
model_name_raw = model_body.get("name")
if "name" in model_body and not isinstance(model_name_raw, str):
# Same malformed-type hazard as a column `name` (see
# `_column_display_name`): the model has no field to skip, so it
# falls back the same way an empty name already does, just with a
# WARNING naming what was dropped.
log.add(
code="TS-MODEL-NAME-INVALID",
severity=Severity.WARNING,
message=(
f"model name {model_name_raw!r} is not a string; the "
f"semantic model is named 'model' instead"
),
object_ref=f"model:{model_name_raw!r}",
)
model_display_name = ""
else:
model_display_name = model_name_raw or ""
| """ | ||
|
|
||
| @pytest.mark.parametrize("bad_name", [42, True]) | ||
| def test_a_non_string_model_name_falls_back_and_is_reported(self, bad_name): |
There was a problem hiding this comment.
Once the type check is adjusted to inspect model_body["name"] before default coercion, 0, False, and None should be reported under test_a_non_string_model_name_falls_back_and_is_reported rather than failing back silently.
We should test:
- All malformed types (
[42, 0, True, False, None]) are reported withTS-MODEL-NAME-INVALID. - Only genuinely missing names (no
namekey) or empty strings (name: "") fall back silently to"model".
… truthy one
model_display_name = model_body.get("name") or "" coerced 0/False/None to
"" before the type check, so those three values fell back to "model"
silently while 42/True correctly logged TS-MODEL-NAME-INVALID. Read the
raw value first and check it for "name" in model_body, so a falsy
non-string is reported the same as a truthy one; a genuinely missing key
or an explicit empty string still falls back silently.
Parametrize the malformed-name test over 42, True, 0, False, None, and
split the old single-case silent-fallback test into one for a missing
key and one for an empty string.
|
Good catch, fixed: the type check now reads model_body.get("name") before the or "" coercion, so 0/False/None are reported with TS-MODEL-NAME-INVALID the same as 42/True. Split the old test into a missing-key case and an empty-string case, both still falling back silently, and parametrized the malformed case over all five bad types. Full suite still 969 passing. |
jbonofre
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround and updates @AmirF194!
The raw-value check nicely handles both truthy and falsy non-string model names, and the split tests for missing keys vs empty strings vs invalid types provide solid coverage. All 969 tests pass cleanly across the matrix.
LGTM!
|
The falsy-vs-truthy split you flagged was a real gap, glad you caught it before merge. Splitting missing-key from empty-string in the tests made the parametrize easier to follow too. |
A
columns[]entry with nonamekey, or one whosenameis present but not astring (an int,
null, or a bool), was read straight intoidentifiers.normalise()and raised a bare
KeyErrororTypeErrorinstead of the converter's own"never a bare traceback" contract (stash.py's phrase, quoted in the issue). A model
nameof the same wrongly typed shape hit the sameTypeErrorinconvert().convert_fieldandconvert_metricnow validate the column'snamethrough ashared
_column_display_name()helper before using it, andconvert()applies thesame type check to the model's own
name. All three degrade the way the existingTS-{KIND}-*gaps already do: log a WARNING and skip the object, rather thanaborting the whole document.
The issue reports this same shape (a value read without a type check) across eight
to-ossiecases and sixto-tmlcases. This fixes the two verified here: a columnwith no
name, andnameas int/null/bool at both column and model scope. Theremaining cases (
aggregation,data_type,formulas[].expr, joinon, anddb_column_namewrong types, plus the wholeto-tmlstash direction) are untouched.Added tests in
test_tml_to_ossie_fields.py,test_tml_to_ossie_metrics.py, andtest_tml_to_ossie.pycovering all four shapes (missing name; name as int, null,bool, float, or list; a metric name as bool; a model name as int or bool). Each
fails on unpatched
mainwith the traceback the issue reports and passes on thisbranch, confirmed both ways in the same container. Full suite: 965 passed (947
existing + 18 new), also run under
uv sync && uv run pyteston the CI matrix'sPython extremes (3.10, 3.14). Coverage confirms every changed line is exercised.
I have not run the other scenarios in the issue's table, listed above.
Refs #469