[lxy-9602]

Purpose

Linked issue: #342

This change aligns the C++ Variant and MAP shared-shredding write paths with Java's rolling-file behavior.

Main changes:

• Add adaptive Variant shredding inference:

  • Introduce per-file and adaptive inference modes.
  • Infer a complete physical schema across all Variant columns with a shared width budget.
  • Preserve and refine selected schemas using evidence from previously completed files.
  • Support nested Variants, type widening, empty samples, and untyped Variant plans.
  • Separate pending inference from committed inference so evidence is published only after the corresponding file is successfully completed.

• Refactor the shredding write-plan lifecycle:

  • Add an explicit file-completion callback to ShreddingWritePlanFactory.
  • Invoke the callback only after the format writer and output stream complete successfully.
  • Reject configurations where multiple shredding implementations are active simultaneously.
  • Apply the same lifecycle to append-only and key-value writers.

• Improve MAP shared-shredding rolling behavior:

  • Let the write-plan factory own the adaptive column-count context for one rolling writer.
  • Start each new rolling writer from the configured maximum column count.
  • Use statistics from successfully completed files to select the column count for subsequent rolled files.
  • Support multiple MAP shared-shredding columns independently.
  • Store the normalized field-dictionary compression codec in file metadata and use that metadata when reading.

• Add deterministic row-count-based rolling:

  • Introduce the target-file-row-num option.
  • Roll normal data files when either the target file size or row count is reached.
  • Disable row-count splitting explicitly for compaction output and physical BLOB files.
  • Use the option in tests that need to verify actual rolling behavior.

• Strengthen MAP shared-shredding validation:

  • Require non-null STRING MAP keys.
  • Reject nested BLOB values.
  • Reject unsupported combinations such as Variant fields or postpone-bucket mode.
  • Validate field-dictionary compression, including rejecting empty or unsupported codecs.
  • Preserve MAP key nullability during field-ID assignment so invalid schemas are detected correctly.

Tests

• Extend unit and integration coverage for:
  • Adaptive and per-file Variant inference.
  • Nested Variant inference, type widening, width budgets, and empty samples.
  • MAP column placement and overflow policies.
  • Rolling adaptation for append-only and primary-key tables.
  • Multiple MAP columns, null/empty MAP values, spill, and compaction paths.
  • New CoreOptions and schema-validation behavior.

API and Format

API changes:

• Add the public option definition target-file-row-num.

  • Default: INT64_MAX, which disables row-count-based rolling.
  • Valid values must be at least 1 when explicitly configured.

• Add the public option definition variant.shredding.inferenceMode.

  • Supported values: per-file and adaptive.
  • Default: per-file.

• Add the public option definition variant.shredding.adaptive.maxInferBufferRow.

  • Controls the maximum number of prefix rows sampled after the first file in an adaptive inference session.
  • Default: 256.

• Add the public option definition variant.shredding.adaptive.retentionRatio.

  • Controls the minimum combined evidence ratio required to retain a previously selected Variant path.
  • Default: 0.05.
  • Valid range: [0, variant.shredding.minFieldCardinalityRatio].

• Add VariantShreddingInferenceMode with:

  • PER_FILE
  • ADAPTIVE

Format changes:

• MAP shared-shredding metadata now records
  paimon.map.shared-shredding.field-dict-compression.
• Deserialization obtains the codec from file metadata instead of an external fallback value.

Documentation

Generative AI tooling

Generated-by: Codex (GPT-5)

[lszskye]

+1