[Discussion][Connector-V2] Define safe file splitting, recoverable reads, and media-aware file ingestion

[DanielLeens]

Purpose

This is a design discussion, not an implementation proposal. We would like feedback on a cohesive File Connector direction that makes parallel reads correct for structured files and creates a production-safe ingestion contract for images, audio, and video. Alternative designs and a formal STIP are very welcome.

Current observations

The File Connector already has useful foundations:

• FileSourceSplit carries a byte range and has fileIdentity and endContentAnchor fields: https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/FileSourceSplit.java
• enable_file_split currently plans text-like ranges by finding the next raw row delimiter, while Parquet uses a separate strategy: https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/AccordingToSplitSizeSplitStrategy.java
• JsonReadStrategy deserializes one physical line at a time: https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/JsonReadStrategy.java
• BinaryReadStrategy emits the stable three-field binary contract [data, relativePath, partIndex]: https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/BinaryReadStrategy.java
• The bounded file reader currently consumes a whole split in one pollNext call and snapshots queued splits, rather than a record-level cursor: https://github.com/apache/seatunnel/blob/dev/seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/BaseFileSourceReader.java

This leaves several design questions. A raw newline is not necessarily a CSV record boundary when quoted fields may contain newlines. JSON Lines and JSON arrays have different split semantics. Large media assets should not be arbitrarily byte-split or assembled as full byte arrays in the JVM heap.

Design goals

1. Preserve existing jobs and old checkpoints. enable_file_split=false remains the default, and the existing binary three-field schema must not change.
2. Never silently produce incomplete, duplicated, or malformed records because a split boundary is unsafe.
3. Support bounded reads with progress checkpoints at complete-record boundaries.
4. Keep connector responsibilities clear: file/object discovery and file reading belong to the File Connector; media decoding belongs to an optional media provider.
5. Avoid moving large image/audio/video payloads through ordinary SeaTunnel rows unless the input is intrinsically a binary message/BLOB stream.

Proposed direction for discussion

1. Format-aware split planning

Introduce a common record-boundary planning SPI rather than treating every supported format as line-delimited.

Format	Proposed split policy
Text	Split on the configured complete row delimiter.
CSV	Use a quote/escape-aware scanner and split only after a complete RFC-style CSV record. Header resolution must happen once before readers run.
JSONL	Split on complete physical lines only.
JSON array / one JSON document	Do not permit byte-range splitting. Use a single-file streaming parser or reject incompatible configuration.
Parquet	Retain RowGroup splitting.
Compressed/archive/binary/PDF/Excel/XML/media	Do not byte-split; distribute whole files.

For CSV, a quote-aware scanner may need to stream the file once to create boundaries or maintain a reusable boundary index keyed by file identity. We should prefer correctness over an unsafe fast path.

2. Recoverable FileReadCursor

For split-enabled bounded reads, introduce a cursor state such as:

file identity + planned range + last committed record end offset
+ resolved header/schema + end-content anchor

A reader should emit a bounded amount of data per poll and checkpoint only after a complete record. On restore, it reopens from the last committed record boundary and validates that the file/object identity and content anchor did not change. A changed object must fail or be re-enumerated by an explicit policy; it must never combine bytes from different versions.

This should be introduced as a new opt-in reader path for split-enabled bounded Text/CSV/JSONL/Parquet reads, keeping the existing full-split and continuous-tailing behavior untouched initially.

3. Media-aware File Source path

For LocalFile/HdfsFile/S3File/OssFile, consider a new media file format/reader (exact configuration is intentionally open for design). It would process one complete file per split and emit one ordinary MediaAsset row rather than binary chunks:

source_uri, relative_path, file_id/version, byte_size, modified_time,
content_hash, detected_mime_type, media_kind, format,
width, height, duration_ms, frame_rate, sample_rate, channels, codec,
parse_status, parse_error_code, provider_properties

The File Connector would provide stable file/object identity and streaming input. An optional media provider would detect and inspect the content. The first provider could be image metadata only; native audio/video decoding would be a separately installed provider with strict resource limits and explicit failure when unavailable.

Media files remain whole-file units. Their parallelism comes from independent files, not arbitrary byte ranges inside a video or audio asset.

4. Keep generic binary compatibility

The existing binary source contract remains exactly [data, relativePath, partIndex] for binary-to-binary synchronization. A later MediaPayload Transform can adapt binary data from Kafka, HTTP, or database BLOBs: validate chunk order, spool under resource limits, compute identity/hash, call the same media provider, and emit the same MediaAsset row. It should not require changing Binary Source or Binary Sink schemas.

Explicit non-goals

• This discussion does not propose a Ray-like execution engine or a new distributed runtime.
• It does not make a Tensor/media logical type a prerequisite for v1. Metadata should initially use existing scalar and map types.
• It does not merge object-storage CDC, event notifications, delete/changelog semantics, or on-demand fetch into this work.
• It does not claim a generic byte-range split is safe for all file formats.

Related work

• [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read (byte-range/row-delimiter + Parquet RowGroup) #10326 added large-file parallel splitting foundations and is the relevant predecessor.
• [Feature][Transform] add PDF Parser Operator and Multimodal Data Labeling Operator #10608 discusses a PDF parser and multimodal labeling operator; this issue proposes the lower-level file/media ingestion contract those features could build upon.
• [Feature][Connector-V2] Object storage CDC (OSS/S3): event-notification driven discovery, changelog semantics, and on-demand content fetch for real-time AI/multimodal pipelines #11758 discusses object-storage CDC and lazy content fetch. Its event/changelog semantics should stay separate from this bounded-file correctness work.

Questions for maintainers and contributors

1. Is a shared format-aware split-planning SPI the right boundary, or should each read strategy own its complete planning logic?
2. Should FileReadCursor be represented in FileSourceSplit or as separate reader state to keep split identity immutable?
3. What is the preferred checkpoint and file-change policy for object stores without a reliable ETag/version?
4. Should media inspection be a File Connector format, a reusable Source SPI/provider, or primarily a Transform with a specialized File adapter?
5. Which minimal image-only output schema would be useful enough for a first PR, without prematurely creating a public media type system?
6. Would maintainers prefer an STIP before new public configuration and provider/SPIs are introduced?

A contributor who would like to drive a better design or submit an STIP is very welcome. The goal here is to agree on a small, backward-compatible first slice before any broad implementation starts.