Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,18 @@
}
---

:::caution Experimental feature

Doris continuous load is experimental and supported starting with Doris 4.1.5.

:::

<!-- Knowledge type: Procedure + Configuration parameters -->
<!-- Applicable scenario: Mirror sync of a whole MySQL database to Doris -->

Auto Table Creation Sync is implemented through the native `FROM MYSQL (...) TO DATABASE (...)` DDL. The target is a Doris database, and Doris automatically creates the corresponding downstream tables based on the upstream table schemas. You can use `include_tables` to control whether to sync one table, several tables, or all tables. On the first sync, Doris automatically creates downstream primary key tables and keeps the primary keys consistent with the upstream. This is suitable for mirror replication scenarios where no SQL processing is required and the downstream table schema should follow the upstream automatically.

By integrating [Flink CDC](https://github.com/apache/flink-cdc) capabilities, Doris reads change logs from MySQL and continuously writes the full and incremental data of a group of tables into Doris through Stream Load. Auto Table Creation Sync supports assigning a target table name to an individual source table or excluding non-key columns. If you need SQL expressions, row filtering, or data transformation, refer to [MySQL CDC with SQL Mapping](./continuous-load-mysql-table.md).

Check notice on line 31 in docs/data-operate/import/import-way/streaming-job/continuous-load-mysql-database.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//github.com/apache/flink-cdc. Owner%3A @apache/doris-website-maintainers

### Applicable Scenarios

Expand Down Expand Up @@ -145,14 +151,14 @@
| `database` | Yes | - | MySQL database name. |
| `include_tables` | No | - | Names of the tables to sync, separated by commas. If not set, all tables in the database are synced. |
| `exclude_tables` | No | - | Names of the tables not to sync, separated by commas. This parameter takes effect only when `include_tables` is not set. |
| `table.<table_name>.target_table` | No | Source table name | Supported since version 4.1.0. Sets the Doris target table name for a source table. `<table_name>` is the source table name. |
| `table.<table_name>.exclude_columns` | No | - | Supported since version 4.1.0. Specifies source columns not to sync, separated by commas. The columns must exist and cannot include primary key columns. |
| `offset` | No | `latest` | Startup offset. `initial`: full and incremental sync; `earliest`: start from the earliest available Binlog offset; `latest`: sync only changes after the job starts. You can also specify an exact JSON offset, such as `{"file":"binlog.000001","pos":"154"}` or `{"gtids":"<gtid_set>"}`. Since version 4.1.0, `snapshot` is also supported for full sync only. |
| `snapshot_split_size` | No | `40960` (`8096` before 4.1.4) | Split size in rows. During full sync, a table is divided into multiple splits. Must be a positive integer. |
| `table.<table_name>.target_table` | No | Source table name | Sets the Doris target table name for a source table. `<table_name>` is the source table name. |
| `table.<table_name>.exclude_columns` | No | - | Specifies source columns not to sync, separated by commas. The columns must exist and cannot include primary key columns. |
| `offset` | No | `latest` | Startup offset. `initial`: full and incremental sync; `earliest`: start from the earliest available Binlog offset; `latest`: sync only changes after the job starts. You can also specify an exact JSON offset, such as `{"file":"binlog.000001","pos":"154"}` or `{"gtids":"<gtid_set>"}`. `snapshot` is also supported for full sync only. |
| `snapshot_split_size` | No | `40960` | Split size in rows. During full sync, a table is divided into multiple splits. Must be a positive integer. |
| `snapshot_parallelism` | No | `1` | Parallelism of the full-sync phase, that is, the maximum number of splits scheduled by a Task at one time. Must be a positive integer. |
| `server_id` | No | Automatically generated | Supported since version 4.1.0. Server ID of the MySQL CDC reader. Supports a single value, such as `5400`, or a closed range, such as `5400-5408`. The range width must be greater than or equal to `snapshot_parallelism`. |
| `ssl_mode` | No | `disable` | Supported since version 4.1.0. SSL mode. Valid values are `disable`, `require`, and `verify-ca`. |
| `ssl_rootcert` | Conditionally required | - | Supported since version 4.1.0. CA certificate file in the format `FILE:<file_name>`. Required when `ssl_mode` is `verify-ca`. Upload the file first using [CREATE FILE](../../../../sql-manual/sql-statements/security/CREATE-FILE.md). |
| `server_id` | No | Automatically generated | Server ID of the MySQL CDC reader. Supports a single value, such as `5400`, or a closed range, such as `5400-5408`. The range width must be greater than or equal to `snapshot_parallelism`. |
| `ssl_mode` | No | `disable` | SSL mode. Valid values are `disable`, `require`, and `verify-ca`. |
| `ssl_rootcert` | Conditionally required | - | CA certificate file in the format `FILE:<file_name>`. Required when `ssl_mode` is `verify-ca`. Upload the file first using [CREATE FILE](../../../../sql-manual/sql-statements/security/CREATE-FILE.md). |

If both `include_tables` and `exclude_tables` are set, `include_tables` takes precedence. The following `FROM MYSQL` parameter fragment syncs `orders` and `customers`, writes `orders` to `ods_orders`, excludes the non-key column `internal_note`, and enables CA verification:

Expand Down Expand Up @@ -242,4 +248,4 @@
- [MySQL CDC with SQL Mapping](./continuous-load-mysql-table.md)
- [CREATE STREAMING JOB](../../../../sql-manual/sql-statements/job/CREATE-STREAMING-JOB.md)
- [JDBC Catalog Overview](../../../../lakehouse/catalogs/jdbc-catalog-overview.md)
- [Flink CDC](https://github.com/apache/flink-cdc)

Check notice on line 251 in docs/data-operate/import/import-way/streaming-job/continuous-load-mysql-database.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//github.com/apache/flink-cdc. Owner%3A @apache/doris-website-maintainers
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,18 @@
}
---

:::caution Experimental feature

Doris continuous load is experimental and supported starting with Doris 4.1.5.

:::

<!-- Knowledge type: Operation steps / Configuration parameters -->
<!-- Applicable scenario: Real-time sync of a single MySQL table to Doris with column mapping or data transformation -->

SQL Mapping Sync is implemented through Job + [CDC Stream TVF](../../../../sql-manual/sql-functions/table-valued-functions/cdc-stream.md), targeting an **existing** Doris table (`INSERT INTO tbl SELECT * FROM cdc_stream(...)`). With the expressive power of Doris SQL, you can perform column mapping, filtering, and data transformation in the sync pipeline. This is suitable for real-time sync scenarios that require data processing.

> SQL Mapping Sync is supported since version 4.1.0.

By integrating the read capability of [Flink CDC](https://github.com/apache/flink-cdc), Doris reads the change log (Binlog) from MySQL and completes **full + incremental** sync from source table to target table. If you want Doris to automatically create downstream tables and sync a group of tables on a per-database basis, see [MySQL CDC with Auto Table Creation](./continuous-load-mysql-database.md).

Check notice on line 31 in docs/data-operate/import/import-way/streaming-job/continuous-load-mysql-table.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//github.com/apache/flink-cdc. Owner%3A @apache/doris-website-maintainers

### Applicable Scenarios

Expand All @@ -36,7 +40,7 @@

| Item | Description |
| ------ | ---- |
| Doris version | 4.1.0 and above |
| Doris version | 4.1.5 and above |
| Table type | The upstream source table must have a primary key; the Doris target table must use the Unique Key model |
| User privileges | `Load` privilege is required |
| MySQL configuration | Binlog must be enabled on the MySQL side. See the [Configuration Guide](./continuous-load-overview.md#supported-data-sources-and-sync-modes) |
Expand Down Expand Up @@ -95,7 +99,7 @@
| `database` | Yes | - | MySQL database name. |
| `table` | Yes | - | Name of the table to sync. Each SQL Mapping job supports one source table. |
| `offset` | Yes | - | Startup offset. `initial`: full and incremental sync; `snapshot`: full sync only; `earliest`: start from the earliest available Binlog offset; `latest`: sync only changes after the job starts. You can also specify an exact JSON offset, such as `{"file":"binlog.000001","pos":"154"}` or `{"gtids":"<gtid_set>"}`. |
| `snapshot_split_size` | No | `40960` (`8096` before 4.1.4) | Split size in rows. During full sync, the table is divided into multiple splits. Must be a positive integer. |
| `snapshot_split_size` | No | `40960` | Split size in rows. During full sync, the table is divided into multiple splits. Must be a positive integer. |
| `snapshot_parallelism` | No | `1` | Parallelism of the full-sync phase, that is, the maximum number of splits scheduled by a Task at one time. Must be a positive integer. |
| `skip_snapshot_backfill` | No | `false` | Whether to skip Binlog backfill during the snapshot. When set to `true`, at-least-once semantics are used. |
| `server_id` | No | Automatically generated | Server ID of the MySQL CDC reader. Supports a single value, such as `5400`, or a closed range, such as `5400-5408`. The range width must be greater than or equal to `snapshot_parallelism`. |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,18 @@
}
---

:::caution Experimental feature

Doris continuous load is experimental and supported starting with Doris 4.1.5.

:::

<!-- Knowledge type: Procedure + Configuration parameters -->
<!-- Applicable scenario: Mirror an OceanBase database in MySQL compatibility mode to Doris -->

OceanBase CDC with Auto Table Creation uses `FROM OCEANBASE (...) TO DATABASE (...)`. Doris reads full and incremental data through the OBBinlog service and automatically creates Doris Unique Key tables from the upstream schemas. Use `include_tables` to sync one table, multiple tables, or all tables in the database.

:::caution Experimental feature

OceanBase CDC continuous load is available as an experimental feature starting from Doris 4.1.4. It currently supports only OceanBase MySQL compatibility mode and Auto Table Creation Sync. Oracle compatibility mode and SQL Mapping Sync are not supported.

:::
OceanBase CDC continuous load currently supports only OceanBase MySQL compatibility mode and Auto Table Creation Sync. Oracle compatibility mode and SQL Mapping Sync are not supported.

### Applicable Scenarios

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
---

Check warning on line 1 in docs/data-operate/import/import-way/streaming-job/continuous-load-overview.md

View workflow job for this annotation

GitHub Actions / Build Check

seo-description-length

SEO description should be 80-160 characters; current length is 192. Owner%3A @apache/doris-website-maintainers
{
"title": "Continuous Load Overview",
"language": "en",
Expand All @@ -20,15 +20,17 @@
}
---

:::caution Experimental feature

Doris continuous load is experimental and supported starting with Doris 4.1.5.

:::

<!-- Knowledge type: Feature overview + architecture selection decision -->
<!-- Applicable scenarios: Real-time data ingestion selection / continuous load operations -->

Doris supports continuously loading data from multiple data sources into Doris tables through **Streaming Job**. After a Job is submitted, Doris keeps the load job running, reading data from the source in real time and writing it into the Doris table.

:::tip
This feature is supported starting from version 4.1.0.
:::

This document helps you answer the following questions:

- Which data sources and sync modes does continuous load support?
Expand All @@ -39,13 +41,13 @@

## Supported Data Sources and Sync Modes

Continuous load supports the following data sources and sync modes:
Continuous load supports the following data sources and sync modes. S3 notification and one-time ingestion are supported starting with Doris 4.1.6:

| Data Source | Supported Versions | SQL Mapping Sync | Auto Table Creation Sync | Configuration Guide |
| :---------- | :----------------- | :---------------------------------------------------------------- | :---------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------- |
| MySQL | 5.6, 5.7, 8.0.x | [MySQL CDC with SQL Mapping](./continuous-load-mysql-table.md) | [MySQL CDC with Auto Table Creation](./continuous-load-mysql-database.md) | [Amazon RDS MySQL](./prerequisites/amazon-rds-mysql.md) · [Amazon Aurora MySQL](./prerequisites/amazon-aurora-mysql.md) |
| PostgreSQL | 14, 15, 16, 17 | [PostgreSQL CDC with SQL Mapping](./continuous-load-postgresql-table.md) | [PostgreSQL CDC with Auto Table Creation](./continuous-load-postgresql-database.md) | [Amazon RDS PostgreSQL](./prerequisites/amazon-rds-postgresql.md) · [Amazon Aurora PostgreSQL](./prerequisites/amazon-aurora-postgresql.md) |
| OceanBase | MySQL compatibility mode | - | [OceanBase CDC with Auto Table Creation](./continuous-load-oceanbase-database.md) (since version 4.1.4) | - |
| OceanBase | MySQL compatibility mode | - | [OceanBase CDC with Auto Table Creation](./continuous-load-oceanbase-database.md) | - |
| S3 | - | [S3 Continuous Load](./continuous-load-s3.md) | - | - |

For how upstream column types map to Doris types, see Data Type Mapping for [MySQL](./data-type-mapping-mysql.md), [PostgreSQL](./data-type-mapping-postgresql.md), and [OceanBase](./data-type-mapping-oceanbase.md).
Expand Down Expand Up @@ -142,8 +144,8 @@
| LoadStatistic | Job statistics |
| ErrorMsg | Error message of the Job |
| JobRuntimeMsg | Runtime hints of the Job |
| LagBytes | Number of backlog bytes in the source log (MySQL / OceanBase binlog or PostgreSQL WAL). `-1` means the value is currently unavailable, for example for an S3 data source or during the full snapshot phase. **Since version 4.1.4**, this column replaces the former `Lag` column (in seconds) and is reported in bytes |
| LastSourceEventTimestamp | Timestamp (in Unix seconds) of the latest source event recorded in the committed offset. Empty when unavailable. **Added in version 4.1.4** |
| LagBytes | Number of backlog bytes in the source log (MySQL / OceanBase binlog or PostgreSQL WAL). `-1` means the value is currently unavailable, for example for an S3 data source or during the full snapshot phase. |
| LastSourceEventTimestamp | Timestamp (in Unix seconds) of the latest source event recorded in the committed offset. Empty when unavailable. |
| LastTaskSuccessTime | Time when the most recent Task completed successfully |

### View Task Status
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,18 @@
}
---

:::caution Experimental feature

Doris continuous load is experimental and supported starting with Doris 4.1.5.

:::

<!-- Knowledge type: Operating procedure -->
<!-- Applicable scenario: Continuously sync an entire PostgreSQL database to Doris -->

Auto Table Creation Sync is implemented through the native `FROM POSTGRES (...) TO DATABASE (...)` DDL. The target is a Doris database, and Doris automatically creates the corresponding downstream tables based on the upstream table schemas. You can use `include_tables` to control whether one table, multiple tables, or all tables are synced. On the first sync, Doris automatically creates the downstream primary-key tables and keeps their primary keys consistent with the upstream. This approach is suitable for mirror-replication scenarios where no SQL processing of the data is needed and the downstream table schemas should follow the upstream automatically.

By integrating [Flink CDC](https://github.com/apache/flink-cdc) capabilities, Doris reads change logs from PostgreSQL and continuously writes the full + incremental data of a group of tables into Doris through Stream Load. Auto Table Creation Sync supports assigning a target table name to an individual source table or excluding non-key columns. If you need SQL expressions, row filtering, or data transformation, see [PostgreSQL CDC with SQL Mapping](./continuous-load-postgresql-table.md).

Check notice on line 33 in docs/data-operate/import/import-way/streaming-job/continuous-load-postgresql-database.md

View workflow job for this annotation

GitHub Actions / Build Check

link-external-report-only

External link is report-only and was not fetched%3A https%3A//github.com/apache/flink-cdc. Owner%3A @apache/doris-website-maintainers

### Use Cases

Expand Down Expand Up @@ -96,15 +102,15 @@
| `schema` | Yes | - | Schema name. |
| `include_tables` | No | - | Names of the tables to sync, separated by commas. If not set, all tables in the Schema are synced. |
| `exclude_tables` | No | - | Names of the tables not to sync, separated by commas. This parameter takes effect only when `include_tables` is not set. |
| `table.<table_name>.target_table` | No | Source table name | Supported since version 4.1.0. Sets the Doris target table name for a source table. `<table_name>` is the source table name. |
| `table.<table_name>.exclude_columns` | No | - | Supported since version 4.1.0. Specifies source columns not to sync, separated by commas. The columns must exist and cannot include primary key columns. |
| `offset` | No | `latest` | Startup offset. `initial`: full and incremental sync; `latest`: sync only changes after the job starts. You can also specify an exact JSON offset, such as `{"lsn":"12345678"}`. Since version 4.1.0, `snapshot` is also supported for full sync only. PostgreSQL does not support `earliest`. |
| `snapshot_split_size` | No | `40960` (`8096` before 4.1.4) | Split size in rows. During full sync, a table is divided into multiple splits. Must be a positive integer. |
| `table.<table_name>.target_table` | No | Source table name | Sets the Doris target table name for a source table. `<table_name>` is the source table name. |
| `table.<table_name>.exclude_columns` | No | - | Specifies source columns not to sync, separated by commas. The columns must exist and cannot include primary key columns. |
| `offset` | No | `latest` | Startup offset. `initial`: full and incremental sync; `latest`: sync only changes after the job starts. You can also specify an exact JSON offset, such as `{"lsn":"12345678"}`. `snapshot` is also supported for full sync only. PostgreSQL does not support `earliest`. |
| `snapshot_split_size` | No | `40960` | Split size in rows. During full sync, a table is divided into multiple splits. Must be a positive integer. |
| `snapshot_parallelism` | No | `1` | Parallelism of the full-sync phase, that is, the maximum number of splits scheduled by a Task at one time. Must be a positive integer. |
| `slot_name` | No | `doris_cdc_<job_id>` | Supported since version 4.1.0. Logical replication slot name. The name can contain only lowercase letters, digits, and underscores, cannot start with a digit, and is limited to 63 characters. A custom slot must be created in advance and is not deleted by Doris. |
| `publication_name` | No | `doris_pub_<job_id>` | Supported since version 4.1.0. Publication name. The naming rules are the same as for `slot_name`. A custom publication must be created in advance, include all synced tables, and is not deleted by Doris. |
| `ssl_mode` | No | `disable` | Supported since version 4.1.0. SSL mode. Valid values are `disable`, `require`, and `verify-ca`. |
| `ssl_rootcert` | Conditionally required | - | Supported since version 4.1.0. CA certificate file in the format `FILE:<file_name>`. Required when `ssl_mode` is `verify-ca`. Upload the file first using [CREATE FILE](../../../../sql-manual/sql-statements/security/CREATE-FILE.md). |
| `slot_name` | No | `doris_cdc_<job_id>` | Logical replication slot name. The name can contain only lowercase letters, digits, and underscores, cannot start with a digit, and is limited to 63 characters. A custom slot must be created in advance and is not deleted by Doris. |
| `publication_name` | No | `doris_pub_<job_id>` | Publication name. The naming rules are the same as for `slot_name`. A custom publication must be created in advance, include all synced tables, and is not deleted by Doris. |
| `ssl_mode` | No | `disable` | SSL mode. Valid values are `disable`, `require`, and `verify-ca`. |
| `ssl_rootcert` | Conditionally required | - | CA certificate file in the format `FILE:<file_name>`. Required when `ssl_mode` is `verify-ca`. Upload the file first using [CREATE FILE](../../../../sql-manual/sql-statements/security/CREATE-FILE.md). |

If both `include_tables` and `exclude_tables` are set, `include_tables` takes precedence. The following `FROM POSTGRES` parameter fragment syncs `orders` and `customers`, writes `orders` to `ods_orders`, excludes the non-key column `internal_note`, and uses pre-created logical replication resources with CA verification:

Expand Down
Loading
Loading