Migration: document object storage ClickPipe settings

integrations/clickpipes/object-storage/amazon-s3/overview.mdx

@@ -47,7 +47,7 @@ When continuous ingestion is enabled, ClickPipes continuously ingests data from
 
 By default, the S3 ClickPipe assumes files are added to a bucket in lexicographical order, and relies on this implicit order to ingest files sequentially. This means that any new file **must** be lexically greater than the last ingested file. For example, files named `file1`, `file2`, and `file3` will be ingested sequentially, but if a new `file 0` is added to the bucket, it will be **ignored** because the file name isn't lexically greater than the last ingested file.
 
-In this mode, the S3 ClickPipe does an initial load of **all files** in the specified path, and then polls for new files at a configurable interval (by default, 30 seconds). It is **not possible** to start ingestion from a specific file or point in time — ClickPipes will always load all files in the specified path.
+In this mode, the S3 ClickPipe does an initial load of **all files** in the specified path, and then polls for new files at a configurable interval (by default, 30 seconds). To begin continuous ingestion after a specific file, configure **Start after**. ClickPipes will skip files that are lexicographically less than or equal to the configured value and continue ingesting newer files from that point.
 
 #### Any order {#continuous-ingestion-any-order}
 
@@ -61,13 +61,13 @@ It's possible to configure an S3 ClickPipe to ingest files that don't have an im
 Unordered mode is **only** supported for Amazon S3 and is **not** supported for public buckets or S3-compatible services. It requires setting up an [Amazon SQS](https://aws.amazon.com/sqs/) queue connected to the bucket, optionally using [Amazon EventBridge](https://aws.amazon.com/eventbridge/) as an event router.
 </Note>
 
-In this mode, the S3 ClickPipe does an initial load of **all files** in the selected path, and then listens for `ObjectCreated:*` events in the queue that match the specified path. Any message for a previously seen file, file not matching the path, or event of a different type will be **ignored**.
+In this mode, the S3 ClickPipe does an initial load of **all files** in the selected path, and then listens for `ObjectCreated:*` events in the queue that match the specified path. Any message for a previously seen file, file not matching the path, or event of a different type will be **ignored**. To ingest only files delivered through the queue, enable **Skip initial load**. When enabled, ClickPipes skips the initial scan of existing files in the selected path and processes only files delivered through the SQS queue.
 
 <Note>
 Setting a prefix/postfix for events is optional. If you do, make sure it matches the path set for the clickpipe. S3 doesn't allow multiple overlapping notification rules for the same event types.
 </Note>
 
-Files are ingested once the threshold configured in `max insert bytes` or `max file count` is reached, or after a configurable interval (by default, 30 seconds). It is **not possible** to start ingestion from a specific file or point in time — ClickPipes will always load all files in the selected path. If a DLQ is configured, failed messages will be reenqueued and reprocessed up to the number of times configured in the DLQ `maxReceiveCount` parameter.
+Files are ingested once the threshold configured in `max insert bytes` or `max file count` is reached, or after a configurable interval (by default, 30 seconds). If a DLQ is configured, failed messages will be reenqueued and reprocessed up to the number of times configured in the DLQ `maxReceiveCount` parameter.
 
 <Tip>
 We strongly recommend configuring a **Dead-Letter-Queue (DLQ)** for the SQS queue, so it's easier to debug and retry failed messages.
@@ -190,6 +190,8 @@ ClickPipes provides sensible defaults that cover the requirements of most use ca
 | `Min insert block size bytes`      | 1GB           | [Minimum size of bytes in the block](/reference/settings/session-settings#min_insert_block_size_bytes) which can be inserted into a table. |
 | `Max download threads`             | 4             | [Maximum number of concurrent download threads](/reference/settings/session-settings#max_download_threads). |
 | `Object storage polling interval`  | 30s           | Configures the maximum wait period before inserting data into the ClickHouse cluster. |
+| `Start after`                      | None          | For ordered continuous ingestion, skips files that are lexicographically less than or equal to the configured filename or path. |
+| `Skip initial load`                | false         | For unordered continuous ingestion, skips the initial scan of existing files and processes only object notifications from the configured SQS queue. |
 | `Parallel distributed insert select` | 2           | [Parallel distributed insert select setting](/reference/settings/session-settings#parallel_distributed_insert_select). |
 | `Parallel view processing`         | false         | Whether to enable pushing to attached views [concurrently instead of sequentially](/reference/settings/session-settings#parallel_view_processing). |
 | `Use cluster function`             | true          | Whether to process files in parallel across multiple nodes. |

integrations/clickpipes/object-storage/amazon-s3/unordered-mode.mdx

@@ -16,7 +16,9 @@ Unordered mode is **only** supported for Amazon S3 and is **not** supported for
 
 ## How it works {#how-it-works}
 
-In this mode, the S3 ClickPipe does an initial load of **all files** in the selected path, and then listens for `ObjectCreated:*` events in the queue that match the specified path. Any message for a previously seen file, a file not matching the path, or an event of a different type will be **ignored**. Files are ingested once the threshold configured in `max insert bytes` or `max file count` is reached, or after a configurable interval (by default, 30 seconds). It is **not possible** to start ingestion from a specific file or point in time — ClickPipes will always load all files in the selected path.
+In this mode, the S3 ClickPipe does an initial load of **all files** in the selected path, and then listens for `ObjectCreated:*` events in the queue that match the specified path. Any message for a previously seen file, a file not matching the path, or an event of a different type will be **ignored**. Files are ingested once the threshold configured in `max insert bytes` or `max file count` is reached, or after a configurable interval (by default, 30 seconds).
+
+To ingest only files delivered through the queue, enable **Skip initial load**. When enabled, ClickPipes skips the initial scan of existing files in the selected path and processes only files delivered through the SQS queue.
 
 Various types of failures can occur when ingesting data, which can result in partial inserts or duplicate data. Object Storage ClickPipes are resilient to insert failures and provide exactly-once semantics using temporary staging tables. Data is first inserted into a staging table; if something goes wrong, the staging table is truncated and the insert is retried from a clean state. Only once an insert completes successfully are the partitions moved to the target table.
 
@@ -195,10 +197,10 @@ S3 doesn't allow multiple overlapping notification rules for the same event type
 
 **1.** In the ClickHouse Cloud console, navigate to **Data Sources > Create ClickPipe** and select **Amazon S3**. Enter the details to connect to your S3 bucket. Under **Authentication method**, choose **IAM role** and provide the ARN of the role you created in the previous step.
 
-**2.** Under **Incoming data**, toggle on **Continuous ingestion**. Select **Any order** as the ingestion mode and provide the **SQS queue URL** for the queue connected to your bucket.
+**2.** Under **Incoming data**, toggle on **Continuous ingestion**. Select **Any order** as the ingestion mode and provide the **SQS queue URL** for the queue connected to your bucket. To process only files from the queue and skip the initial scan of existing files in the selected path, enable **Skip initial load**.
 
 **3.** Under **Parse information**, define a **Sorting key** for the target table. Make any necessary adjustments to the mapped schema, then configure a role for the ClickPipes database user.
 
-**4.** Review the configuration and click **Create ClickPipe**. ClickPipes will perform an initial scan of your bucket to load all existing files that match the specified path, and will then begin processing files as new `ObjectCreated:*` events arrive in the queue.
+**4.** Review the configuration and click **Create ClickPipe**. Unless **Skip initial load** is enabled, ClickPipes will perform an initial scan of your bucket to load all existing files that match the specified path, and will then begin processing files as matching `ObjectCreated:*` events are delivered from the queue.
 </Step>
 </Steps>

integrations/clickpipes/object-storage/azure-blob-storage/overview.mdx

@@ -34,7 +34,7 @@ When continuous ingestion is enabled, ClickPipes continuously ingests data from
 
 The ABS ClickPipe assumes files are added to a container in lexicographical order, and relies on this implicit order to ingest files sequentially. This means that any new file **must** be lexically greater than the last ingested file. For example, files named `file1`, `file2`, and `file3` will be ingested sequentially, but if a new `file 0` is added to the container, it will be **ignored** because the file name isn't lexically greater than the last ingested file.
 
-In this mode, the ABS ClickPipe does an initial load of **all files** in the specified path, and then polls for new files at a configurable interval (by default, 30 seconds). It is **not possible** to start ingestion from a specific file or point in time — ClickPipes will always load all files in the specified path.
+In this mode, the ABS ClickPipe does an initial load of **all files** in the specified path, and then polls for new files at a configurable interval (by default, 30 seconds). To begin continuous ingestion after a specific file, configure **Start after**. ClickPipes will skip files that are lexicographically less than or equal to the configured value and continue ingesting newer files from that point.
 
 ### File pattern matching {#file-pattern-matching}
 
@@ -140,6 +140,7 @@ ClickPipes provides sensible defaults that cover the requirements of most use ca
 | `Min insert block size bytes`      | 1GB           | [Minimum size of bytes in the block](/reference/settings/session-settings#min_insert_block_size_bytes) which can be inserted into a table. |
 | `Max download threads`             | 4             | [Maximum number of concurrent download threads](/reference/settings/session-settings#max_download_threads). |
 | `Object storage polling interval`  | 30s           | Configures the maximum wait period before inserting data into the ClickHouse cluster. |
+| `Start after`                      | None          | For ordered continuous ingestion, skips files that are lexicographically less than or equal to the configured filename or path. |
 | `Parallel distributed insert select` | 2           | [Parallel distributed insert select setting](/reference/settings/session-settings#parallel_distributed_insert_select). |
 | `Parallel view processing`         | false         | Whether to enable pushing to attached views [concurrently instead of sequentially](/reference/settings/session-settings#parallel_view_processing). |
 | `Use cluster function`             | true          | Whether to process files in parallel across multiple nodes. |

integrations/clickpipes/object-storage/google-cloud-storage/overview.mdx

@@ -34,7 +34,7 @@ When continuous ingestion is enabled, ClickPipes continuously ingests data from
 
 The GCS ClickPipe assumes files are added to a bucket in lexicographical order, and relies on this implicit order to ingest files sequentially. This means that any new file **must** be lexically greater than the last ingested file. For example, files named `file1`, `file2`, and `file3` will be ingested sequentially, but if a new `file 0` is added to the bucket, it will be **ignored** because the file name isn't lexically greater than the last ingested file.
 
-In this mode, the GCS ClickPipe does an initial load of **all files** in the specified path, and then polls for new files at a configurable interval (by default, 30 seconds). It is **not possible** to start ingestion from a specific file or point in time — ClickPipes will always load all files in the specified path.
+In this mode, the GCS ClickPipe does an initial load of **all files** in the specified path, and then polls for new files at a configurable interval (by default, 30 seconds). To begin continuous ingestion after a specific file, configure **Start after**. ClickPipes will skip files that are lexicographically less than or equal to the configured value and continue ingesting newer files from that point.
 
 #### Any order {#continuous-ingestion-any-order}
 
@@ -48,7 +48,7 @@ It's possible to configure a GCS ClickPipe to ingest files that don't have an im
 Unordered mode is **not** supported for public buckets. It requires **Service Account** authentication and a [Google Cloud Pub/Sub](https://cloud.google.com/pubsub) subscription connected to the bucket.
 </Note>
 
-In this mode, the GCS ClickPipe does an initial load of **all files** in the selected path, and then listens for `OBJECT_FINALIZE` notifications via the Pub/Sub subscription that match the specified path. Any message for a previously seen file, file not matching the path, or event of a different type will be **ignored**. It is **not possible** to start ingestion from a specific file or point in time — ClickPipes will always load all files in the selected path.
+In this mode, the GCS ClickPipe does an initial load of **all files** in the selected path, and then listens for `OBJECT_FINALIZE` notifications via the Pub/Sub subscription that match the specified path. Any message for a previously seen file, file not matching the path, or event of a different type will be **ignored**. To ingest only files delivered through the subscription, enable **Skip initial load**. When enabled, ClickPipes skips the initial scan of existing files in the selected path and processes only files delivered through the Pub/Sub subscription.
 
 ### File pattern matching {#file-pattern-matching}
 
@@ -149,6 +149,8 @@ ClickPipes provides sensible defaults that cover the requirements of most use ca
 | `Min insert block size bytes`      | 1GB           | [Minimum size of bytes in the block](/reference/settings/session-settings#min_insert_block_size_bytes) which can be inserted into a table. |
 | `Max download threads`             | 4             | [Maximum number of concurrent download threads](/reference/settings/session-settings#max_download_threads). |
 | `Object storage polling interval`  | 30s           | Configures the maximum wait period before inserting data into the ClickHouse cluster. |
+| `Start after`                      | None          | For ordered continuous ingestion, skips files that are lexicographically less than or equal to the configured filename or path. |
+| `Skip initial load`                | false         | For unordered continuous ingestion, skips the initial scan of existing files and processes only object notifications from the configured Pub/Sub subscription. |
 | `Parallel distributed insert select` | 2           | [Parallel distributed insert select setting](/reference/settings/session-settings#parallel_distributed_insert_select). |
 | `Parallel view processing`         | false         | Whether to enable pushing to attached views [concurrently instead of sequentially](/reference/settings/session-settings#parallel_view_processing). |
 | `Use cluster function`             | true          | Whether to process files in parallel across multiple nodes. |

integrations/clickpipes/object-storage/google-cloud-storage/unordered-mode.mdx

@@ -14,7 +14,9 @@ Unordered mode is **not** supported for public buckets. It requires **Service Ac
 
 ## How it works {#how-it-works}
 
-In this mode, the GCS ClickPipe does an initial load of **all files** in the selected path, and then listens for object notifications via the Pub/Sub subscription that match the specified path. Any message for a previously seen file, file not matching the path, or event of a different type will be **ignored**. It is **not possible** to start ingestion from a specific file or point in time — ClickPipes will always load all files in the selected path.
+In this mode, the GCS ClickPipe does an initial load of **all files** in the selected path, and then listens for object notifications via the Pub/Sub subscription that match the specified path. Any message for a previously seen file, file not matching the path, or event of a different type will be **ignored**.
+
+To ingest only files delivered through the subscription, enable **Skip initial load**. When enabled, ClickPipes skips the initial scan of existing files in the selected path and processes only files delivered through the Pub/Sub subscription.
 
 Various types of failures can occur when ingesting data, which can result in partial inserts or duplicate data. Object Storage ClickPipes are resilient to insert failures and provide exactly-once semantics using temporary staging tables. Data is first inserted into a staging table; if something goes wrong, the staging table is truncated and the insert is retried from a clean state. Only once an insert completes successfully are the partitions moved to the target table.
 
@@ -68,14 +70,14 @@ Various types of failures can occur when ingesting data, which can result in par
 
 **1.** In the ClickHouse Cloud console, navigate to **Data Sources > Create ClickPipe** and select **Google Cloud Storage**. Enter the details to connect to your GCS bucket. Under **Authentication method**, choose **Service Account** and provide the `.json` service account key.
 
-**2.** Toggle on **Continuous ingestion**, then select **Any order** as the ingestion mode and provide the **Pub/Sub subscription** name for the subscription connected to your bucket. The subscription name must follow the following format:
+**2.** Toggle on **Continuous ingestion**, then select **Any order** as the ingestion mode and provide the **Pub/Sub subscription** name for the subscription connected to your bucket. To process only files from the subscription and skip the initial scan of existing files in the selected path, enable **Skip initial load**. The subscription name must use the following format:
 
 ```text
 projects/${YOUR_PROJECT_ID}/subscriptions/${YOUR_SUBSCRIPTION_NAME}
 ```
 
 **3.** Click **Incoming data**. Define a **Sorting key** for the target table. Make any necessary adjustments to the mapped schema, then configure a role for the ClickPipes database user.
 
-**4.** Review the configuration and click **Create ClickPipe**. ClickPipes will perform an initial scan of your bucket to load all existing files that match the specified path, and will then begin processing files as new `OBJECT_FINALIZE` events arrive in the topic.
+**4.** Review the configuration and click **Create ClickPipe**. Unless **Skip initial load** is enabled, ClickPipes will perform an initial scan of your bucket to load all existing files that match the specified path, and will then begin processing files as matching `OBJECT_FINALIZE` events are delivered from the subscription.
 </Step>
 </Steps>