Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
115 changes: 74 additions & 41 deletions docs/features/mirroring.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,103 +4,136 @@ icon: material/mirror-rectangle
# Mirroring


Database mirroring replicates traffic, byte for byte, from one database to another. This allows you to test how databases respond to real, production traffic.
Database mirroring streams traffic, byte for byte, from one database to another. This allows you to test how databases respond to real, production traffic.

## How it works

Mirroring in PgDog is asynchronous and should have minimal impact on production databases: transactions are sent to a background task, which in turn forwards them to one or more mirror databases. If any statement fails, the error is ignored and the next one is executed.
Mirroring in PgDog is asynchronous and should have minimal impact on production databases: transactions are sent to a background worker, which in turn forwards them to one or more mirror databases. If any statement fails, the error is ignored and the next one is executed. All query results are discarded as well.

<center>
<img src="/images/mirroring.png" width="80%" height="auto" alt="Mirroring">
<img src="/images/mirroring.png" width="80%" height="auto" alt="Mirroring" class="theme-aware-image">
<p>Mirroring architecture</p>
</center>

### Configuration

To use mirroring, first configure both the mirror and the production database in [`pgdog.toml`](../configuration/pgdog.toml/databases.md). Once both databases are running, add a `[[mirroring]]` section:

```toml
[[databases]]
name = "prod"
host = "10.0.0.1"

[[databases]]
name = "staging_db"
host = "10.0.2.25"

[[mirroring]]
source_db = "prod"
destination_db = "staging_db"
# queue_length = 256 # Optional: overrides general.mirror_queue
# exposure = 0.5 # Optional: overrides general.mirror_exposure
```
=== "pgdog.toml"
```toml
[[databases]]
name = "prod"
host = "10.0.0.1"

[[databases]]
name = "staging_db"
host = "10.0.2.25"

[[mirroring]]
source_db = "prod"
destination_db = "staging_db"
# queue_length = 256 # Optional: overrides general.mirror_queue
# exposure = 0.5 # Optional: overrides general.mirror_exposure
```
=== "Helm chart"
```yaml
databases:
- name: prod
host: 10.0.0.1
- name: staging_db
host: 10.0.2.25
mirroring:
- sourceDb: prod
destinationDb: stagingDb
# queueLength: 256
# exposure: 0.5
```

!!! note "Required users"
Mirrored databases are, underneath, regular connection pools and require a user & password configured in `users.toml`. PgDog will use those settings to connect to the mirror database and forward queries.
!!! note "Matching users"
Mirrored databases are regular connection pools and require a user and password, configured in [`users.toml`](../configuration/users.toml/users.md). PgDog will use those settings to connect to the mirror database and forward queries, so make sure the **same** users are configured on both databases.

You can connect to the mirror database like any other. The same connection pool will be used for mirrored queries. Production database connection pool will not be affected, since all replication happens in the background.
You can connect to the mirror database like any other. The same connection pool will be used for mirrored queries. The production database connection pools will not be affected, since all traffic streaming happens in the background.

Each client connected to the main database has its own queue, so concurrency scales linearly with the number of clients.

You can have as many mirror databases as you like. Queries will be sent to each one of them, in parallel. More mirrors will require more CPU and network resources, so make sure to allocate enough compute to PgDog in production.

### Mirror queue

If the mirror database(s) can't keep up with production traffic, queries will back up in the queue. To make sure it doesn't overflow and cause out-of-memory errors, the size of the queue is limited:

=== "General settings"
If mirror databases can't keep up with production traffic, queries will back up in the queue. To make sure it doesn't overflow and cause out-of-memory errors, the size of the queue is limited:

=== "pgdog.toml"
```toml
[general]
mirror_queue = 500
```
=== "Mirroring settings"
!!! note "Settings priority"
Settings values in individual `[[mirroring]]` sections take priority over general settings, for that particular mirror database.
=== "Helm chart"
```yaml
mirrorQueue: 500
```

You can also configure the mirror queue settings on a per-mirror basis, for example:

=== "pgdog.toml"
```toml
[[mirroring]]
source_db = "source"
destination_db = "dest"
queue_length = 500
```
=== "Helm chart"
```yaml
mirroring:
- sourceDb: source
destinationDb: dest
queueLength: 500
```

If the queue gets full, all subsequent mirrored transactions will be dropped until there is space in the queue again.

!!! note "Replication"
Since mirror queues can drop queries, it is not a replacement for Postgres replication and should be used for testing & benchmarking purposes only.
!!! warning "Mirroring is not replication"
Since mirror queues can drop queries, it is not a replacement for Postgres replication and should be used for testing/benchmarking purposes only.

### Exposure

It's possible to limit how much traffic mirror databases receive. This is useful when warming up databases from a snapshot or if the mirror databases are smaller than production and can't handle as many transactions.

This is configurable using a percentage, relative to the amount of transactions sent to the source database:
It's possible to limit how much traffic mirror databases receive. This is useful when warming up databases restored from a backup, or if the mirror databases are smaller than production and can't handle as many transactions.

=== "General settings"
This is configurable using a percentage, relative to the number of transactions sent to the source database:

=== "pgdog.toml"
```toml
[general]
mirror_exposure = 0.5 # 50%
```
=== "Mirroring settings"
!!! note "Settings priority"
Settings values in individual `[[mirroring]]` sections take priority over general settings, for that particular mirror database.
=== "Helm chart"
```yaml
mirrorExposure: 0.5
```

The same setting can be configured on individual mirrors:

=== "pgdog.toml"
```toml
[[mirroring]]
source_db = "source"
destination_db = "dest"
exposure = 0.5
```
=== "Helm chart"
```yaml
mirroring:
- sourceDb: source
destinationDb: dest
exposure: 0.5
```

Acceptable values are between **0.0** (0%) and **1.0** (100%).

This is changeable at runtime, without restarting PgDog. When adding a mirror, it's a good idea to start slow, e.g., with only 0.1% exposure (`mirror_exposure = 0.01`), and gradually increase it over time.
This is changeable at runtime, without restarting PgDog. When adding a mirror, it's a good idea to start small, e.g., with only 1% exposure (i.e., `mirror_exposure = 0.01`), and gradually increase it over time.

### Realism

We try to make mirrored traffic as realistic as possible. For each statement inside a transaction, we record the timing between that statement and the next one.

When replaying traffic against a mirror, we pause between statements for the same amount of time. This helps reproduce lock contention experienced by production databases, on the mirrors.
PgDog tries to make mirrored traffic as realistic as possible. For each statement inside a transaction, we record the timing between that statement and the next one. When replaying traffic against a mirror, we pause between statements for the same amount of time. This helps reproduce lock contention experienced by production databases, on the mirror databases.

### Filtering

Expand All @@ -125,7 +158,7 @@ The `level` setting supports the following arguments:

| Argument | Description |
|-|-|
| `ddl` | Mirror only DDL statements like `CREATE`, `DROP`, etc. |
| `ddl` | Mirror only DDL statements, e.g., `CREATE`, `DROP`, etc. |
| `dml` | Mirror all statements except DDL, e.g. `INSERT`, `UPDATE`, etc. |
| `all` | Mirror all statements. This is the default. |

Expand Down
2 changes: 1 addition & 1 deletion docs/features/sharding/cross-shard-queries/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Just like with [direct-to-shard](../query-routing.md) queries, each SQL command
| Commands | Summary |
|-|-|
| [SELECT](select.md) | PgDog implements a scatter/gather query engine to fetch rows from multiple shards concurrently. |
| [INSERT](insert.md) | Statements targeting [omnisharded](omnishards.md) tables are sent to all shards concurrently. Sharded tables with automatic [primary key](../unique-ids.md) generation are sent to one shard only. |
| [INSERT](insert.md) | Statements targeting [omnisharded](../omnishards.md) tables are sent to all shards concurrently. Sharded tables with automatic [primary key](../unique-ids.md) generation are sent to one shard only. |
| [UPDATE and DELETE](update.md) | Statements are sent to all shards concurrently. Sharding key updates are partially supported. |
| [DDL statements, e.g., CREATE, ALTER, DROP](ddl.md) | DDL is sent to all shards concurrently, to make sure the schema is identical on all shards. |
| [COPY command](copy.md) | Rows sent via COPY are automatically distributed between all shards using the configured [sharding function](../sharding-functions.md). |
Expand Down
Binary file modified docs/images/mirroring.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading