Skip to content

Add postgres_data_apis bundle resource - #7139

Closed
ruslantalpa wants to merge 9 commits into
databricks:mainfrom
ruslantalpa:postgres-data-api
Closed

ruslantalpa wants to merge 9 commits into
databricks:mainfrom
ruslantalpa:postgres-data-api

Conversation

@ruslantalpa

@ruslantalpa ruslantalpa commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Changes

Add the postgres_data_apis resource (Private Preview). It enables and configures the Data API (PostgREST) of a Lakebase Postgres database:

resources:
  postgres_data_apis:
    app:
      parent: ${resources.postgres_databases.app_db.name}
      db_schemas: [public]
      db_max_rows: 1000
      jwt_cache_max_lifetime: 300s
  • One Data API per database. Its ID is {parent}/data-api; changing parent recreates it. Removing the resource turns the Data API off; the database and its data are unaffected.
  • The serving URL is status.url; reference it as ${resources.postgres_data_apis.<key>.status.url}. It is a stable output field: it is built from the read-write endpoint host, the workspace and the database name, so an in-place update of the Data API cannot change it, and the reference resolves without planning a no-op update of the resources that use it.
  • Durations such as jwt_cache_max_lifetime use the protobuf seconds format, for example 300s.
  • Create, update and delete wait on long-running operations. Update sends the configured spec with update_mask=spec.
  • spec is INPUT_ONLY, so GET does not return it. DoRead fills the remote spec from status, which reports every setting in effect, so bundle plan detects out-of-band changes and bundle deployment bind applies the configured spec on the next deploy. The generated spec:input_only rules are dropped (apitypes.yml: postgres_data_apis: null) and every spec field is a hand-written backend_default, so a field the config omits is left to the server.
  • Status omits a bool that is false and was never set, so the bool fields are forced on the remote side.
  • A field removed from the config keeps its last value on the server: update_mask=spec merges, and per-field clearing is not used because clearing some fields currently leaves the Data API unable to serve requests. The removal plans one no-op update; the next plan is clean.
  • Labelled Private Preview to match the API's launch stage (the other postgres_* resources are Beta), so it is not suggested in the JSON schema and not exposed in PyDABs yet.

Why

Lakebase users already manage projects, branches and databases with bundles and want to manage the Data API alongside them. The API is at Private Preview.

Notes for reviewers:

  • Like every other postgres_* resource, this one waits on its operation inline in create, update and delete, because the operation ID isn't persisted.
  • Registering the new type with the deployment metadata service is a separate server-side change. Until it is deployed, deploys with experimental.deployment_history enabled reject this resource type, so the cloud acceptance cells that run with deployment history on (DMS=true) will fail. Deploys without it are unaffected.
  • Cloud coverage: the resource acceptance tests run on cloud (AWS only, like the siblings); the invariant and bind tests are local-only, like the siblings. A manual live run against a dev workspace covered the full lifecycle.
  • jwt_role_claim_key and jwt_cache_max_lifetime are exposed because the API accepts them. Their runtime behaviour is still evolving at Private Preview; happy to hide them if you prefer.

Tests

  • Acceptance tests against the local test server: basic, update, recreate, out-of-band, bind, and url_reference (a job parameter that references status.url resolves on the first deploy, and an in-place update of the Data API plans no update of the job). The test server's Data API fake mirrors the real API's defaults, status.url, merge-on-update semantics, and error codes and messages.
  • Invariant config covering strings, lists, bools (true and an explicit false), a duration and an enum: no drift, removal of every optional field converges, delete/destroy idempotent.
  • Unit tests for the status-to-spec mapping and the fake.
  • Manual run against a dev workspace with a CLI built from this branch: deploy, serving through the Data API URL with a service principal token, a job parameter referencing status.url resolved on the first deploy, update (row cap applied, job not updated, next plan clean), removed field (value kept), out-of-band change detected and reverted, bind of an existing Data API applied on deploy, destroy (Data API off, data intact).

This pull request and its description were written by Isaac.

ruslantalpa and others added 3 commits October 21, 2026 06:22
Add the `postgres_data_apis` resource, which enables and configures the
Data API (PostgREST) of a Lakebase Postgres database.

The Data API is a per-database singleton with no ID of its own: its
resource name is "{parent}/data-api", and `parent` is a provided id
field, so changing it recreates. Create, update, and delete are
long-running operations. Updates send the full spec with
update_mask=spec, so a field removed from the config resets to its
server default.

spec is INPUT_ONLY in the API, so GET does not return it and remote
spec changes are not detected (ignore_remote_changes spec:input_only,
generated). Deleting the resource turns the Data API off; the database
and its data are unaffected.

The resource is labelled Private Preview, matching the API's launch
stage, rather than the Beta its postgres_* siblings are pinned to.

Co-authored-by: Isaac <no-reply@databricks.com>
Observed against a real workspace:

- status reports the configuration in effect, server defaults included:
  db_aggregates_enabled=true, db_schemas=["public"],
  jwt_role_claim_key=".sub", openapi_mode=OPEN_API_MODE_DISABLED and
  server_cors_allowed_origins=["none"]. A bool that was never set and is
  false is omitted; once set it is reported, false included.
- update_mask=spec keeps the current value of a field the request omits;
  it does not reset it to the default.
- Errors: 400 INVALID_PARAMETER_VALUE "Field 'data_api' is required and
  must contain at least one subfield with a non-default value!" for an
  empty spec, 409 ALREADY_EXISTS "Data API is already enabled for this
  database", and 404 NOT_FOUND "Data API is not enabled for this
  database" for get, update and delete when it is not enabled.

The acceptance tests now also print server_cors_allowed_origins, and the
update test shows that a field removed from the config keeps its value.

Co-authored-by: Isaac <no-reply@databricks.com>
GET does not return spec (it is INPUT_ONLY), so the plan could not see a
remote spec change, and a bound Data API kept its old configuration until
the bundle config changed. status reports every spec field in effect, so
DoRead now fills the remote spec from it.

The generated spec:input_only rules no longer apply, so apitypes.yml drops
the type for postgres_data_apis and the field rules are hand-written:
every spec field is a backend_default with no allowed values. A field the
config omits is left to the server: create applies the server default, and
update_mask=spec keeps the current value of an omitted field, so a field
removed from the config keeps its last value. Leaf update masks would reset
it, but resetting db_max_rows or jwt_role_claim_key that way leaves the
Data API unable to serve requests, so the mask stays at spec.

Status omits a false bool that was never set, so the remote spec forces
db_aggregates_enabled and server_timing_enabled: an explicit false in the
config compares against false, not against an unknown value.

With this, binding an existing Data API plans an update that applies the
configured spec, and an out-of-band spec change shows up in the plan.

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution! This pull request was opened from a fork.

If you're a member of the Databricks organization, please request access to the cli-contrib team and open your pull request from a branch in this repository instead of a fork — fork PRs don't run integration tests automatically and have reduced CI access.

If you're an external contributor, no action is needed; a maintainer will review your pull request.

@github-actions github-actions Bot added the DABs DABs related issues label Oct 21, 2026
ruslantalpa and others added 6 commits October 21, 2026 06:55
Co-authored-by: Isaac <no-reply@databricks.com>
The real API reports the serving URL in status.url:
https://{read-write endpoint host}/api/2.0/workspace/{workspace_id}/rest/{postgres_database}.
The fake now fills it in GET and in the create and update operation
responses, from the branch's read-write endpoint host, the fake
workspace ID and the database's Postgres name.

The new url_reference acceptance test has a job parameter that references
${resources.postgres_data_apis.<key>.status.url} and shows it resolves on
the first deploy. It also records that an in-place update of the Data API
currently plans an update of the job too.

Co-authored-by: Isaac <no-reply@databricks.com>
status.url is built from the read-write endpoint host, the workspace and
the database's Postgres name. An update of the Data API changes none of
them, but without this the reference resolver delays any
${resources.postgres_data_apis.<key>.status.url} reference during an
in-place update, so every dependent got a no-op update planned too.

The url_reference acceptance test now shows that changing db_max_rows
plans an update of the Data API only, and the next plan is clean.

Co-authored-by: Isaac <no-reply@databricks.com>
The invariant config now also sets db_aggregates_enabled: false,
server_timing_enabled: true, jwt_cache_max_lifetime: 300s and
openapi_mode: OPEN_API_MODE_IGNORE_PRIVILEGES, so no_drift checks the
status readback for a forced false bool, a true bool, a duration and an
enum.

Every optional field is tagged CAN_REMOVE, the bools included. Removing
either bool on its own plans an update and converges: the server keeps
the last value, status reports it, and the field is a backend default.

Co-authored-by: Isaac <no-reply@databricks.com>
The resource's schema description was inherited from the embedded SDK
spec type ("Desired PostgREST configuration (input)."). Override it to
say what the resource is, that a database has at most one Data API,
where the serving URL is (status.url) and how durations are written
(300s).

Also override the postgres.OpenApiMode description, which mentioned a
value that may be added later, with what the two supported values do.

Co-authored-by: Isaac <no-reply@databricks.com>
The comment above the PostgresDataApi entry split gofmt's alignment
block, which re-indented the unrelated PostgresBranch and
PostgresCatalog lines. Make it a trailing comment, so the diff against
main is only the added entry.

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

Copy link
Copy Markdown
Contributor

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

  • PR number: 7139
  • Commit SHA: c30a3682aaf1dbabae0c5e7209300ba0a524c4e3

Checks will be approved automatically on success.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant