feat: add privacy authorization and output integration hooks

[yz271544]

Add experimental privacy authorization and output integration hooks

Related discussion: #1943

What does this PR do?

This PR introduces the Cloudberry core integration points for an experimental, extension-based output privacy design. It allows a companion extension to apply policy-based transformations or permit authorized access immediately before values are delivered to a client, without modifying executor input tuples.

This is the core integration portion, not a complete privacy protection implementation. JWT verification, session authorization, policy management, MASK/CPT transformations, and policy-version validation are implemented in the separate cloudberry-privacy project and are not included in this PR.

The core changes are:

1. Authorization command parsing

   • Add the unreserved AUTHORIZE keyword and parse AUTHORIZE PRIVACY FROM STDIN.
   • Represent the command as a dedicated CopyStmt marker for the extension's utility hook. With the companion extension, JWT bytes are received through CopyData rather than embedded in SQL text.
   • Parsing this command does not itself verify a JWT or grant any privilege.

2. Final-output hooks

   • Add versioned output and endpoint-export interfaces in utils/privacy_output.h (experimental ABI version 2).
   • Invoke the field-output hook in printtup.c and copyto.c before text or binary encoding.
   • Distinguish SELECT, RETURNING, COPY, external/callback output, and RETRIEVE operations.
   • Resolve the original cursor's execution context for FETCH so the extension can check authorization on continuation.

3. Distributed security labels

   • Dispatch SECURITY LABEL statements for the cloudberry_privacy provider with snapshot and two-phase transaction flags.
   • Add the required SecLabelStmt binary serialization/deserialization and text-output support.
   • Leave other label providers' dispatch behavior unchanged.

4. Parallel RETRIEVE integration

   • Pass the planned statement to endpoint setup and let the extension export an output-metadata-only plan.
   • Store that metadata in separately keyed endpoint shared-memory entries and copy it into the retrieving portal.
   • Apply the output hook at final delivery, allowing the extension to check RETRIEVE scope, expiry, and policy versions during continuation.
   • Refuse RETRIEVE when an endpoint contains privacy metadata but no output handler is installed.

The companion extension uses the existing PlannedStmt.extensionContext transport for QD-to-QE policy fingerprints and scoped capabilities. No new client protocol message is introduced.

Type of Change

• Bug fix (non-breaking change)
• New feature (non-breaking change)
• Breaking change (fix or feature with breaking changes)
• Documentation update

This is an experimental feature with internal ABI and endpoint shared-memory compatibility changes.

Breaking Changes

• PortalData gains an output-metadata field, and SetupEndpointExecState() gains a planned-statement argument.
• The endpoint shared-memory layout and magic value change. Mixed old/new endpoint producers and receivers are not supported.
• Deploy matching core and extension builds across coordinator, segment, and retrieve backends, rebuild affected binary modules, and restart the cluster together. Rolling mixed-version compatibility is not claimed.
• The hook interface is experimental and is not a stable extension ABI.

No on-disk table format change is introduced. Without registered handlers, these hooks do not apply privacy transformations; this patch alone does not protect labeled data.

Test Plan

Local validation used Cloudberry 3-devel / PostgreSQL 16.9, based on 867c6a147df, with this core patch and the companion extension. Tests used disposable installations and a one-coordinator/one-segment topology, not production data.

• Unit tests added/updated in this repository
• Integration tests added/updated in this repository
• Companion-extension algorithm, integration, isolation, and regression tests executed locally
• Passed make installcheck
• Passed make -C src/test installcheck-cbdb-parallel

Passed locally:

• Core and companion-extension builds.
• JWT rejection cases, fragmented CopyData reception, session cleanup, and log scans for test JWT/claim/CopyFail payloads.
• UTF-8 MASK and 768 CPT compatibility vectors, including round trips and invalid input domains.
• Masked and authorized SELECT, direct COPY TO, query COPY TO, binary COPY, RETURNING, and ordinary cursor output.
• Rejection of FETCH after clearing the cursor's session authorization.
• Policy publication/revision, publication waiting on reader locks, and committed-label cache invalidation.
• QD/QE policy-version mismatch rejection and prepared-plan authorization isolation.
• Parallel RETRIEVE masking, independent operation scope, authorized continuation, and expired-capability rejection.
• The companion extension's basic SQL regression through Cloudberry pg_regress: 1/1 passed.

The full Cloudberry regression and parallel installcheck suites have not been run. The test scripts and expected output are in the companion project, not in this core diff; repository-local coverage is still needed for upstream integration.

Reproduction from the companion project, with the patched Cloudberry source in an adjacent cloudberry directory:

python3 tests/prepare_stage.py
# Replace the path with the temporary stage printed by the command above.
python3 tests/run_all.py --stage /tmp/cloudberry-privacy-build.REPLACE_ME --benchmark-rows 10000

Impact

Performance:

The patch adds per-field hook checks and endpoint metadata serialization/storage. Standalone core overhead has not been isolated.

An exploratory companion-extension run fetched 10,000 rows with two fields using a single utility backend. Median complete PQexec times over seven measured runs were:

Scenario	Median
Extension loaded, no policy binding	13.45 ms
Two MASK-protected fields	33.88 ms
Two fields with authorized raw output	17.56 ms
Two CPT-protected fields	169.28 ms

These are end-to-end prototype measurements, not p95, multi-segment benchmarks, or performance acceptance results. They are affected by cache state, run order, and machine load. The current companion CPT adapter opens and validates its key file for every value; safe caching and key-rotation semantics remain future work.

User-facing changes:

• Adds the authorization command syntax, which requires the companion extension to handle it.
• With that extension enabled, supported protected outputs are transformed unless a matching capability permits raw values.
• Expired or changed capabilities can interrupt output; clients must treat interrupted results as incomplete.
• This does not change stored data or provide encryption at rest. Executor tuples and endpoint queues can still contain plaintext inside the trusted server boundary.

Dependencies:

• This core diff adds no third-party library dependency.
• The separately built companion extension uses OpenSSL and contains a CPT compatibility adapter.
• Its runtime configuration, policy APIs, and tests are not shipped by this PR.

Checklist

• Followed contribution guide
• Added/updated documentation in this repository
• Reviewed code for security implications during implementation
• This PR contains AI-assisted code generation
• Requested review from cloudberry committers

The implementation review is not an independent security audit. Companion-project documentation exists, but core documentation and upstream test integration remain follow-up work.

Additional Context

This change is submitted as an experimental implementation for design review, not as a production-ready security boundary.

The companion prototype is opt-in and supports a restricted direct-column query subset. General expression provenance, inference resistance, complete view/partition/DDL/restore coverage, durable fail-closed auditing, cluster-wide immediate revocation, protected internal-copy workflows, and fault/failover testing remain incomplete. Protected WITH HOLD cursors and several unsupported query shapes are currently rejected by the companion extension.

The new privacy metadata is intended to contain only field origins, policy/version identifiers, and scoped capabilities—not raw JWTs, keys, or protected values. This does not imply that ordinary query plans or internal tuples contain no plaintext.

Implementation references:

• Core commit: 9a36e58b961
• Companion project: cloudberry-privacy, commit a3c2831
• Companion validation report: docs/test-results-2026-09-09.md
• Companion test entry points: tests/prepare_stage.py and tests/run_all.py

Feedback is particularly welcome on the hook API, the provider-specific SECURITY LABEL dispatch, endpoint metadata ownership/lifetime, and the appropriate split between core and extension responsibilities.

CI

Please run the normal CI jobs for this core change. No CI skip marker is requested.

[github-actions]

Hi, @yz271544 welcome!🎊 Thanks for taking the effort to make our project better! 🙌 Keep making such awesome contributions!