Write egress access stats to Data Acess V3

[Merteg]

No description provided.

[Copilot]

Pull request overview

This PR adds an asynchronous Delta Lake writer for per-share egress access logs, allowing ACCESS_LOG telemetry to be persisted to per-tenant Delta tables (in addition to the existing JSON log stream) when configured.

Changes:

• Introduce DeltaAccessLogWriter and wire it behind AccessLogEmitter.create via a new CompositeAccessLogEmitter.
• Add new accessLogging configuration fields (deltaTablePath, deltaFlushIntervalSeconds, deltaFlushBatchSize) across templates/manifests and document the behavior.
• Ensure the writer is closed on JVM shutdown and add a dedicated test suite for the Delta writer.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
server/src/universal/conf/delta-sharing-server.yaml.template	Documents new access-log-to-Delta configuration knobs in the server YAML template.
server/src/test/scala/io/delta/sharing/server/telemetry/GcpPricingTierSuite.scala	Updates header year to satisfy the updated scalastyle header check.
server/src/test/scala/io/delta/sharing/server/telemetry/GcpIpRangeLookupSuite.scala	Updates header year to satisfy the updated scalastyle header check.
server/src/test/scala/io/delta/sharing/server/telemetry/DeltaAccessLogWriterSuite.scala	Adds test coverage for per-tenant Delta writing, partitioning, flush behavior, and schema checks.
server/src/test/scala/io/delta/sharing/server/telemetry/AccessLogEmitterSuite.scala	Adds tests for composite emitter creation and fan-out behavior.
server/src/main/scala/io/delta/sharing/server/telemetry/GcpPricingTier.scala	Updates header year to satisfy the updated scalastyle header check.
server/src/main/scala/io/delta/sharing/server/telemetry/GcpIpRangeLookup.scala	Updates header year to satisfy the updated scalastyle header check.
server/src/main/scala/io/delta/sharing/server/telemetry/DeltaAccessLogWriter.scala	New async, buffered per-tenant Delta writer implementation.
server/src/main/scala/io/delta/sharing/server/telemetry/AccessLogEmitter.scala	Adds close() default method, composite emitter, and conditional Delta writer wiring.
server/src/main/scala/io/delta/sharing/server/DeltaSharingService.scala	Registers a shutdown hook to close the access log emitter/writer.
server/src/main/scala/io/delta/sharing/server/config/ServerConfig.scala	Extends AccessLoggingConfig to include Delta writer configuration fields and defaults.
scalastyle-config.xml	Relax header check to allow any 20xx copyright year.
manifests/zing-preview/configmap.yaml	Enables deltaTablePath for the zing-preview environment.
manifests/zcloud-prod3/configmap.yaml	Enables deltaTablePath for the zcloud-prod3 environment.
manifests/zcloud-prod2/configmap.yaml	Enables deltaTablePath for the zcloud-prod2 environment.
manifests/zcloud-prod/configmap.yaml	Enables deltaTablePath for the zcloud-prod environment.
manifests/base/configmap.yaml	Enables deltaTablePath for the base (dev) environment.
docs/PER_SHARE_EGRESS_MONITORING.md	Documents Delta storage behavior, naming, partitioning, and configuration.

Comments suppressed due to low confidence (1)

server/src/main/scala/io/delta/sharing/server/config/ServerConfig.scala:182

• deltaFlushBatchSize and deltaFlushIntervalSeconds are user-configurable but not validated. In particular, deltaFlushBatchSize <= 0 will cause queue.drainTo(batch, flushBatchSize - 1) to throw IllegalArgumentException, breaking the background flush loop.

  override def checkConfig(): Unit = {
    // No required fields to validate
  }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (1)

server/src/main/scala/io/delta/sharing/server/config/ServerConfig.scala:182

• AccessLoggingConfig.checkConfig() is currently a no-op, but the new Delta writer settings can be misconfigured (e.g., deltaFlushIntervalSeconds <= 0 or deltaFlushBatchSize <= 0). Those values will lead to runtime errors (negative drainTo limit / invalid grouped() size) and repeated error logs.

  override def checkConfig(): Unit = {
    // No required fields to validate
  }

[bharatZen]

See minor comment