diff --git a/README.md b/README.md index acd837ce..425c0c62 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,30 @@ # Databricks SQL Driver for Go - ![http://www.apache.org/licenses/LICENSE-2.0.txt](http://img.shields.io/:license-Apache%202-brightgreen.svg) -## Description - -This repo contains a Databricks SQL Driver for Go's [database/sql](https://golang.org/pkg/database/sql) package. It can be used to connect and query Databricks clusters and SQL Warehouses. - -## Documentation - -See `doc.go` for full documentation or the Databrick's documentation for [SQL Driver for Go](https://docs.databricks.com/dev-tools/go-sql-driver.html). - -## Usage +A [database/sql](https://golang.org/pkg/database/sql) driver for Databricks SQL. It +connects to Databricks SQL Warehouses and clusters and runs queries through Go's +standard `database/sql` interface. + +## Contents + +- [Quick start](#quick-start) +- [Choosing a backend (Thrift vs SEA/kernel)](#choosing-a-backend-thrift-vs-seakernel) +- [Building](#building) +- [Connecting](#connecting) + - [DSN (Data Source Name)](#dsn-data-source-name) + - [Connector object](#connector-object) +- [Connection properties](#connection-properties) +- [Authentication](#authentication) +- [Cloud Fetch](#cloud-fetch) +- [TLS](#tls) +- [Proxy](#proxy) +- [Data types](#data-types) +- [Telemetry](#telemetry) +- [Examples](#examples) +- [Develop](#develop) + +## Quick start ```go import ( @@ -26,128 +39,156 @@ if err != nil { } defer db.Close() - rows, err := db.QueryContext(context.Background(), "SELECT 1") defer rows.Close() ``` -Additional usage examples are available [here](https://github.com/databricks/databricks-sql-go/tree/main/examples). +See [`doc.go`](./doc.go) for full package documentation or the Databricks documentation +for the [SQL Driver for Go](https://docs.databricks.com/dev-tools/go-sql-driver.html). -### Connecting with DSN (Data Source Name) +## Choosing a backend (Thrift vs SEA/kernel) -The DSN format is: +The driver has **two execution backends**, selected once per connection: -``` -token:[your token]@[Workspace hostname]:[Port number][Endpoint HTTP Path]?param=value -``` +| Backend | Transport | Default? | Build requirement | +|---|---|---|---| +| **Thrift / HiveServer2** | Thrift RPC over HTTP | ✅ yes | pure Go, `CGO_ENABLED=0`, cross-compilable | +| **SEA / kernel** *(experimental)* | Statement Execution API (REST) via the Rust [`databricks-sql-kernel`](https://github.com/databricks/databricks-sql-kernel), over a cgo C ABI | no (opt-in) | `-tags databricks_kernel` **and** `CGO_ENABLED=1`; links the kernel static library | -The `token:[your token]@` prefix authenticates with a personal access token (PAT). For other authentication types, omit the prefix and use the `authType`, `clientID`/`clientSecret`, or `accessToken` parameters described below. +Thrift is the default and needs no special setup. Select the SEA/kernel backend per +connection either way: -#### Supported connection parameters +- **Connector option:** `dbsql.WithUseKernel(true)` +- **DSN parameter:** `useKernel=true` -Optional parameters can be appended to the DSN as `?param=value¶m=value`: +If the binary was **not** built with the `databricks_kernel` tag, selecting the kernel +backend returns an error wrapping `dbsqlerr.ErrKernelNotCompiled` at connect — it never +silently falls back to Thrift. -| Parameter | Description | Default | -|---|---|---| -| `catalog` | Sets the initial catalog name in the session | | -| `schema` | Sets the initial schema name in the session | | -| `maxRows` | Max rows fetched per network request | `100000` | -| `timeout` | Server-side query execution timeout, in seconds | no timeout | -| `userAgentEntry` | Identifies your application (partners/ISVs). Format: `` | | -| `useCloudFetch` | Enables Cloud Fetch to fetch large results in parallel via cloud storage | `true` | -| `maxDownloadThreads` | Number of concurrent Cloud Fetch download goroutines | `10` | -| `authType` | Authentication type. One of `Pat`, `OauthM2M`, `OauthU2M` | inferred from params | -| `accessToken` | Personal access token. Used when `authType=Pat` | | -| `clientID` | Service principal client ID. Used with OAuth M2M | | -| `clientSecret` | Service principal client secret. Used with OAuth M2M | | +**Parameter parity.** Parameters are intended to behave identically on both backends. +Where a backend can't honor an option it is **rejected** at connect or execute (wrapping +`dbsqlerr.ErrNotSupportedByKernel`), not silently ignored. The +[Connection properties](#connection-properties) **Protocol** column records, per +parameter, whether it applies to **Both**, **Thrift-only**, or **SEA-only**. -Any parameter not listed above (e.g. `ansi_mode`, `timezone`) is passed through as a session parameter. +## Building -For example, to set a query timeout and max rows per request: +The two backends differ **at build time**, not just at connect. -``` -token:[your token]@[Workspace hostname]:[Port number][Endpoint HTTP Path]?timeout=1000&maxRows=1000 -``` +### Thrift (default) — pure Go, no extra step -#### Cloud Fetch +Pure Go, `CGO_ENABLED=0`, `go get`-able, cross-compilable to any `GOOS`/`GOARCH`. No C, +no Rust, no linked native library. -Cloud Fetch (enabled by default) increases the performance of extracting large query results by fetching data in parallel via cloud storage (more info [here](https://www.databricks.com/blog/2021/08/11/how-we-achieved-high-bandwidth-connectivity-with-bi-tools.html)). You can set the number of concurrently fetching goroutines with `maxDownloadThreads`: +```bash +go build ./... +go test ./... -``` -token:[your token]@[Workspace hostname]:[Port number][Endpoint HTTP Path]?useCloudFetch=true&maxDownloadThreads=3 -``` -To disable Cloud Fetch (e.g., when handling smaller datasets or to avoid additional overhead), append `useCloudFetch=false`: -``` -token:[your token]@[Workspace hostname]:[Port number][Endpoint HTTP Path]?useCloudFetch=false +# Repo Makefile equivalents (both CGO_ENABLED=0): +make build # multi-arch pure-Go binaries (linux + darwin) +make test # pure-Go unit tests ``` -#### Authenticating with OAuth (client ID and secret) +Cross-compiling is free, e.g. `GOOS=windows GOARCH=amd64 go build ./...`. -To authenticate with OAuth machine-to-machine (M2M) credentials instead of a personal access token, leave the `token:...@` prefix off the DSN and pass the service principal's `clientID` and `clientSecret` as query parameters: +### SEA/kernel — cgo + a linked Rust static library -``` -[Workspace hostname]:[Port number][Endpoint HTTP Path]?authType=OauthM2M&clientID=[your client ID]&clientSecret=[your client secret] -``` +The kernel backend compiles in **only** under the `databricks_kernel` build tag with +`CGO_ENABLED=1`, and links the Rust kernel's C ABI as a static library +(`libdatabricks_sql_kernel.a`). That archive is **not committed** — build it first. -The `authType=OauthM2M` parameter is optional — supplying `clientID` and `clientSecret` is enough to select OAuth M2M authentication. +**Prerequisites for a source build:** -To authenticate interactively with OAuth user-to-machine (U2M) credentials (opens a browser login flow), set `authType=OauthU2M` with no token or client credentials: +- A C toolchain (cgo) and `CGO_ENABLED=1`. +- A Rust toolchain (`cargo`), pinned to the channel in + [`rust-toolchain.toml`](./rust-toolchain.toml) so the archive is reproducible. +- Network access to clone the kernel repo at the pinned revision (the + [`KERNEL_REV`](./KERNEL_REV) file). -``` -[Workspace hostname]:[Port number][Endpoint HTTP Path]?authType=OauthU2M -``` +```bash +# 1. Build the pinned kernel static lib + C header into the cgo link dir. +# Clones databricks-sql-kernel @ KERNEL_REV and cargo-builds a self-contained +# archive with pure-Rust TLS. +make kernel-lib -#### Setting the user agent +# 2. Build the driver with the kernel backend linked (implies step 1). +make build-kernel # == CGO_ENABLED=1 go build -tags databricks_kernel ./... -To identify your application (e.g. for partners/ISVs), append a `userAgentEntry` query parameter with the format ``: +# Run the kernel-tagged unit tests (no warehouse needed; step 1 implied): +make test-kernel # == CGO_ENABLED=1 go test -tags databricks_kernel ./... +``` +Once the archive exists you can invoke `go` directly, but you must carry **both** +`CGO_ENABLED=1` and `-tags databricks_kernel` — dropping either produces a pure-Go +binary where `WithUseKernel(true)` fails at connect: + +```bash +CGO_ENABLED=1 go build -tags databricks_kernel ./... ``` -token:[your token]@[Workspace hostname]:[Port number][Endpoint HTTP Path]?userAgentEntry=[your-isv-name+product-name] + +**Cross-compiling.** The source build is host-native only (`cargo` emits a host-native +`.a`, and `make kernel-lib` rejects a cross-build). For a non-host target, either build +on a native per-OS runner, or stage a prebuilt archive and skip the clone + cargo +entirely (no Rust toolchain needed): + +```bash +make kernel-lib KERNEL_LOCAL_A=/path/to/libdatabricks_sql_kernel.a \ + KERNEL_LOCAL_HEADER=/path/to/databricks_kernel.h # header optional ``` -### Telemetry Configuration (Optional) +### Build differences at a glance -The driver includes optional telemetry to help improve performance and reliability. Telemetry is **disabled by default** and requires explicit opt-in. +| | Thrift (default) | SEA/kernel | +|---|---|---| +| Build tag | none | `-tags databricks_kernel` | +| cgo | `CGO_ENABLED=0` | `CGO_ENABLED=1` | +| Native lib | none | links `libdatabricks_sql_kernel.a` (not committed) | +| Extra toolchain | none | Rust (`cargo`) + C toolchain | +| Prep step | none | `make kernel-lib` (or stage `.a` via `KERNEL_LOCAL_A`) | +| One-shot build | `go build ./...` | `make build-kernel` | +| Cross-compile | free (any `GOOS`/`GOARCH`) | host-native only; per-OS runner or staged `.a` | -**Opt-in to telemetry** (respects server-side feature flags): -``` -token:[your token]@[Workspace hostname]:[Port number][Endpoint HTTP Path]?enableTelemetry=true -``` +## Connecting + +### DSN (Data Source Name) -**Opt-out of telemetry** (explicitly disable): ``` -token:[your token]@[Workspace hostname]:[Port number][Endpoint HTTP Path]?enableTelemetry=false +token:[your token]@[Workspace hostname]:[Port number][Endpoint HTTP Path]?param=value¶m=value ``` -**What data is collected:** -- ✅ Query latency and performance metrics -- ✅ Error codes (not error messages) -- ✅ Feature usage (CloudFetch, LZ4, etc.) -- ✅ Driver version and environment info +The `token:[your token]@` prefix authenticates with a personal access token (PAT). For +other authentication types, omit the prefix and use the `authType`, +`clientID`/`clientSecret`, or `accessToken` parameters — see +[Authentication](#authentication). + +```go +db, err := sql.Open("databricks", + "token:@:443/sql/1.0/warehouses/?timeout=1000&maxRows=1000") +``` -**What is NOT collected:** -- ❌ SQL query text -- ❌ Query results or data values -- ❌ Table/column names -- ❌ User identities or credentials +To use the SEA/kernel backend, append `useKernel=true` (and, optionally, +`warehouseId=`): -Telemetry has < 1% performance overhead and uses circuit breaker protection to ensure it never impacts your queries. For more details, see `telemetry/DESIGN.md` and `telemetry/TROUBLESHOOTING.md`. +``` +token:@:443/sql/1.0/warehouses/?useKernel=true +``` -### Connecting with a new Connector +### Connector object -You can also connect with a new connector object. For example: +You can also connect with a connector object built from functional options: ```go import ( -"database/sql" - _ "github.com/databricks/databricks-sql-go" + "database/sql" + dbsql "github.com/databricks/databricks-sql-go" ) connector, err := dbsql.NewConnector( - dbsql.WithServerHostname(), - dbsql.WithPort(), - dbsql.WithHTTPPath(), - dbsql.WithAccessToken() + dbsql.WithServerHostname(), + dbsql.WithPort(), + dbsql.WithHTTPPath(), + dbsql.WithAccessToken(), + // dbsql.WithUseKernel(true), // opt into the SEA/kernel backend ) if err != nil { log.Fatal(err) @@ -156,24 +197,216 @@ db := sql.OpenDB(connector) defer db.Close() ``` -View `doc.go` or `connector.go` to understand all the functional options available when creating a new connector object. +See [`doc.go`](./doc.go) or [`connector.go`](./connector.go) for the full set of +functional options. + +## Connection properties + +Optional DSN parameters are appended as `?param=value¶m=value`; the equivalent +connector options are listed alongside. The **Protocol** column shows applicability: + +- **Both** — honored identically on Thrift and SEA/kernel. +- **Thrift only** — honored on Thrift; **rejected** at connect on the kernel path + (wraps `ErrNotSupportedByKernel`) unless noted "inert" (accepted, no effect). +- **SEA only** — kernel path only. The experimental `WithKernel*` options are + **rejected** on Thrift when set without `WithUseKernel(true)` (wraps + `ErrRequiresKernelBackend`); `warehouseId` is the exception — Thrift silently ignores + it (see its row). + +Any parameter not recognized below (e.g. `ansi_mode`, `timezone`) is passed through as a +session parameter on both backends. + +### Endpoint & routing + +| DSN parameter | Connector option | Protocol | Default | Description | +|---|---|---|---|---| +| *(host)* | `WithServerHostname` | Both | *(required)* | Workspace hostname. | +| *(path)* | `WithHTTPPath` | Both | *(required)* | Warehouse/endpoint HTTP path. | +| *(port)* | `WithPort` | Thrift only | `443` | Kernel connects on **443 only** and rejects any other port. | +| `warehouseId` | `WithWarehouseID` | SEA only | | Bare warehouse id; the kernel routes by it (preferred over the HTTP path). **The Thrift backend ignores it.** | +| `catalog` | `WithInitialNamespace` | Both | | Initial catalog. Kernel applies it post-connect via `USE CATALOG`. | +| `schema` | `WithInitialNamespace` | Both | | Initial schema. Kernel applies it post-connect via `USE SCHEMA`. | +| `useKernel` | `WithUseKernel` | Both | `false` | Select the SEA/kernel backend. Requires a `databricks_kernel` build. | + +### Query execution + +| DSN parameter | Connector option | Protocol | Default | Description | +|---|---|---|---|---| +| `maxRows` | `WithMaxRows` | Thrift only (inert on kernel) | `100000` | Max rows per fetch. On the kernel path the kernel manages paging, so this is accepted but has no effect. | +| `timeout` | `WithTimeout` | Thrift only | no timeout | Server-side query timeout, in seconds. On the kernel path use the `STATEMENT_TIMEOUT` session parameter instead. | +| `userAgentEntry` | `WithUserAgentEntry` | Both | | Identifies your application (partners/ISVs), format ``. | +| *(session param)* | `WithSessionParams` | Both | | Arbitrary session confs (e.g. `ansi_mode`, `STATEMENT_TIMEOUT`, `QUERY_TAGS`). | +| *(via session param)* | `WithQueryTags` | Both | | Session-level query tags (serialized into `QUERY_TAGS`). | +| `timezone` | `WithSessionParams(timezone=…)` | Both | | Session time zone (e.g. `America/Los_Angeles`). | +| `enableMetricViewMetadata` | `WithEnableMetricViewMetadata` | Both | `false` | Enables metric-view metadata (`spark.sql.thriftserver.metadata.metricview.enabled=true`). | + +### Retry / backoff + +| Connector option | Protocol | Default | Description | +|---|---|---|---| +| `WithRetries(retryMax, waitMin, waitMax)` | Both | `4`, `1s`, `30s` | Retry attempts and exponential-backoff bounds. `retryMax < 0` disables retries. | +| `WithKernelRetryOverallTimeout(d)` | SEA only | kernel default (900s) | Cumulative retry budget across all attempts. No Thrift equivalent. | + +### Result rendering + +| DSN parameter | Connector option | Protocol | Default | Description | +|---|---|---|---|---| +| `useArrowNativeDecimal` | `WithArrowNativeDecimal` | Thrift only (inert on kernel) | `false` | Thrift: return DECIMAL as native Arrow `decimal128` (lossless string when scanned via `database/sql`). The kernel path already renders DECIMAL as the exact string regardless. | +| | `WithKernelDecimalAsFloat(b)` | SEA only | `false` | Scan top-level DECIMAL as lossy `float64` instead of the exact string. | + +See [Cloud Fetch](#cloud-fetch), [TLS](#tls), and [Proxy](#proxy) for the remaining +groups. Telemetry parameters are covered under [Telemetry](#telemetry). + +## Authentication + +| Method | DSN | Connector option | Protocol | +|---|---|---|---| +| Personal access token (PAT) | `token:@…`, or `accessToken=` / `authType=Pat` | `WithAccessToken` | Both | +| OAuth machine-to-machine (M2M) | `clientID=`+`clientSecret=` / `authType=OauthM2M` | `WithClientCredentials` | Both | +| OAuth user-to-machine (U2M) | `authType=OauthU2M` | `WithAuthenticator` (u2m) | Both | +| Custom token provider / external / static / federated | — | `WithTokenProvider`, `WithExternalToken`, `WithStaticToken`, `WithFederatedTokenProvider*` | Thrift only | + +**PAT** (default): supply `token:@…` in the DSN, or `WithAccessToken`. + +**OAuth M2M**: leave the `token:...@` prefix off and pass the service principal's +`clientID` and `clientSecret`: + +``` +[host]:443[path]?authType=OauthM2M&clientID=&clientSecret= +``` + +`authType=OauthM2M` is optional — supplying `clientID` + `clientSecret` selects M2M. + +**OAuth U2M** (interactive browser login): + +``` +[host]:443[path]?authType=OauthU2M +``` + +Notes for the SEA/kernel backend: + +- Custom OAuth **M2M scopes** are rejected on the kernel path (the kernel applies its + own default scopes). Default scopes work on both. +- **U2M** is interactive: on a cache miss, connecting launches the browser and a + connect-context **deadline is not honored** during the login window. U2M scopes are at + parity with Thrift. Use PAT or M2M for headless/deadline-bound connects. +- Custom token-provider / external / static / federated authenticators are **Thrift + only**. +- OAuth token caching/refresh is owned by the kernel on the kernel path (no driver + config). + +## Cloud Fetch + +Cloud Fetch increases performance of extracting large results by fetching data in +parallel via cloud storage +([more info](https://www.databricks.com/blog/2021/08/11/how-we-achieved-high-bandwidth-connectivity-with-bi-tools.html)). + +| DSN parameter | Connector option | Protocol | Default | Description | +|---|---|---|---|---| +| `useCloudFetch` | `WithCloudFetch` | Thrift only (inert on kernel) | `true` | Enable Cloud Fetch. On the kernel path Cloud Fetch is always managed internally, so the flag is inert. | +| `maxDownloadThreads` | `WithMaxDownloadThreads` | Thrift only (inert on kernel) | `10` | Concurrent download goroutines (Thrift). Inert on the kernel path. | +| | `WithKernelMaxChunksInMemory(n)` | SEA only | kernel default (16) | Bounds decompressed Cloud Fetch chunks held in memory — trades large-result throughput for peak memory. | + +On the Thrift backend: + +``` +token:@:443[path]?useCloudFetch=true&maxDownloadThreads=3 +# or disable it entirely: +token:@:443[path]?useCloudFetch=false +``` + +## TLS + +| Connector option | Protocol | Description | +|---|---|---| +| `WithSkipTLSHostVerify()` | Both | Disable TLS chain + hostname verification. **Use only for internal private-link hostnames** — this is susceptible to machine-in-the-middle attacks. | +| `WithTransport(http.RoundTripper)` | Thrift only | Supply a custom HTTP transport (e.g. a custom CA, mTLS, or proxy). **Rejected** on the kernel path (wraps `ErrNotSupportedByKernel`) — the kernel uses its own HTTP stack; use `WithKernelTrustedCerts` / `WithKernelProxy` there. | +| `WithKernelTrustedCerts(pem)` | SEA only | Add a PEM CA bundle on top of the system roots (for a re-signing proxy / on-prem CA). Needed because the kernel's TLS stack does not read `SSL_CERT_FILE`. | +| `WithKernelSkipHostnameVerify()` | SEA only | Skip **only** the hostname check while keeping chain validation (finer-grained than `WithSkipTLSHostVerify`). | + +## Proxy + +The Thrift and kernel backends both honor the standard `HTTP_PROXY` / `HTTPS_PROXY` / +`NO_PROXY` environment variables. + +| Connector option | Protocol | Description | +|---|---|---| +| *(environment)* | Both | `HTTP(S)_PROXY` / `NO_PROXY`. | +| `WithKernelProxy(KernelProxy{URL, Username, Password, BypassHosts})` | SEA only | Explicit proxy with out-of-band basic-auth credentials and a structured bypass list — the "advanced" fields the env-var path can't express. Overrides the environment proxy; a malformed URL is rejected at connect. | + +## Data types + +Results render **byte-for-byte identically** on both backends. Scalars, DECIMAL (exact +string), TIMESTAMP / TIMESTAMP_NTZ (shifted into the session time zone), INTERVAL, nested +ARRAY / MAP / STRUCT and VARIANT (as JSON), and GEOMETRY / GEOGRAPHY (as WKT) are all +supported. BINARY is returned as `sql.RawBytes`. + +Metadata is reached through SQL (`SHOW`, `DESCRIBE`, `information_schema`) on both +backends — the driver exposes no `GetCatalogs`/`GetSchemas`/`GetTables`/`GetColumns` API +(a `database/sql` limitation, not backend-specific). + +## Telemetry + +The driver includes optional telemetry to help improve performance and reliability; it +applies to both backends. When `enableTelemetry` is left unset (the default), a +**server-side feature flag** decides whether telemetry is active — so it may be enabled +without an explicit opt-in. Setting `enableTelemetry` explicitly overrides the flag. +(One exception: on the kernel backend with OAuth **U2M**, telemetry is skipped entirely +— regardless of `enableTelemetry` — to avoid a second interactive browser flow at +connect.) + +``` +# force on (regardless of the server flag): +token:@:443[path]?enableTelemetry=true +# force off: +token:@:443[path]?enableTelemetry=false +``` + +| DSN parameter | Default | Description | +|---|---|---| +| `enableTelemetry` | unset (server flag decides) | Force telemetry on/off, overriding the server feature flag. | +| `telemetry_batch_size` | `200` | Events per batch. | +| `telemetry_flush_interval` | `30s` | Flush interval. | +| `telemetry_retry_count` | — | **Deprecated and ignored** (retries are owned by the HTTP client + circuit breaker); logs a one-time warning. | +| `telemetry_retry_delay` | — | **Deprecated and ignored** (see above). | + +**Collected:** query latency/performance, error codes (not messages), feature usage, +driver version/environment. **Not collected:** SQL text, query results/values, +table/column names, user identities or credentials. Telemetry has < 1% overhead and is +protected by a circuit breaker. The kernel path additionally emits a connection-config +telemetry event at connect (mode, auth mechanism/flow, proxy, arrow, query tags, +metric-view); the Thrift path's telemetry is unchanged. See +[`telemetry/DESIGN.md`](./telemetry/DESIGN.md). + +## Examples + +Runnable examples live in [`examples/`](./examples). Notable ones: + +- [`examples/workflow`](./examples/workflow) — end-to-end connector setup, logging, query. +- [`examples/oauth`](./examples/oauth) — OAuth U2M and M2M. +- [`examples/parameters`](./examples/parameters) — bound query parameters. +- [`examples/cloudfetch`](./examples/cloudfetch) — large results via Cloud Fetch. +- [`examples/kernel`](./examples/kernel) — the SEA/kernel backend (requires a + `-tags databricks_kernel`, `CGO_ENABLED=1` build; see [Building](#building)). ## Develop ### Lint -We use `golangci-lint` as the lint tool. If you use vs code, just add the following settings: -``` json + +We use `golangci-lint`. In VS Code: + +```json { - "go.lintTool": "golangci-lint", - "go.lintFlags": [ - "--fast" - ] + "go.lintTool": "golangci-lint", + "go.lintFlags": ["--fast"] } ``` -### Unit Tests + +### Unit tests ```bash -go test +go test # default (Thrift) backend, pure Go +make test-kernel # kernel-tagged unit tests (requires make kernel-lib) ``` ## Issues @@ -182,7 +415,7 @@ If you find any issues, feel free to create an issue or send a pull request dire ## Contributing -See [CONTRIBUTING.md](CONTRIBUTING.md) +See [CONTRIBUTING.md](CONTRIBUTING.md). ## License diff --git a/connector.go b/connector.go index 21471505..040c3271 100644 --- a/connector.go +++ b/connector.go @@ -384,7 +384,7 @@ func WithWarehouseID(id string) ConnOption { } } -// WithMaxRows sets up the max rows fetched per request. Default is 10000 +// WithMaxRows sets up the max rows fetched per request. Default is 100000 func WithMaxRows(n int) ConnOption { return func(c *config.Config) { if n != 0 { @@ -483,7 +483,7 @@ func WithTransport(t http.RoundTripper) ConnOption { } } -// WithCloudFetch sets up the use of cloud fetch for query execution. Default is false. +// WithCloudFetch sets up the use of cloud fetch for query execution. Default is true. func WithCloudFetch(useCloudFetch bool) ConnOption { return func(c *config.Config) { c.UseCloudFetch = useCloudFetch diff --git a/doc.go b/doc.go index 24a67391..266deff0 100644 --- a/doc.go +++ b/doc.go @@ -228,11 +228,10 @@ blocks until login completes or the kernel's ~120s callback timeout expires. Bec the C ABI can't interrupt session open mid-call, a connection-context deadline is not honored during that window. Use PAT or OAuth M2M for headless / deadline-bound connects. The kernel and Thrift backends use the same (cloud-inferred) U2M client id -but request different default scopes — the kernel applies all-apis + offline_access, -the Thrift path offline_access + sql (or, on Azure, user_impersonation). Neither -backend exposes a U2M-scopes option, so this is a fixed difference between the two -default sets, not a dropped setting; both authorize against the built-in public -client. +and request the same default scopes (offline_access + sql, or, on Azure, +offline_access + /user_impersonation): the kernel path forwards exactly the +scopes the Thrift path computes, so both authorize identically against the built-in +public client. Neither backend exposes a U2M-scopes option. Experimental kernel-only options (rejected by the default backend; the WithKernel* prefix marks them experimental): diff --git a/examples/kernel/main.go b/examples/kernel/main.go new file mode 100644 index 00000000..51d99c31 --- /dev/null +++ b/examples/kernel/main.go @@ -0,0 +1,100 @@ +// Command kernel demonstrates the experimental SEA-via-kernel backend. +// +// The kernel backend is opt-in and compiled in only under the databricks_kernel +// build tag with CGO enabled, and it links the Rust kernel static library. Build +// the library first, then run this example with the tag: +// +// make -C ../.. kernel-lib # build the pinned kernel .a + header +// CGO_ENABLED=1 go run -tags databricks_kernel . +// +// Without the tag, the program still compiles (it uses only the stable public +// API), but WithUseKernel(true) returns an error wrapping +// dbsqlerr.ErrKernelNotCompiled at connect — which this example detects and +// reports rather than treating as a hard failure. +// +// Required environment (a .env file in this directory is loaded automatically): +// +// DATABRICKS_HOST workspace hostname (no scheme) +// DATABRICKS_ACCESSTOKEN personal access token +// DATABRICKS_HTTPPATH warehouse http path (or set DATABRICKS_WAREHOUSEID) +// DATABRICKS_WAREHOUSEID bare warehouse id (kernel routes by this when set) +package main + +import ( + "context" + "database/sql" + "errors" + "fmt" + "log" + "os" + "time" + + dbsql "github.com/databricks/databricks-sql-go" + dbsqlerr "github.com/databricks/databricks-sql-go/errors" + "github.com/joho/godotenv" +) + +func main() { + _ = godotenv.Load() + + host := os.Getenv("DATABRICKS_HOST") + token := os.Getenv("DATABRICKS_ACCESSTOKEN") + httpPath := os.Getenv("DATABRICKS_HTTPPATH") + warehouseID := os.Getenv("DATABRICKS_WAREHOUSEID") + if host == "" || token == "" || (httpPath == "" && warehouseID == "") { + log.Fatal("set DATABRICKS_HOST, DATABRICKS_ACCESSTOKEN, and DATABRICKS_HTTPPATH (or DATABRICKS_WAREHOUSEID)") + } + + opts := []dbsql.ConnOption{ + dbsql.WithServerHostname(host), + dbsql.WithAccessToken(token), + // Select the SEA-via-kernel backend (same as the useKernel=true DSN param). + dbsql.WithUseKernel(true), + } + if httpPath != "" { + opts = append(opts, dbsql.WithHTTPPath(httpPath)) + } + // The kernel routes by bare warehouse id (preferred over the http path); + // the Thrift backend ignores this option. + if warehouseID != "" { + opts = append(opts, dbsql.WithWarehouseID(warehouseID)) + } + + connector, err := dbsql.NewConnector(opts...) + if err != nil { + log.Fatalf("NewConnector: %v", err) + } + + db := sql.OpenDB(connector) + defer db.Close() //nolint:errcheck + + ctx, cancel := context.WithTimeout(context.Background(), 120*time.Second) + defer cancel() + + if err := db.PingContext(ctx); err != nil { + // The kernel backend isn't linked into a build without the tag; detect + // that case with errors.Is (a caller could fall back to Thrift here). + if errors.Is(err, dbsqlerr.ErrKernelNotCompiled) { + log.Fatal("kernel backend not compiled in — rebuild with `make kernel-lib` " + + "then `CGO_ENABLED=1 go run -tags databricks_kernel .`") + } + log.Fatalf("ping: %v", err) + } + fmt.Println("connected via the SEA-via-kernel backend") + + var version string + if err := db.QueryRowContext(ctx, "SELECT current_version()").Scan(&version); err != nil { + log.Fatalf("query: %v", err) + } + fmt.Printf("server version: %s\n", version) + + // Options the kernel can't honor are rejected with an error wrapping + // ErrNotSupportedByKernel, never silently ignored. Staging (PUT/GET/REMOVE on a + // Unity Catalog volume) is one such feature, rejected at execute. + _, err = db.ExecContext(ctx, "PUT '/tmp/local.csv' INTO '/Volumes/main/default/vol/f.csv' OVERWRITE") + if errors.Is(err, dbsqlerr.ErrNotSupportedByKernel) { + fmt.Println("staging is not supported on the kernel backend (use Thrift for staging)") + } else if err != nil { + fmt.Printf("staging: %v\n", err) + } +} diff --git a/internal/config/config.go b/internal/config/config.go index 00d159de..6434eb15 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -231,8 +231,8 @@ type UserConfig struct { // Uses config overlay pattern: client > server > default. // Unset = check server feature flag; explicitly true/false overrides the server. EnableTelemetry ConfigValue[bool] - TelemetryBatchSize int // 0 = use default (100) - TelemetryFlushInterval time.Duration // 0 = use default (5s) + TelemetryBatchSize int // 0 = use default (200) + TelemetryFlushInterval time.Duration // 0 = use default (30s) Transport http.RoundTripper UseLz4Compression bool EnableMetricViewMetadata bool