|
1 | | -# stackable-odbc-sqlite |
| 1 | +<!-- markdownlint-disable MD041 MD033 --> |
2 | 2 |
|
3 | | -ODBC 3.x driver for [SQLite](https://sqlite.org), built on the |
4 | | -[stackable-odbc-core](https://github.com/stackabletech/stackable-odbc-core) |
5 | | -framework. |
| 3 | +<p align="center"> |
| 4 | + <img width="150" src="./.readme/static/borrowed/Icon_Stackable.svg" alt="Stackable Logo"/> |
| 5 | +</p> |
6 | 6 |
|
7 | | -The driver compiles to a C dynamic library that an ODBC Driver Manager |
8 | | -(unixODBC on Linux, the built-in Driver Manager on Windows) loads at runtime. |
9 | | -It opens a local SQLite database file through `rusqlite` with the bundled |
10 | | -SQLite library, so it needs no server and no external SQLite installation. |
| 7 | +<h1 align="center">Stackable ODBC Driver for SQLite</h1> |
11 | 8 |
|
12 | | -## Requirements |
| 9 | +<p align="center"><em>Open a SQLite file from Excel, DBeaver, LibreOffice or Python, with no server to run.</em></p> |
13 | 10 |
|
14 | | -- Rust 1.95.0+ (pinned in `rust-toolchain.toml`) |
15 | | -- Linux: `unixODBC` and `isql` (`pacman -S unixodbc-dev` / `apt install unixodbc-dev`) |
16 | | -- `sqlite3` CLI for creating the test database (`pacman -S sqlite` / `apt install sqlite3`) |
| 11 | +[](https://github.com/stackabletech/stackable-odbc-sqlite/actions/workflows/build.yaml) |
| 12 | +[](https://github.com/stackabletech/stackable-odbc-sqlite/actions/workflows/security_audit.yaml) |
| 13 | +[](https://docs.stackable.tech/home/stable/contributor/index.html) |
| 14 | +[](./LICENSE) |
| 15 | +[](#what-it-deliberately-does-not-do) |
| 16 | +[](#quick-start) |
| 17 | +[](https://sqlite.org) |
17 | 18 |
|
18 | | -## Building |
| 19 | +[Stackable Data Platform](https://stackable.tech/) | [Platform Docs](https://docs.stackable.tech/) | [Discussions](https://github.com/orgs/stackabletech/discussions) | [Discord](https://discord.gg/7kZ3BNnCAF) |
| 20 | + |
| 21 | +## What is this? |
| 22 | + |
| 23 | +[SQLite](https://sqlite.org) is a database that lives in a single file. There |
| 24 | +is nothing to install and nothing to start: the whole database is one `.db` |
| 25 | +file you can copy onto a USB stick. Your phone is running several of them right |
| 26 | +now. |
| 27 | + |
| 28 | +Most desktop tools cannot open one of those files directly, but nearly all of |
| 29 | +them speak **ODBC**. ODBC is a widely supported standard: a tool loads a small library called a *driver*, calls a fixed set of functions on it, and the driver translates those calls into whatever the actual database understands. Write one driver, and every ODBC-speaking tool on the machine can talk to that database. |
| 30 | + |
| 31 | +This repository is that driver for SQLite. Install it, and Excel, LibreOffice |
| 32 | +Base, DBeaver, `isql` and Python's `pyodbc` can query a SQLite file as if it |
| 33 | +were a full database server. Linux and Windows are both supported. |
| 34 | + |
| 35 | +Two things make it unusual: |
| 36 | + |
| 37 | +- **It carries its own SQLite.** Version 3.53.2 is compiled straight into the |
| 38 | + driver, so there is no separate SQLite to install and no version of it on the |
| 39 | + machine that could disagree with the one the driver actually uses. |
| 40 | +- **It is a testbed.** Everything generic about being an ODBC driver lives in |
| 41 | + [`stackable-odbc-core`](https://github.com/stackabletech/stackable-odbc-core), |
| 42 | + which also powers the |
| 43 | + [Trino driver](https://github.com/stackabletech/stackable-odbc-trino). SQLite |
| 44 | + is small, fast and needs no server, which makes it the ideal backend for |
| 45 | + proving that shared framework behaves. |
| 46 | + |
| 47 | +## Quick start |
| 48 | + |
| 49 | +No release has been cut yet, so build the driver yourself. You need Rust (the |
| 50 | +version in `rust-toolchain.toml` is installed automatically by `rustup`) and |
| 51 | +the unixODBC development headers, because the ODBC bindings link against them: |
| 52 | + |
| 53 | +```bash |
| 54 | +sudo apt-get install unixodbc-dev # Debian/Ubuntu |
| 55 | +sudo pacman -S unixodbc # Arch |
| 56 | +``` |
| 57 | + |
| 58 | +Clone this repository: |
19 | 59 |
|
20 | 60 | ```bash |
21 | | -cargo build |
| 61 | +git clone https://github.com/stackabletech/stackable-odbc-sqlite |
| 62 | +cd stackable-odbc-sqlite |
| 63 | +cargo build --release |
22 | 64 | ``` |
23 | 65 |
|
24 | | -Linux output: `target/debug/libstackable_odbc_sqlite.so`. |
| 66 | +Output: `target/release/libstackable_odbc_sqlite.so`. |
25 | 67 |
|
26 | | -## Connection string parameters |
| 68 | +For Windows, cross-compile with MinGW (`gcc-mingw-w64-x86-64`): |
27 | 69 |
|
28 | | -| Parameter | Required | Default | Description | |
29 | | -|-----------|----------|---------|-------------| |
30 | | -| Database | Yes | -- | Path to the SQLite database file (`:memory:` for an in-memory database) | |
| 70 | +```bash |
| 71 | +rustup target add x86_64-pc-windows-gnu |
| 72 | +cargo build --release --target x86_64-pc-windows-gnu |
| 73 | +``` |
31 | 74 |
|
32 | | -## Testing |
| 75 | +Output: `target/x86_64-pc-windows-gnu/release/stackable_odbc_sqlite.dll`. |
33 | 76 |
|
34 | | -Run all commands from the repository root. |
| 77 | +### Installing it |
| 78 | + |
| 79 | +`packaging/build-archives.sh` turns those binaries into the same release |
| 80 | +archives CI publishes, each with an installer inside: |
35 | 81 |
|
36 | 82 | ```bash |
37 | | -# Build the driver, create the test database, write the ODBC config |
38 | | -./test/setup.sh |
| 83 | +VERSION=0.0.1 ./packaging/build-archives.sh |
| 84 | +``` |
| 85 | + |
| 86 | +On Linux, unpack `stackable-odbc-sqlite-<version>-linux-x64.tar.gz` and run |
| 87 | +`sudo ./install.sh`. It copies the library into place and registers it with |
| 88 | +unixODBC; check it worked with `odbcinst -q -d`, which should list |
| 89 | +`[stackable_odbc_sqlite]`. |
| 90 | + |
| 91 | +On Windows, unpack the `.zip` and run `install.bat` from an Administrator |
| 92 | +Command Prompt, then look for `stackable_odbc_sqlite` on the Drivers tab of |
| 93 | +**ODBC Data Sources (64-bit)**. |
| 94 | + |
| 95 | +The full install, uninstall and DSN reference is in |
| 96 | +[`packaging/README.md`](packaging/README.md). |
39 | 97 |
|
40 | | -# Connect interactively |
41 | | -export ODBCSYSINI=$(pwd)/test |
42 | | -export ODBCINI=$(pwd)/test/odbc.ini |
43 | | -isql -3 test_sqlite -v |
| 98 | +### Then use it |
| 99 | + |
| 100 | +```python |
| 101 | +import pyodbc |
| 102 | + |
| 103 | +conn = pyodbc.connect("Driver=stackable_odbc_sqlite;Database=/path/to/your.db") |
| 104 | +for row in conn.cursor().execute("SELECT name FROM sqlite_master WHERE type = 'table'"): |
| 105 | + print(row.name) |
44 | 106 | ``` |
45 | 107 |
|
46 | | -Or with a DSN-less connection string: |
| 108 | +Or straight from a source checkout, without installing anything at all: |
47 | 109 |
|
48 | 110 | ```bash |
49 | | -isql -3 -k "Driver=$(pwd)/target/debug/libstackable_odbc_sqlite.so;Database=$(pwd)/test/test.db" -v |
| 111 | +isql -3 -k "Driver=$(pwd)/target/release/libstackable_odbc_sqlite.so;Database=$(pwd)/test/test.db" -v |
| 112 | +``` |
| 113 | + |
| 114 | +## Highlights |
| 115 | + |
| 116 | +- **The stop button actually stops the query.** Cancelling from your tool calls |
| 117 | + SQLite's `sqlite3_interrupt` on the connection, so a runaway query really |
| 118 | + stops instead of quietly running to the end while your tool pretends it was |
| 119 | + cancelled. The statement reports "operation canceled" and can be re-run. |
| 120 | + |
| 121 | +- **Real transactions.** Turn autocommit off and the driver opens a transaction |
| 122 | + for you, then commits or rolls back when you say so and immediately opens the |
| 123 | + next one. Your open result sets survive both, because the driver has already |
| 124 | + read every row into memory by the time you commit. |
| 125 | + |
| 126 | +- **Foreign keys are switched on.** SQLite ships with foreign-key enforcement |
| 127 | + *off* for backwards compatibility, which surprises almost everyone. This |
| 128 | + driver turns it on for every connection, so a `REFERENCES` clause in your |
| 129 | + schema is a rule the database enforces rather than a comment. |
| 130 | + |
| 131 | +- **Your tool can browse the database.** Tables, views, columns, primary keys, |
| 132 | + foreign keys, indexes and row identifiers all show up in the object browser, |
| 133 | + read out of SQLite's own `PRAGMA` introspection. So you can click through what |
| 134 | + is there instead of guessing table names. |
| 135 | + |
| 136 | +- **Columns get sensible types even though SQLite has almost none.** SQLite is |
| 137 | + dynamically typed: any value can go in any column, and there is no `DATE` or |
| 138 | + `BOOLEAN` type at all. The driver reads each column's declared type and its |
| 139 | + actual storage class and maps them onto proper ODBC types, including the |
| 140 | + three different ways SQLite people store a timestamp (ISO text, Unix seconds, |
| 141 | + Julian day numbers). |
| 142 | + |
| 143 | +- **Nothing is claimed that was not measured.** What a driver reports about |
| 144 | + itself is how tools decide which SQL to send, so guessing wrong there breaks |
| 145 | + things in confusing ways. The tests here run the actual SQL to check: the list |
| 146 | + of `ALTER TABLE` clauses is verified by executing each one, and the list of |
| 147 | + reserved words is read out of the linked SQLite library at runtime instead of |
| 148 | + being copied from documentation that can drift. |
| 149 | + |
| 150 | +- **Windows is a real target, not an afterthought.** It gets its own installer, |
| 151 | + the DLL is cross-compiled and export-checked on every pull request, and the |
| 152 | + test suite can be run through the Windows Driver Manager in a VM, which is far |
| 153 | + stricter than unixODBC and tends to fail silently rather than loudly. |
| 154 | + |
| 155 | +## Connecting |
| 156 | + |
| 157 | +Connection strings are `Key=Value` pairs joined by `;`. Keys are |
| 158 | +case-insensitive. There is exactly one key: |
| 159 | + |
| 160 | +| Key | Required | Meaning | |
| 161 | +|-----|----------|---------| |
| 162 | +| `Database` | Yes | Path to the SQLite file, or `:memory:` for a throwaway in-memory database | |
| 163 | + |
| 164 | +```text |
| 165 | +Driver=stackable_odbc_sqlite;Database=/path/to/your.db |
| 166 | +``` |
| 167 | + |
| 168 | +Instead of typing that every time you can save it as a **DSN**, which is just a |
| 169 | +named, stored connection, like a browser bookmark. On Linux, add a section to |
| 170 | +`~/.odbc.ini`: |
| 171 | + |
| 172 | +```ini |
| 173 | +[SQLite Test] |
| 174 | +Driver = stackable_odbc_sqlite |
| 175 | +Database = /path/to/your.db |
50 | 176 | ``` |
51 | 177 |
|
52 | | -The test database (`test/test.db`) has a `types_test` table with integer, text, |
53 | | -real, boolean, blob, and text-based datetime columns (see |
54 | | -`test/create_test_db.sql`). The full integration suite runs via |
55 | | -`./test/run-tests.sh` (add `--windows` for the VM suite); see |
56 | | -[AGENTS.md](AGENTS.md#testing) for the complete matrix. |
| 178 | +On Windows, see [`packaging/README.md`](packaging/README.md). |
57 | 179 |
|
58 | 180 | ### Logging |
59 | 181 |
|
| 182 | +Two environment variables turn on tracing, which is by far the fastest way to |
| 183 | +see which ODBC functions your tool actually calls, and in what order: |
| 184 | + |
60 | 185 | ```bash |
61 | | -# Log to stderr at debug level |
| 186 | +# Levels: trace, debug, info, warn, error |
62 | 187 | ODBC_LOG_LEVEL=debug isql -3 test_sqlite -v |
63 | 188 |
|
64 | | -# Log to a file (levels: trace, debug, info, warn, error) |
| 189 | +# Or send it to a file instead of stderr |
65 | 190 | ODBC_LOG_LEVEL=debug ODBC_LOG_FILE=/tmp/odbc.log isql -3 test_sqlite -v |
66 | 191 | ``` |
67 | 192 |
|
68 | | -This is invaluable for seeing which ODBC functions are called, and in what order. |
| 193 | +## What it deliberately does not do |
| 194 | + |
| 195 | +Every one of these is reported to the application as unsupported rather than |
| 196 | +quietly faked, so a tool can react to it instead of trusting a wrong answer. |
| 197 | + |
| 198 | +- **No catalogs and no schemas.** SQLite has neither, so the driver says so |
| 199 | + rather than inventing a fake one-level hierarchy for the sake of looking |
| 200 | + familiar. |
| 201 | +- **No stored procedures.** SQLite has none, so those lookups return nothing. |
| 202 | +- **No query timeout.** You can cancel a running statement from another thread, |
| 203 | + but asking for "give up after 30 seconds" is answered with "you have no |
| 204 | + timeout" and a warning, instead of a promise that would never be kept. |
| 205 | +- **Result sets are read into memory in one go.** Simple, and it is what makes |
| 206 | + cursors survive a commit or rollback, but a `SELECT` over a table larger than |
| 207 | + your RAM is not going to work. |
| 208 | +- **One isolation level.** SQLite gives you serializable transactions, so that |
| 209 | + is the only level offered, and asking for a weaker one is refused up front |
| 210 | + rather than accepted and silently ignored. |
| 211 | +- **No setup dialog.** The driver has no GUI, so the **Add** button in Windows' |
| 212 | + ODBC administrator stores whatever it was handed without prompting you for a |
| 213 | + database path. Create DSNs with `odbcconf` or by editing `odbc.ini` instead. |
| 214 | + |
| 215 | +## Testing |
| 216 | + |
| 217 | +```bash |
| 218 | +cargo test # unit and FFI tests; needs no database file and no setup |
| 219 | +cargo bench # Criterion fetch-throughput benchmark against :memory: |
| 220 | +``` |
| 221 | + |
| 222 | +`cargo test` drives the real exported C entry points against real handles, so |
| 223 | +it catches the marshalling bugs that ordinary Rust tests cannot. |
| 224 | + |
| 225 | +The integration suite goes one layer further out and runs through real |
| 226 | +unixODBC, using Python's `pyodbc` exactly like a normal application would: |
| 227 | + |
| 228 | +```bash |
| 229 | +./test/setup.sh # build the driver, create test/test.db, write the ODBC config |
| 230 | +./test/run-tests.sh # run the pyodbc suite, then cargo test |
| 231 | +``` |
| 232 | + |
| 233 | +Both are run on every pull request. `./test/run-tests.sh --windows` additionally |
| 234 | +runs the same suite inside a Windows VM; see |
| 235 | +[windows/WINDOWS.md](windows/WINDOWS.md) for how to provision one. |
| 236 | + |
| 237 | +For the architecture, the conventions and the full testing reference, see |
| 238 | +[AGENTS.md](AGENTS.md). |
69 | 239 |
|
70 | 240 | ## Releasing |
71 | 241 |
|
72 | | -See [packaging/README.md](packaging/README.md) for building release archives, |
73 | | -and `release.toml` for the `cargo-release` configuration. |
| 242 | +See [packaging/README.md](packaging/README.md) for building the release |
| 243 | +archives, and `release.toml` for the `cargo-release` configuration. |
74 | 244 |
|
75 | 245 | ## License |
76 | 246 |
|
|
0 commit comments