Skip to content

Support ADO.NET schema data types and metadata probing - #364

Merged
Giorgi merged 6 commits into
Giorgi:developfrom
pengdows:draft/ado-schema-metadata
Oct 20, 2026
Merged

Giorgi merged 6 commits into
Giorgi:developfrom
pengdows:draft/ado-schema-metadata

Conversation

@alaricd

@alaricd alaricd commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add standard DataSourceInformation metadata needed by ADO.NET schema probing.
  • Add DataTypes metadata for DuckDB native types.
  • Align metadata column ordering, types, normalized versions, DBNull values, and restriction validation with the common ADO.NET contract.
  • Correct index restriction metadata.
  • Add focused schema coverage.

Validation

  • dotnet test DuckDB.NET.Test/Test.csproj --no-restore --filter FullyQualifiedName~SchemaTests -m:1
  • 23 tests passed.

@codecov

codecov Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.09091% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 87.17%. Comparing base (e28f086) to head (513a2ef).
⚠️ Report is 122 commits behind head on develop.

Files with missing lines Patch % Lines
DuckDB.NET.Data/DuckDBSchema.cs 99.09% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #364      +/-   ##
===========================================
+ Coverage    86.79%   87.17%   +0.37%     
===========================================
  Files           82       82              
  Lines         3560     3665     +105     
  Branches       553      557       +4     
===========================================
+ Hits          3090     3195     +105     
+ Misses         331      330       -1     
- Partials       139      140       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coveralls

coveralls commented Oct 14, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 39458161086

Coverage increased (+0.2%) to 92.755%

Details

  • Coverage increased (+0.2%) from the base build.
  • Patch coverage: 3 uncovered changes across 1 file (180 of 183 lines covered, 98.36%).
  • No coverage regressions found.

Uncovered Changes

File Changed Covered %
DuckDB.NET.Data/DuckDBSchema.cs 183 180 98.36%

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 4196
Covered Lines: 3956
Line Coverage: 94.28%
Relevant Branches: 2498
Covered Branches: 2253
Branch Coverage: 90.19%
Branches in Coverage %: Yes
Coverage Strength: 957062.4 hits per line

💛 - Coveralls

@Giorgi
Giorgi changed the base branch from main to develop October 15, 2026 07:42
@Giorgi

Giorgi commented Oct 15, 2026

Copy link
Copy Markdown
Owner

Is it ready for review?

@alaricd

alaricd commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

The metadata probing changes are ready for review from my side. The implementation and tests are in place, with no unresolved review threads. I noticed the branch is now behind develop; I can rebase it if you would prefer a fully current review base.

@Giorgi

Giorgi commented Oct 20, 2026

Copy link
Copy Markdown
Owner

Thanks. DataSourceInformation, the restriction checks and the Indexes restriction fix look good. Two things before this can go in: a rebase, and some fixes to DataTypes.

Rebase: DuckDBType has new values on develop

develop now has the types DuckDB 1.5 defines (e4ec50f): StringLiteral = 37, IntegerLiteral = 38, TimeNs = 39, Geometry = 40 and Variant = 41. After merging develop into this branch, DataTypesExposeEveryDuckDbType fails with Missing: StringLiteral, IntegerLiteral, TimeNs, Geometry, Variant.

This also exposes a mapping gap that the test currently hides by excluding VarInt and SqlNull. In DuckDB 1.5, duckdb_types() reports type 35 as BIGNUM (VARINT was renamed in 1.4) and SQLNULL as NULL, so ParseDuckDBType maps the bignum, varint and null rows to Invalid. I tried this change on top of develop, and SchemaTests passes 24/24:

"VARINT" or "BIGNUM" => DuckDBType.VarInt,
// ...
"SQLNULL" or "NULL" => DuckDBType.SqlNull,
"TIME_NS" => DuckDBType.TimeNs,
"GEOMETRY" => DuckDBType.Geometry,
"VARIANT" => DuckDBType.Variant,

and in the test, exclude the literal types, which duckdb_types() never lists, instead of VarInt and SqlNull:

.Where(type => type is not (DuckDBType.Invalid or DuckDBType.Any or DuckDBType.StringLiteral or DuckDBType.IntegerLiteral))

The only row then left with ProviderDbType 0 is type, which has no type in the C API.

DataType disagrees with what the reader returns

I compared DataTypes.DataType with reader.GetFieldType() / GetValue() for each type:

Type DataTypes Reader
BLOB System.Byte[] System.IO.Stream
TIMESTAMP WITH TIME ZONE System.DateTimeOffset System.DateTime
BIGNUM System.Object System.Numerics.BigInteger

The other types I checked match. GetClrType is a second, hand-written type map, and it has already drifted from the reader's. Could it use the reader's mapping, or at least be fixed for these three? (For timestamp with time zone, Npgsql reports System.DateTime as well.)

CreateFormat is empty for almost every type

Only dec, decimal and numeric have a CreateFormat. Tools that generate DDL from DataTypes read this column. Npgsql fills it for all of its types (e.g. INTEGER, NUMERIC({0},{1})). The type name would do for every type without parameters.

Duplicate rows that differ only in case

duckdb_types() lists both spellings of some names, and SELECT DISTINCT type_name is case-sensitive, so INTEGER/integer and VARCHAR/varchar each appear twice, both marked IsBestMatch. Could these be deduplicated case-insensitively, keeping the canonical name?

Nit: there is a stray blank line before the closing } of ValidateRestrictions.

@alaricd
alaricd force-pushed the draft/ado-schema-metadata branch from 69f1fb9 to d20770c Compare October 20, 2026 13:52
@alaricd

alaricd commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Rebased and pushed the branch onto current develop, then addressed the DataTypes feedback in d20770c:\n\n- Added DuckDB 1.5 logical-type parsing for BIGNUM/VARINT, NULL/SQLNULL, TIME_NS, GEOMETRY, and VARIANT.\n- Updated DataTypes.DataType to match the reader for BLOB (Stream), TIMESTAMP WITH TIME ZONE (DateTime), and BIGNUM/VARINT (BigInteger).\n- Populated CreateFormat for every type; DECIMAL retains its precision/scale format.\n- Deduplicated type names case-insensitively while retaining the canonical spelling.\n- Updated DataTypes coverage to include VarInt and SqlNull and exclude only the literal types that duckdb_types() does not list.\n- Removed the stray blank line.\n\nSchemaTests passes 24/24 against DuckDB 1.5.6.

@alaricd

alaricd commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Also pushed the final formatting cleanup as 08068b9. A fresh CodeQL run is now in progress for that SHA: run 39444031577.

@Giorgi

Giorgi commented Oct 20, 2026

Copy link
Copy Markdown
Owner

Thanks, I checked the update in a clone: the mappings, DataType against the reader (20 types, no mismatches), the deduplication and CreateFormat all look right, and SchemaTests passes.

One remaining problem: the spelling of TypeName depends on what the database contains. duckdb_types() returns a row per catalog (system, memory, temp), and the built-in names there are lower-case (integer, decimal, timestamp with time zone). Upper-case rows such as VARCHAR only show up in the temp catalog, and only for some types. Because the deduplication keeps the row whose type_name equals logical_type ordinally, a fresh in-memory connection gets VARCHAR but integer, decimal, list, timestamp with time zone, …

DataTypesExposeStandardAdoNetMetadata (Single(row => (string)row["TypeName"] == "INTEGER"), and CreateFormat == "INTEGER") passes only because the test fixture's database has an upper-case INTEGER row; on a fresh connection there is only integer and the Single throws.

Could you output logical_type as the TypeName whenever type_name equals it ignoring case? Built-in types then always come out as INTEGER, DECIMAL, TIMESTAMP WITH TIME ZONE, …, and aliases (int, string, numeric) and user types keep their own names. A test that reads DataTypes from a new DuckDBConnection rather than the fixture would pin it.

Minor, not blocking: for types that need parameters, CreateFormat is the bare name (list, struct, map, array, union, enum, and also null and type), which isn't valid DDL on its own. DBNull there would tell tools that they can't create the type from this row.

@alaricd

alaricd commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the remaining TypeName/CreateFormat feedback in 4de083f:\n\n- When a catalog name matches logical_type case-insensitively, TypeName now uses the canonical logical type spelling (for example INTEGER, DECIMAL, and TIMESTAMP WITH TIME ZONE). Aliases and user-defined names remain unchanged.\n- Added a fresh in-memory connection regression test so the result does not depend on the fixture catalog.\n- CreateFormat is now DBNull for parameterized/non-creatable composite and sentinel types (LIST, STRUCT, MAP, ARRAY, UNION, ENUM, NULL, and type); scalar types retain a usable format.\n\nSchemaTests passes 25/25 against DuckDB 1.5.6.

@Giorgi
Giorgi merged commit 034e262 into Giorgi:develop Oct 20, 2026
11 of 12 checks passed

This branch is waiting to be deployed

1 waiting deployment
external — 4de083f1 Waiting Oct 20, 2026 by alaricd via Authorize #703
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants