[feature](timestamp_ns) Add end-to-end TIMESTAMP_NS support - #66761
Open
jacktengg wants to merge 29 commits into
Open
[feature](timestamp_ns) Add end-to-end TIMESTAMP_NS support#66761jacktengg wants to merge 29 commits into
jacktengg wants to merge 29 commits into
Conversation
jacktengg
requested review from
924060929,
CalvinKirs,
airborne12,
csun5285,
eldenmoon,
englefly,
gavinchou,
liaoxin01,
luwei16,
morrySnow,
mymeiyi,
seawinde,
starocean999 and
yiguolei
as code owners
August 14, 2026 05:44
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
jacktengg
force-pushed
the
timestamp_ns-split-commit1-cast-and-functions
branch
from
August 14, 2026 17:03
a75a76c to
f7b433d
Compare
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 17603 ms |
Contributor
TPC-DS: Total hot run time: 84745 ms |
Contributor
ClickBench: Total hot run time: 14.56 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Issue Number: None Related PR: apache#66761 Problem Summary: TIMESTAMP_NS had basic Arrow round-trip coverage but lacked complete time-unit boundary checks, nested FE schema checks, and an end-to-end Arrow Flight SQL regression. Add exact conversion and overflow tests, nested schema assertions, and Flight queries for nanosecond boundary values under multiple session time zones. None - Test: Unit Test and Regression test - `./run-fe-ut.sh --run org.apache.doris.service.arrowflight.FlightSqlSchemaHelperArrowTypeTest` - `./run-be-ut.sh --run --filter=DataTypeSerDeArrowTest.TimeStampNsArrowRoundTrip:DataTypeSerDeArrowTest.TimeStampNsArrowUnitConversion -j 16` - `./run-regression-test.sh --run -d arrow_flight_sql_p0 -s test_select` - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 16931 ms |
Contributor
TPC-DS: Total hot run time: 84247 ms |
Contributor
ClickBench: Total hot run time: 14.94 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: TIMESTAMP_NS delete-condition validation duplicated the date-time validation path with a type-specific parser. The shared validator was limited to six fractional digits, so it could not be reused for the fixed nine-digit TIMESTAMP_NS scale. Generalize fractional-second validation through scale 9 and route TIMESTAMP_NS through the same validation branch as DATETIMEV2 while preserving the DATETIMEV2 six-digit limit.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- ./run-be-ut.sh --run --filter=DeleteConditionValueTest.ValidDatetimeSupportsNanosecondScale -j 32
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: TIMESTAMP_NS index tests did not prove that default V3 inverted indexes and synchronous rollup indexes were selected, and lacked NULL-path coverage for ZoneMap, Bloom Filter, V2 inverted indexes, and SNII. Add BE writer/reader checks for NULL metadata and bitmaps, add end-to-end profile counter assertions for V3, SNII, explicit Bloom Filter, and rollup rewrite, and fix unsupported NGRAM_BF and ANN behavior with negative DDL tests.
### Release note
None
### Check List (For Author)
- Test:
- Unit Test: targeted TIMESTAMP_NS Bloom Filter, V2 inverted index, and ZoneMap BE tests
- Regression test: test_timestamp_ns_index and test_timestamp_ns_mv
- Static checks: check-format, check-build-hygiene, and git diff --check
- Clang-tidy: attempted, but the local GCC 15 toolchain could not resolve stddef.h while parsing the test translation units
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: TIMESTAMP_NS numeric casts converted rows directly in the cast implementation instead of using the strict and non-strict SerDe batch paths used by DATETIMEV2. TIMESTAMP_NS JSONB and textual number serialization also bypassed CastToString helpers, and no SQL regression directly covered legacy JSONB write/read round trips. Add the missing integer, floating-point, and decimal batch SerDe interfaces for both cast modes, route numeric casts through them, centralize string conversion, and add unit and regression coverage.
### Release note
None
### Check List (For Author)
- Test: Regression test / Unit Test / Build
- `./run-be-ut.sh --run --filter=DatelikeSerDeBatchTest.timestamp_ns_*:DataTypeTimeStampNsTest.SerDeStrictBatchJsonJsonbMysqlAndBinaryField -j 24`
- `./run-regression-test.sh --run -d datatype_p0/timestamp_ns -s test_timestamp_ns_cast`
- `./run-regression-test.sh --run -d datatype_p0/timestamp_ns -s test_timestamp_ns_jsonb`
- `./build.sh --be -j 24`
- Behavior changed: No; the conversion is routed through shared helpers and batch SerDe interfaces with equivalent strict/non-strict semantics.
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: TIMESTAMP_NS predicate creator branches for comparison, IN, NOT IN, and runtime Bloom Filter lacked complete direct unit-test coverage. Add focused tests that instantiate each factory and verify the generated predicate type and filtering behavior.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- ./run-be-ut.sh --run --filter=PredicateCreatorTest.*:ScanNormalizePredicate.TimestampNsInPredicateBuildsPointScanKeys -j 8
- Behavior changed: No
- Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: TimeStampNsValue exposed unused arithmetic, date-difference, and seeded-hash helpers while several retained calendar, difference, formatting, interval, and truncation interfaces lacked direct unit-test coverage. Remove the unused public interfaces, keep the internally required time-difference helper private, and add boundary-focused tests for every retained behavior.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- ./run-be-ut.sh --run --filter='DataTypeTimeStampNsTest.*' -j 8
- build-support/check-build-hygiene.sh
- build-support/check-format.sh with clang-format 16
- build-support/run-clang-tidy.sh --build-dir be/ut_build_ASAN with clang-tidy 16
- Behavior changed: No; unused C++ interfaces were removed and retained behavior was not changed
- Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 17103 ms |
Contributor
TPC-DS: Total hot run time: 83651 ms |
Contributor
ClickBench: Total hot run time: 14.82 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Docs: apache/doris-website#4091
Problem Summary:
Doris DATETIME only support at most six fractional digits, it's not sufficient for event times, traces, etc which require higher precision of sub-seconds.
Introduce
TIMESTAMP_NSas a separate fixed-scale SQL type backed by signed Int64 Unix-epoch nanoseconds. It always has scale 9 and supports values from1677-09-21 00:12:43.145224192through2262-04-11 23:47:16.854775807. Wire the type through Thrift and protobuf metadata, the FE catalog and Nereids, BE columns and SerDe, storage and indexes, result protocols, JNI and Java UDF transport, complex and VARIANT types, and schema metadata.Add exact casts, comparisons, type inference, and constant folding across date-like types while preserving nanoseconds and rejecting partial or precision-losing implicit conversions. Add timestamp-aware scalar and calendar functions, aggregation, joins and ASOF joins, scan predicates, runtime filters, partition pruning, query-cache normalization, automatic partitions, and MTMV roll-up. Keep FE folding and BE runtime behavior consistent at signed endpoints, leap years, DST gaps, calendar-clamped boundaries, and fractional formatting and rounding boundaries.
Keep existing DATETIME and DATETIMEV2 scale semantics unchanged. FROM_UNIXTIME retains a DECIMAL64 microsecond path for scale 0 through 6 and uses the DECIMAL128 nanosecond path only for scale 7 through 9. The %f formatter rounds half-up to microseconds with second carry, while %n preserves the nanosecond fraction.
Release note
Add
TIMESTAMP_NS, a fixed-scale nanosecond timestamp type stored as signed Unix-epoch nanoseconds. It supports SQL literals, casts, date/time functions, storage and indexes, partitions, joins, aggregates, complex types, Java UDF/JNI transport, and FE constant folding and type inference. Existing DATETIME and DATETIMEV2 types remain limited to scale 0 through 6.Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)