Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 123 additions & 8 deletions docs/source/user_guide/data_types.rst
Original file line number Diff line number Diff line change
Expand Up @@ -160,14 +160,129 @@ and `Arrow DataTypes <https://arrow.apache.org/docs/format/Columnar.html#data-ty
stored as milliseconds of the day whatever p is, so ``TIME(6)`` and
``TIME(9)`` values keep only millisecond precision.

Paimon C++ reads TIME columns from Parquet, Lance and Vortex data files
and writes them to Lance and Vortex data files. ORC and Avro data files
do not support TIME.

**Note:** Only append-only tables support TIME columns. Primary-key
tables do not support TIME, whether as a primary key or as a value
column. A TIME column also cannot be a partition or bucket key, and
predicates cannot reference TIME columns.
Both append-only and primary-key tables support TIME columns, as value
columns, primary keys, partition keys and bucket keys:

- A TIME partition value is written like Paimon Java: ``HH:mm:ss``
followed by at most p fractional digits with trailing zeros dropped,
except that a whole second keeps one digit when p is greater than 0.
For example ``12:34:56.123`` and ``01:02:03.0`` for ``TIME(3)``, and
``01:02:03`` for ``TIME(0)``. With ``partition.legacy-name`` set to
true, it is the milliseconds of the day.
- A TIME bucket key is hashed as its int32 milliseconds of the day, like
Paimon Java.
- Predicates can reference TIME columns. A TIME literal is created with
``Literal(FieldType::TIME, millis)``, where ``millis`` is the
milliseconds of the day and must be in ``[0, 86400000)``. A query auth
row filter of a REST catalog table reads a TIME literal like Paimon
Java ``LocalTime.parse``, such as ``12:34:56.123``.

An Arrow time column of any unit other than milliseconds is rejected by
predicates, by partition and bucket keys, and when writing ORC and Avro
files. ``variant_get`` rejects such a target type too, including one
nested in a ROW, ARRAY or MAP. Reading ORC and Avro files rejects it
too, including a read schema such as ``time32[s]`` passed to
``SetReadSchema``, so a value is never written or read in another unit.

TIME statistics are int statistics. A format stats extractor returns an
``IntColumnStats`` for a TIME column, and manifest files keep its int32
values and read them back with the TIME type of the table schema. The
following table shows the TIME support of each data file format:

.. list-table::
:header-rows: 1

* - Format
- Physical type, same as Paimon Java
- Statistics
- Tested in Paimon C++
- Verified against Paimon Java
* - Parquet
- ``INT32`` annotated as ``TIME(MILLIS)``
- min and max
- append-only and primary-key tables
- read only
* - ORC
- ``int``
- min and max
- append-only and primary-key tables
- read only
* - Avro
- ``int`` annotated as ``time-millis``; ``TIME(p)`` with p greater
than 3 is rejected, like Paimon Java
- empty
- append-only and primary-key tables
- read only
* - Lance
- ``time32[ms]``
- empty
- append-only and primary-key tables
- no
* - Vortex
- ``time32[ms]``
- empty
- append-only and primary-key tables, without deletion vectors
- no
* - Mosaic
- ``time32[ms]``
- min and max of the columns in ``mosaic.stats-columns``, decoded as
big-endian int32 like Paimon Java
- append-only and primary-key tables
- no

The tests of Paimon C++ cover:

- Append-only tables: TIME partition keys read with predicates that
combine the partition key, a TIME value column and ``IS [NOT] NULL``.
- Primary-key tables: TIME partition, primary and bucket keys with
several buckets, updates, deletes, full compaction, bucket pruning
and deletion vectors. Key statistics pruning is tested for Parquet and
ORC only.

Lance, Vortex and Mosaic are only tested when Paimon C++ is built with
them enabled.

Across languages, "read only" means that Paimon C++ reads append-only
tables with TIME columns that Paimon Java writes. The following is
implemented following Paimon Java but not verified against it yet:

- Paimon Java reading tables that Paimon C++ writes.
- Paimon C++ reading primary-key tables that Paimon Java writes,
including TIME partitions, buckets, updates, deletes and compaction.
- TIME partition values and bucket ids computed by Paimon Java. The tests
of Paimon C++ compute the expected bucket ids with Paimon C++ itself.

A TIME string, such as a partition value, is parsed like Paimon Java
``BinaryStringUtils.toTime``: an integer number of milliseconds, or
``H[H][:m[m][:s[s][.f...]]]`` whose fraction is rounded half up to
milliseconds, optionally followed by an offset ``[+|-]hh[:mm]`` that is
added to the time, so ``12:34:56+01:00`` is ``13:34:56``. A string whose
components are all in range gives the same value as in Paimon Java.
Unlike Paimon Java, which may return an out-of-range value, Paimon C++
rejects an hour above 23, a minute, second or offset minute above 59, an
offset hour above 23, a non-digit in the fraction and a result outside
the day. It also accepts an ``hh:mm`` offset after ``H`` or ``H:m``,
such as ``12:34+01:00``, which Paimon Java rejects.

As in Paimon Java, schema evolution can change a column between TIME
and ``STRING``, ``TIMESTAMP`` or ``TIMESTAMP WITH LOCAL TIME ZONE``, and
reading older files casts their values:

- A string is parsed as described above, and a TIME value is formatted
as a partition value is, with the precision of the TIME type it was
written with.
- A timestamp gives its time of day, and a TIME value becomes that time
on 1970-01-01, both in the local time zone for ``TIMESTAMP WITH LOCAL
TIME ZONE``.

System tables, such as ``$files``, format TIME values in the same way as
a cast to ``STRING``.

**Note:** The following are not supported for TIME yet:

- Column default values (``fields.<name>.default-value``). Paimon C++
does not validate or apply default values for any type yet.
- File or global indexes on TIME columns.

* - ``TIMESTAMP``

Expand Down
3 changes: 3 additions & 0 deletions include/paimon/defs.h
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,9 @@ enum class FieldType {
VARIANT = 17,
/// Fixed-length dense vector represented by Arrow FixedSizeList.
VECTOR = 18,
/// TIME(p) without time zone, stored as milliseconds of the day in an int32 (Arrow
/// `time32[ms]`) whatever p is, matching Paimon Java.
TIME = 19,
UNKNOWN = 128,
};

Expand Down
11 changes: 6 additions & 5 deletions include/paimon/predicate/literal.h
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ namespace paimon {
/// Literal represents a constant value used in predicate expressions.
///
/// Literal support BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, FLOAT, DOUBLE, STRING, BINARY,
/// TIMESTAMP, DECIMAL, DATE.
/// TIMESTAMP, DECIMAL, DATE, TIME.
class PAIMON_EXPORT Literal {
public:
/// Creates a null literal of the specified type.
Expand Down Expand Up @@ -62,10 +62,11 @@ class PAIMON_EXPORT Literal {
/// if false, the caller must ensure the data remains valid.
Literal(FieldType binary_type, const char* str, size_t size, bool own_data);

/// Creates a date literal from an integer value.
/// @param date_type Must be `DATE` field type.
/// @param date_value Date value as days since epoch (1970-01-01).
Literal(FieldType date_type, int32_t date_value);
/// Creates a date or time literal from an integer value.
/// @param field_type Must be `DATE` or `TIME` field type.
/// @param value For `DATE`, days since epoch (1970-01-01); for `TIME`, milliseconds of the
/// day (0 to 86399999).
Literal(FieldType field_type, int32_t value);

Literal(const Literal& other);
~Literal();
Expand Down
4 changes: 4 additions & 0 deletions src/paimon/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -259,10 +259,14 @@ set(PAIMON_CORE_SRCS
core/casting/string_to_date_cast_executor.cpp
core/casting/string_to_decimal_cast_executor.cpp
core/casting/string_to_numeric_primitive_cast_executor.cpp
core/casting/string_to_time_cast_executor.cpp
core/casting/string_to_timestamp_cast_executor.cpp
core/casting/time_to_string_cast_executor.cpp
core/casting/time_to_timestamp_cast_executor.cpp
core/casting/timestamp_to_date_cast_executor.cpp
core/casting/timestamp_to_numeric_primitive_cast_executor.cpp
core/casting/timestamp_to_string_cast_executor.cpp
core/casting/timestamp_to_time_cast_executor.cpp
core/casting/timestamp_to_timestamp_cast_executor.cpp
core/casting/casting_utils.cpp
core/catalog/catalog.cpp
Expand Down
2 changes: 2 additions & 0 deletions src/paimon/common/data/binary_array_writer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ void BinaryArrayWriter::SetNullAt(int32_t ordinal, const arrow::Type::type& arro
SetNullValue<int16_t>(ordinal);
return;
case arrow::Type::type::DATE32:
case arrow::Type::type::TIME32:
case arrow::Type::type::INT32:
SetNullValue<int32_t>(ordinal);
return;
Expand Down Expand Up @@ -124,6 +125,7 @@ int32_t BinaryArrayWriter::GetElementSize(const arrow::Type::type& arrow_type) {
case arrow::Type::type::INT16:
return sizeof(int16_t);
case arrow::Type::type::DATE32:
case arrow::Type::type::TIME32:
case arrow::Type::type::INT32:
return sizeof(int32_t);
case arrow::Type::type::INT64:
Expand Down
3 changes: 2 additions & 1 deletion src/paimon/common/data/binary_row_writer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,8 @@ Result<BinaryRowWriter::FieldSetterFunc> BinaryRowWriter::CreateFieldSetter(
};
break;
}
case arrow::Type::type::DATE32: {
case arrow::Type::type::DATE32:
case arrow::Type::type::TIME32: {
field_setter = [field_idx](const VariantType& field, BinaryRowWriter* writer) -> void {
return writer->WriteInt(field_idx, DataDefine::GetVariantValue<int32_t>(field));
};
Expand Down
10 changes: 9 additions & 1 deletion src/paimon/common/data/binary_row_writer_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
#include "paimon/testing/utils/testharness.h"
namespace paimon::test {
TEST(BinaryRowWriterTest, TestFieldSetter) {
int32_t arity = 24;
int32_t arity = 25;
BinaryRow row(arity);
BinaryRowWriter writer(&row, /*initial_size=*/0, GetDefaultPool().get());
writer.Reset();
Expand Down Expand Up @@ -106,6 +106,9 @@ TEST(BinaryRowWriterTest, TestFieldSetter) {
ASSERT_OK_AND_ASSIGN(
BinaryRowWriter::FieldSetterFunc setter23,
BinaryRowWriter::CreateFieldSetter(23, arrow::timestamp(arrow::TimeUnit::NANO, timezone)));
ASSERT_OK_AND_ASSIGN(
BinaryRowWriter::FieldSetterFunc setter24,
BinaryRowWriter::CreateFieldSetter(24, arrow::time32(arrow::TimeUnit::MILLI)));

setter0(true, &writer);
setter1(static_cast<char>(1), &writer);
Expand Down Expand Up @@ -135,6 +138,7 @@ TEST(BinaryRowWriterTest, TestFieldSetter) {
setter21(Timestamp(1758030902002l, 0), &writer);
setter22(Timestamp(1758030902002l, 2000), &writer);
setter23(Timestamp(1758030902002l, 2002), &writer);
setter24(45296123, &writer);

writer.Complete();

Expand Down Expand Up @@ -208,6 +212,9 @@ TEST(BinaryRowWriterTest, TestFieldSetter) {
ASSERT_OK_AND_ASSIGN(InternalRow::FieldGetterFunc getter23,
InternalRow::CreateFieldGetter(23, arrow::timestamp(arrow::TimeUnit::NANO),
/*use_view=*/true));
ASSERT_OK_AND_ASSIGN(InternalRow::FieldGetterFunc getter24,
InternalRow::CreateFieldGetter(24, arrow::time32(arrow::TimeUnit::MILLI),
/*use_view=*/true));

ASSERT_EQ(DataDefine::GetVariantValue<bool>(getter0(row)), true);
ASSERT_EQ(DataDefine::GetVariantValue<char>(getter1(row)), static_cast<char>(1));
Expand Down Expand Up @@ -240,6 +247,7 @@ TEST(BinaryRowWriterTest, TestFieldSetter) {
Timestamp(1758030902002l, 2000));
ASSERT_EQ(DataDefine::GetVariantValue<Timestamp>(getter23(row)),
Timestamp(1758030902002l, 2002));
ASSERT_EQ(DataDefine::GetVariantValue<int32_t>(getter24(row)), 45296123);
}

TEST(BinaryRowWriterTest, TestFieldSetterWithNull) {
Expand Down
2 changes: 1 addition & 1 deletion src/paimon/common/data/columnar/columnar_array.h
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ class ColumnarArray : public InternalArray {
}

int32_t GetInt(int32_t pos) const override {
return ColumnarUtils::GetGenericValue<arrow::Int32Type, int32_t>(array_, offset_ + pos);
return ColumnarUtils::GetInt32Value(array_, offset_ + pos);
}

int32_t GetDate(int32_t pos) const override {
Expand Down
4 changes: 3 additions & 1 deletion src/paimon/common/data/columnar/columnar_batch_context.h
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,8 @@ struct CachedColumnMeta {
!arrow::bit_util::GetBit(null_bitmap, array_offset + row_id);
}

/// Fast fixed-width value access (INT8/INT16/INT32/INT64/FLOAT/DOUBLE/DATE32/TIMESTAMP).
/// Fast fixed-width value access
/// (INT8/INT16/INT32/INT64/FLOAT/DOUBLE/DATE32/TIME32/TIMESTAMP).
template <typename T>
inline T GetFixed(int64_t row_id) const {
return reinterpret_cast<const T*>(values_data)[array_offset + row_id];
Expand Down Expand Up @@ -103,6 +104,7 @@ struct ColumnarBatchContext {
case arrow::Type::INT16:
case arrow::Type::INT32:
case arrow::Type::DATE32:
case arrow::Type::TIME32:
case arrow::Type::INT64:
case arrow::Type::FLOAT:
case arrow::Type::DOUBLE: {
Expand Down
2 changes: 1 addition & 1 deletion src/paimon/common/data/columnar/columnar_row.h
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ class ColumnarRow : public InternalRow {
}

int32_t GetInt(int32_t pos) const override {
return ColumnarUtils::GetGenericValue<arrow::Int32Type, int32_t>(array_vec_[pos], row_id_);
return ColumnarUtils::GetInt32Value(array_vec_[pos], row_id_);
}

int32_t GetDate(int32_t pos) const override {
Expand Down
11 changes: 8 additions & 3 deletions src/paimon/common/data/columnar/columnar_row_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -85,19 +85,24 @@ TEST(ColumnarRowTest, TestSimple) {
TEST(ColumnarRowRefTest, TestSimple) {
auto pool = GetDefaultPool();
std::shared_ptr<arrow::DataType> target_type =
arrow::struct_({arrow::field("f1", arrow::int32()), arrow::field("f2", arrow::utf8())});
arrow::struct_({arrow::field("f1", arrow::int32()), arrow::field("f2", arrow::utf8()),
arrow::field("f3", arrow::time32(arrow::TimeUnit::MILLI))});
auto f1 =
arrow::ipc::internal::json::ArrayFromJSON(arrow::int32(), R"([1, 2, 3])").ValueOrDie();
auto f2 =
arrow::ipc::internal::json::ArrayFromJSON(arrow::utf8(), R"(["alpha", "beta", "gamma"])")
.ValueOrDie();
auto data = arrow::StructArray::Make({f1, f2}, target_type->fields()).ValueOrDie();
auto f3 = arrow::ipc::internal::json::ArrayFromJSON(arrow::time32(arrow::TimeUnit::MILLI),
R"([0, 45296123, 86399999])")
.ValueOrDie();
auto data = arrow::StructArray::Make({f1, f2, f3}, target_type->fields()).ValueOrDie();

auto ctx = std::make_shared<ColumnarBatchContext>(data->fields(), pool);
ColumnarRowRef row(ctx, 1);
ASSERT_EQ(row.GetFieldCount(), 2);
ASSERT_EQ(row.GetFieldCount(), 3);
ASSERT_EQ(row.GetInt(0), 2);
ASSERT_EQ(std::string(row.GetStringView(1)), "beta");
ASSERT_EQ(row.GetInt(2), 45296123);

auto row_kind = row.GetRowKind();
ASSERT_TRUE(row_kind.ok());
Expand Down
8 changes: 8 additions & 0 deletions src/paimon/common/data/columnar/columnar_utils.h
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,14 @@ class ColumnarUtils {
return typed_array->Value(pos);
}

/// Reads an int32 value from an INT32 or a TIME32 array.
static int32_t GetInt32Value(const arrow::Array* array, int32_t pos) {
if (array->type_id() == arrow::Type::TIME32) {
return GetGenericValue<arrow::Time32Type, int32_t>(array, pos);
}
return GetGenericValue<arrow::Int32Type, int32_t>(array, pos);
}

static std::string_view GetView(const arrow::Array* array, int32_t pos) {
auto type_id = array->type_id();
bool is_dict = (type_id == arrow::Type::type::DICTIONARY);
Expand Down
2 changes: 2 additions & 0 deletions src/paimon/common/data/data_define.h
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,8 @@ class DataDefine {
return Literal(GetVariantValue<Decimal>(value));
case arrow::Type::type::DATE32:
return Literal(FieldType::DATE, GetVariantValue<int32_t>(value));
case arrow::Type::type::TIME32:
return Literal(FieldType::TIME, GetVariantValue<int32_t>(value));
default:
return Status::Invalid(
fmt::format("Not support arrow type {} in VariantValueToLiteral",
Expand Down
7 changes: 7 additions & 0 deletions src/paimon/common/data/data_define_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,13 @@ TEST(DataDefineTest, VariantValueToLiteral) {
DataDefine::VariantValueToLiteral(value, arrow::Type::type::INT32));
ASSERT_EQ(literal.GetValue<int32_t>(), 100000);
}
{
// TIME32
VariantType value = static_cast<int32_t>(45296123);
ASSERT_OK_AND_ASSIGN(auto literal,
DataDefine::VariantValueToLiteral(value, arrow::Type::type::TIME32));
ASSERT_EQ(literal, Literal(FieldType::TIME, 45296123));
}
{
// INT64
VariantType value = static_cast<int64_t>(123456789L);
Expand Down
3 changes: 2 additions & 1 deletion src/paimon/common/data/internal_row.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,8 @@ Result<InternalRow::FieldGetterFunc> InternalRow::CreateFieldGetter(
};
break;
}
case arrow::Type::type::INT32: {
case arrow::Type::type::INT32:
case arrow::Type::type::TIME32: {
field_getter = [field_idx](const InternalRow& row) -> VariantType {
return row.GetInt(field_idx);
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,8 @@ Status BinarySerializerUtils::WriteBinaryData(const std::shared_ptr<arrow::DataT
writer->WriteInt(pos, getter->GetDate(pos));
break;
}
case arrow::Type::type::INT32: {
case arrow::Type::type::INT32:
case arrow::Type::type::TIME32: {
writer->WriteInt(pos, getter->GetInt(pos));
break;
}
Expand Down
Loading
Loading