feat: write value stats for append data files

[QuakeWang]

Purpose

Rust append writes currently emit empty _VALUE_STATS, so manifest-level stats pruning cannot skip data files written by Rust even when min/max/null-count information is available.

Brief change log

• Accumulate min/max/null-count value stats across batches in DataFileWriter.
• Persist stats in existing DataFileMeta.value_stats and dense value_stats_cols.
• Skip unsupported variable-length value stats and add nanosecond timestamp extraction support.
• Add tests for value stats writing, timestamp(9), skipped variable-length columns, and scan pruning.

Tests

• cargo fmt --all -- --check
• cargo clippy --all-targets --workspace --features fulltext,vortex,mosaic -- -D warnings
• cargo test -p paimon --all-targets --features fulltext,vortex,mosaic

API and Format

Documentation

[JingsongLi]

In Java Side, the performance of the accumulator is poor, and the only truly useful link is the one that extracts stats from Parquet. Perhaps we should optimize its implementation instead of the accumulator.

[QuakeWang]

In Java Side, the performance of the accumulator is poor, and the only truly useful link is the one that extracts stats from Parquet. Perhaps we should optimize its implementation instead of the accumulator.

@JingsongLi Thanks, addressed.

I changed the implementation to avoid per-row stats accumulation. Append value stats are now extracted from Parquet writer metadata on close, aligned with the Java ParquetSimpleStatsExtractor path.

Added/updated tests for append value stats, stats pruning, and timestamp stats.

[JingsongLi]

Just return stats from parquet? It is not good to expose ParquetMetaData in core.

[JingsongLi]

We need to support default stats mode: truncate(16), otherwise the stats are too big.

[QuakeWang]

@JingsongLi Addressed, thanks. Conflicts are resolved, ParquetMetaData is no longer exposed to core, and string value stats now use default truncate(16) with test coverage.

[JingsongLi]

Thanks for the update. I think there are still a couple of Java compatibility gaps here.

1. The manifest stats mode is currently hard-coded to the default truncate(16) behavior. In Java, stats collectors are created from metadata.stats-mode, fields.<name>.stats-mode, and metadata.stats-keep-first-n-columns. With this implementation, a table configured with metadata.stats-mode = none/counts/full or a field-level override would still get truncate(16)-style stats in Rust-written manifests. Could we thread the table stats options into this extraction path and apply the same collector semantics as Java?

2. supports_manifest_value_stats currently skips unsupported min/max types completely. Java does not do that: for unsupported Parquet min/max types such as BINARY/VARBINARY, ParquetSimpleStatsExtractor still returns SimpleColStats(null, null, nullCount), and dense stats storage keeps the column when the null count is known. The current Rust behavior drops those columns from value_stats_cols, so their null counts are lost. Also, Java supports TIME_WITHOUT_TIME_ZONE through IntStatistics, while Rust's conversion code has Time support but the support list excludes DataType::Time. Could we keep null counts for unsupported min/max columns and include Time to match Java?