Native Iceberg scan drops rows from data files that store a narrower type than the column #7138

Describe the bug

Native Iceberg scans can drop rows when filtering on a column whose data files store a narrower type, for example INT data in a BIGINT column. iceberg-rust's page index skips all pages of such a file for range and IN predicates, and its row filter casts the literal to the file's type, so an out-of-range value becomes NULL. The post-scan filter cannot restore the rows.

Such files come from type promotion and from imports by add_files, migrate or snapshot, which do not check file types against the schema.

Steps to reproduce

CREATE TABLE t (id INT) USING iceberg;
INSERT INTO t VALUES (1), (2), (3);
ALTER TABLE t ALTER COLUMN id TYPE BIGINT;
INSERT INTO t VALUES (4), (5);
SELECT * FROM t WHERE id > 0;

Reproduced at 9e13649.

Expected behavior

All five rows, as Spark returns. Comet returns only 4 and 5. id < 3 and id IN (1, 2) return no rows.

Additional context

Imported files show the same behavior without any schema change. A file storing id as INT in a BIGINT column, or t as TINYINT in an INT column, lost rows for id > 0, id IN (1, 2), t < 200 and t IN (1, 200); only = returned the right rows. Reproduced on the #6847 branch, which pushes these predicates as main does.

Upstream: apache/iceberg-rust#3122. Its fix, apache/iceberg-rust#3123, was closed as stale and does not cover TINYINT or SMALLINT files.

#6847 pushes only = and null checks on columns whose type changed in the schema history, and on every BIGINT and DOUBLE column when a residual converts only in part. Tables with a name mapping push a residual only whole. Whole residuals on imported files, or on columns promoted before Iceberg 0.12, still need the upstream fix.