From e34a29232434d5611f979bf3c6e8f5fc6858bed4 Mon Sep 17 00:00:00 2001 From: Benjamin Schwendinger Date: Thu, 10 Sep 2026 10:29:41 +0200 Subject: [PATCH 1/2] update NEWS after CRAN release --- NEWS.md | 35 +++++++++++++++++++++++------------ 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/NEWS.md b/NEWS.md index 6756f1edd..2f418a4dd 100644 --- a/NEWS.md +++ b/NEWS.md @@ -34,8 +34,6 @@ 6. `yearqtr()` and `yearmon()` now gain an optional format specifier [#7694](https://github.com/Rdatatable/data.table/issues/7694). 'numeric' is the default, which preserves the original behavior, but 'character' formats `yearqtr()` as YYYYQ# (e.g. 2025Q2) and `yearmon()` as YYYYM## (e.g. 2025M02, 2025M10). Thanks to @jan-swissre for the report and @LunaticSage218 for the implementation. -7. Rows can now be deleted by reference using `DT[i, .ROW := NULL]`, avoiding a full copy of the table for large row-removal operations, [#635](https://github.com/Rdatatable/data.table/issues/635). This has been one of data.table's most requested features. Target rows must be selected with the `i` expression, `by`/`keyby` are not supported, and keys/indices are cleared after deletion. The new experimental helper `setallocrow()` prepares columns for by-reference row operations. Thanks @arunsrinivasan for the feature request, @ben-schwen for the implementation, and @aitap for review and assistance. - 8. `tstrsplit()` gains a `rev` argument to facilitate extracting elements anchored from the end of the string, [#6341](https://github.com/Rdatatable/data.table/issues/6341). This is especially useful when strings have a varying number of components and you only want to extract the last or second-to-last element. Thanks to @JBrownArcGen for the suggestion and @venom1204 for the implementation. 9. The `give.names` argument of rolling functions (`froll*()`, `frollapply()`, and `frolladapt()`) now accepts a character vector to directly specify output names, [#5744](https://github.com/Rdatatable/data.table/issues/5744). Thanks to @jangorecki for the suggestion and @ben-schwen for the implementation. @@ -76,21 +74,19 @@ 12. `print.data.table()` now truncates long character columns and list-column summaries by default to avoid horizontal console overflow, [#7718](https://github.com/Rdatatable/data.table/issues/7718). When `datatable.prettyprint.char` is `NULL` (the default), the truncation limit is now dynamically calculated based on the available console width. Use `options(datatable.prettyprint.char=Inf)` for the old default behavior (never truncate). Thanks @tdhock for the report and @venom1204 for the fix. -13. `rbindlist()` (and therefore the `rbind()` method for `data.table`s) no longer raises an error upon encountering more than approximately 50000 columns in a list entry, [#7793](https://github.com/Rdatatable/data.table/issues/7793). The bug was introduced in `data.table` version 1.18.2.1. Thanks to @rickhelmus for the report and @aitap for the fix. - -14. Subtracting an `IDate` from a `Date` is fast again by avoiding unnecessary conversion to `POSIXlt`/`POSIXct`, [#7825](https://github.com/Rdatatable/data.table/issues/7825). Thanks @gilesheywood for the report and @ben-schwen for the fix. +13. Subtracting an `IDate` from a `Date` is fast again by avoiding unnecessary conversion to `POSIXlt`/`POSIXct`, [#7825](https://github.com/Rdatatable/data.table/issues/7825). Thanks @gilesheywood for the report and @ben-schwen for the fix. -15. `as.IDate()` and `as.ITime()` now preserve names, matching base `as.Date()` behavior, [#7252](https://github.com/Rdatatable/data.table/issues/7252). Thanks @DavisVaughan for the report, @venom1204 for the PR, and @MichaelChirico for patching the fix back to old versions of R. +14. `as.IDate()` and `as.ITime()` now preserve names, matching base `as.Date()` behavior, [#7252](https://github.com/Rdatatable/data.table/issues/7252). Thanks @DavisVaughan for the report, @venom1204 for the PR, and @MichaelChirico for patching the fix back to old versions of R. -16. `copy()` is now more consistent about reallocating nested `data.table`s, [#7456](https://github.com/Rdatatable/data.table/issues/7456). The resulting list is now only overwritten when necessary, list columns inside data.tables are searched recursively, and their attributes are walked in search of data.tables to reallocate as well. Thanks to @be-marc for the report, @david-cortes for additional information, and @aitap for the fix. +15. `copy()` is now more consistent about reallocating nested `data.table`s, [#7456](https://github.com/Rdatatable/data.table/issues/7456). The resulting list is now only overwritten when necessary, list columns inside data.tables are searched recursively, and their attributes are walked in search of data.tables to reallocate as well. Thanks to @be-marc for the report, @david-cortes for additional information, and @aitap for the fix. -17. `print()` works with multi-byte characters on R before 4.2.0, [#7848](https://github.com/Rdatatable/data.table/pull/7848). Thanks @MichaelChirico for the fix and @aitap for the improvement. +16. `print()` works with multi-byte characters on R before 4.2.0, [#7848](https://github.com/Rdatatable/data.table/pull/7848). Thanks @MichaelChirico for the fix and @aitap for the improvement. -18. `example(local=TRUE)` where the example uses `[.data.table` works again (e.g. `example(':=', package='data.table', local=TRUE, echo=FALSE)`), [#7855](https://github.com/Rdatatable/data.table/issues/7855) re-fixing [#2972](https://github.com/Rdatatable/data.table/issues/2972). Thanks @michaelChirico for the fix. +17. `example(local=TRUE)` where the example uses `[.data.table` works again (e.g. `example(':=', package='data.table', local=TRUE, echo=FALSE)`), [#7855](https://github.com/Rdatatable/data.table/issues/7855) re-fixing [#2972](https://github.com/Rdatatable/data.table/issues/2972). Thanks @michaelChirico for the fix. -19. `DT[order(double, ..., -non_double, na.last=TRUE)]`, i.e., a double/complex column (in any order) followed by a non-double column in descending order with `na.last=TRUE`, is fixed to respect `na.last` again, [#7875](https://github.com/Rdatatable/data.table/issues/7875). The problematic behavior only occurred under specific conditions on the cardinality of the non-double column. +18. `DT[order(double, ..., -non_double, na.last=TRUE)]`, i.e., a double/complex column (in any order) followed by a non-double column in descending order with `na.last=TRUE`, is fixed to respect `na.last` again, [#7875](https://github.com/Rdatatable/data.table/issues/7875). The problematic behavior only occurred under specific conditions on the cardinality of the non-double column. -20. `print.data.table()` now correctly displays data when `col.names="none"` and `row.names=FALSE`, [#7735](https://github.com/Rdatatable/data.table/issues/7735). Thanks to @jan-swissre for the report and @venom1204 for the fix. +19. `print.data.table()` now correctly displays data when `col.names="none"` and `row.names=FALSE`, [#7735](https://github.com/Rdatatable/data.table/issues/7735). Thanks to @jan-swissre for the report and @venom1204 for the fix. ### Notes @@ -114,7 +110,22 @@ 10. `fwrite()` returns a clearer error message when `na = data.frame()` is used, [#7866](https://github.com/Rdatatable/data.table/issues/7866). Thanks @mcol for the report and the fix. -11. Handled OpenMP deprecation of `master` construct, [#7882](https://github.com/Rdatatable/data.table/pull/7882). Thanks @TimTaylor for the PR. + +## data.table [v1.18.6.1](https://github.com/Rdatatable/data.table/milestone/47) (22 August 2026) + +### NEW FEATURES + +1. Rows can now be deleted by reference using `DT[i, .ROW := NULL]`, avoiding a full copy of the table for large row-removal operations, [#635](https://github.com/Rdatatable/data.table/issues/635). This has been one of data.table's most requested features. Target rows must be selected with the `i` expression, `by`/`keyby` are not supported, and keys/indices are cleared after deletion. The new experimental helper `setallocrow()` prepares columns for by-reference row operations. Thanks @arunsrinivasan for the feature request, @ben-schwen for the implementation, and @aitap for review and assistance. + +### BUG FIXES + +1. Adapted the tests to stop using the "special" attribute names in `structure()` calls, [#7813](https://github.com/Rdatatable/data.table/issues/7813) & [#7814](https://github.com/Rdatatable/data.table/issues/7814), avoiding deprecation warnings on R-devel. Thanks @ben-schwen for the report and @ben-schwen & @MichaelChirico for the fix. + +2. `rbindlist()` (and therefore the `rbind()` method for `data.table`s) no longer raises an error upon encountering more than approximately 50000 columns in a list entry, [#7793](https://github.com/Rdatatable/data.table/issues/7793). The bug was introduced in `data.table` version 1.18.2.1. Thanks to @rickhelmus for the report and @aitap for the fix. + +## NOTES + +1. Handled OpenMP deprecation of `master` construct, [#7882](https://github.com/Rdatatable/data.table/pull/7882). Thanks @TimTaylor for the PR. ## data.table [v1.18.4](https://github.com/Rdatatable/data.table/milestone/45) (6 May 2026) From 6b750e0f1c9dd4f0aacdd7cdae8bc3bad820ca26 Mon Sep 17 00:00:00 2001 From: Benjamin Schwendinger Date: Thu, 10 Sep 2026 11:05:53 +0200 Subject: [PATCH 2/2] add strict column overrides for yaml schema --- NEWS.md | 2 ++ R/fread.R | 21 +++++++++++++++------ inst/tests/other.Rraw | 16 ++++++++++++++++ 3 files changed, 33 insertions(+), 6 deletions(-) diff --git a/NEWS.md b/NEWS.md index 2f418a4dd..ce4297031 100644 --- a/NEWS.md +++ b/NEWS.md @@ -88,6 +88,8 @@ 19. `print.data.table()` now correctly displays data when `col.names="none"` and `row.names=FALSE`, [#7735](https://github.com/Rdatatable/data.table/issues/7735). Thanks to @jan-swissre for the report and @venom1204 for the fix. +21. `fread(yaml=TRUE)` now honors column types in the YAML schema, including empty columns written by `fwrite(yaml=TRUE)`, [#3999](https://github.com/Rdatatable/data.table/issues/3999). Unknown schema types now fall back to type inference even when no schema types are recognized. Thanks to @PavoDive and @Noskario for the report and @ben-schwen for the fix. + ### Notes 1. {data.table} now depends on R 3.5.0 (2018). diff --git a/R/fread.R b/R/fread.R index 4aa6a01ed..ec3613897 100644 --- a/R/fread.R +++ b/R/fread.R @@ -289,15 +289,22 @@ yaml=FALSE, tmpdir=tempdir(), tz="UTC") new_types = sapply(yaml_header$schema$fields, `[[`, 'type') if (any(null_idx <- vapply_1b(new_types, is.null))) new_types = do.call(c, new_types) - synonms = rbindlist(list( + synonyms = rbindlist(list( character = list(syn = c('character', 'string')), integer = list(syn = c('integer', 'int')), numeric = list(syn = c('numeric', 'number', 'double')), factor = list(syn = c('factor', 'categorical')), + ordered = list(syn = 'ordered'), + logical = list(syn = 'logical'), + complex = list(syn = 'complex'), + Date = list(syn = 'Date'), + IDate = list(syn = 'IDate'), + ITime = list(syn = 'ITime'), + POSIXct = list(syn = 'POSIXct'), integer64 = list(syn = c('integer64', 'int64')) ), idcol = 'r_type') - setkeyv(synonms, 'syn') - new_types = synonms[list(new_types)]$r_type + setkeyv(synonyms, 'syn') + new_types = synonyms[list(new_types)]$r_type new_names = sapply(yaml_header$schema$fields[!null_idx], `[[`, 'name') if ('col.names' %chin% call_args) messagef("User-supplied column names in 'col.names' will override those found in YAML metadata.") @@ -313,8 +320,8 @@ yaml=FALSE, tmpdir=tempdir(), tz="UTC") brackify(new_names[matched_name_idx[!idx_type]])) } } - # only add unmentioned columns - for (ii in which(!idx_name)) { + # only add unmentioned columns # dont override unknown types + for (ii in which(!idx_name & !is.na(new_types))) { colClasses[[ new_types[ii] ]] = c(colClasses[[ new_types[ii] ]], new_names[ii]) } } else { @@ -325,7 +332,8 @@ yaml=FALSE, tmpdir=tempdir(), tz="UTC") if (!'col.names' %chin% call_args) col.names = new_names new_names = paste0('V', seq_along(new_names)) } - colClasses = tapply(new_names, new_types, c, simplify=FALSE) + # deactivate override for all unknown types + colClasses = if (all(is.na(new_types))) NULL else tapply(new_names, new_types, c, simplify=FALSE) } } sep_syn = c('sep', 'delimiter') @@ -383,6 +391,7 @@ yaml=FALSE, tmpdir=tempdir(), tz="UTC") new_v = tryCatch({ # different to read.csv; i.e. won't error if a column won't coerce (fallback with warning instead) switch(new_class, "factor" = as_factor(v), + "ordered" = as.ordered(v), "complex" = as.complex(v), "raw" = as_raw(v), # Internal implementation "Date" = as.Date(v), diff --git a/inst/tests/other.Rraw b/inst/tests/other.Rraw index 928bc8e63..78b20dede 100644 --- a/inst/tests/other.Rraw +++ b/inst/tests/other.Rraw @@ -457,6 +457,22 @@ if (loaded[["yaml"]]) { # csvy; #1701. Was 2032-2033 in tests.Rraw, #5516 close(fcon) test(17.14, fread(f), DT) unlink(f) + + # honor YAML column types even without values to infer, #3999 + DT = data.table(date = as.Date('2006-05-01'), idate = as.IDate('2006-05-01'), + time = as.ITime('12:00:00'), datetime = as.POSIXct('2006-05-01', tz='UTC'), + ordered = ordered('a'), factor = factor('a'), logical = TRUE, complex = 1+2i) + f = tempfile() + fwrite(DT[0L], f, yaml=TRUE) + # we need the options since otherwise fread converts Date to IDate naturally + test(17.15, lapply(fread(f, yaml=TRUE), class), lapply(DT, class), options=c(datatable.old.fread.datetime.character=TRUE)) + fwrite(DT, f, yaml=TRUE) + test(17.16, lapply(fread(f, yaml=TRUE), class), lapply(DT, class), options=c(datatable.old.fread.datetime.character=TRUE)) + # unknown yaml schema types fall back to inference, even when none are recognized + DT = data.table(x = structure(1:2, class='unknown')) + fwrite(DT, f, yaml=TRUE) + test(17.17, lapply(fread(f, yaml=TRUE), class), list(x='integer')) + unlink(f) } if (loaded[["xts"]]) { # was 1465 in tests.Rraw, #5516