`loo_compare()` returns `elpd_diff` with the opposite sign for `psis_loo_ss` objects

[florence-bockting]

What happened

loo_compare() returns elpd_diff with the opposite sign for subsampled
psis_loo_ss objects. It also returns a matrix, not a data.frame.

What I expected

In loo_compare() the difference is comparison-model minus reference-model,
and the output is a data.frame.

Minimal reproducible example

library(loo)

set.seed(4711)
N <- 500
S <- 500
data <- data.frame(y = rnorm(N, mean = 0, sd = 1))

# two models: model1 is well centred, model2 is biased by 0.3
draws1 <- matrix(rnorm(S, 0.0, 0.05), ncol = 1, dimnames = list(NULL, "mu"))
draws2 <- matrix(rnorm(S, 0.3, 0.05), ncol = 1, dimnames = list(NULL, "mu"))

llfun <- function(data_i, draws, ...) {
  dnorm(data_i$y, mean = draws[, "mu"], sd = 1, log = TRUE)
}

l1 <- loo(llfun, data = data, draws = draws1, r_eff = rep(1, N))
l2 <- loo(llfun, data = data, draws = draws2, r_eff = rep(1, N))
loo_compare(list(model1 = l1, model2 = l2))

set.seed(4711)
s1 <- loo_subsample(llfun, data = data, draws = draws1,
                    observations = 200, loo_approximation = "plpd",
                    r_eff = rep(1, N))
set.seed(4711)
s2 <- loo_subsample(llfun, data = data, draws = draws2,
                    observations = s1, loo_approximation = "plpd",
                    r_eff = rep(1, N))
loo_compare(list(model1 = s1, model2 = s2))

Actual output

> loo_compare(list(model1 = l1, model2 = l2))
  model elpd_diff se_diff p_worse diag_diff diag_elpd
 model1       0.0     0.0      NA
 model2     -25.8     6.6    1.00

> loo_compare(list(model1 = s1, model2 = s2))
       elpd_diff se_diff subsampling_se_diff
model1  0.0       0.0     0.0
model2 25.7       6.6     0.0

The return types also differ:

full: class compare.loo/data.frame                  | rownames 1,2
ss  : class compare.loo_ss/compare.loo/matrix/array | rownames model1,model2

Environment

• OS: Ubuntu 24.04.4 LTS
• Package version: loo 2.10.0
• R version: 4.6.1 (2026-06-24)

[jgabry]

Wow, that's not good. I'm glad you noticed this! Do you possibly have time to fix this?

[florence-bockting]

Wow, that's not good. I'm glad you noticed this! Do you possibly have time to fix this?

Yes, I can make a PR tomorrow.

[jgabry]

Thank you

[florence-bockting]

@jgabry I split the issue in two: first addresses the sign flip and the second the change from matrix to data.frame.
My reasoning was that the first one is the urgent change while the second one is actually slow as we have to do again reverse dependency checks here. I am actually not sure whether we want to do this separately or rather wait for loo v3 where we will have to run reverse dependency checks any way again.

However, I created both PRs already. Only the reverse dependency checks need to run.

What do you think about it?