perf(sparse): trim the stages around the CSR solver hand-off by FabianHofmann · Pull Request #992 · PyPSA/linopy

FabianHofmann · 2026-09-26T19:21:06Z

Note

The following content was generated by AI.

Changes proposed in this Pull Request

Follow-up to #985/#987/#988/#990. An assessment of the sparse (Model(sparse=True)) data flow from expression construction to the solver found the core hand-off close to optimal (one CSR build per operation, one label remap, a no-copy pass to the direct APIs) but avoidable cost in the stages around it. This PR removes that cost. Every change keeps results bit-identical to master; each was implemented, independently reviewed and revised in its own round.

Solver hand-off

The MOSEK direct API builds bound keys, bounds and names vectorised instead of looping over every row and column in Python. MOSEK is not installed on the development machine, so this path was verified by reading the API only; please run test_optimization.py -k mosek on a licensed machine.
HiGHS names, when requested, are attached through getModel()/passModel() instead of getLp(). Re-passing the LP-only object drops the Hessian, so a QP solved with names on got a wrong objective. A regression test is added.

LP writer

The scaling lookups are computed once per file instead of four times, and skipped entirely when every factor is 1. Rows that already arrive grouped (frozen constraints) are not re-sorted, and each constraint line is built from two string branches instead of seven. Constraint writing is ~2x faster. A golden-text test covers scaled and unscaled, frozen and mutable, and the unsorted fallback.

Expressions

linopy.merge of CSR-backed expressions concatenates all operands once instead of folding pairwise, so an N-way merge is linear in total nonzeros rather than quadratic in operand count.
LinearExpression.solution / QuadraticExpression.solution map the solution by variable label and evaluate CSR-backed expressions on the sparse backing. They no longer rebuild the full constraint matrix nor densify. Model.dualize reads model.matrices once.

Matrix assembly and read-back

model.matrices gathers constraint blocks into preallocated CSR buffers in one pass, without vstack, eliminate_zeros or a full-grid scaling round trip; scaling is skipped when identity. Label lookups use the model's label dtype, so matrices.clabels and matrices.indicator_binvar are int32 by default (behaviour change, noted in the release notes).
Duals of frozen constraints are set on their active rows directly instead of through three full-grid arrays.
Freezing a masked constraint gathers rows once instead of up to three times; mixed-sign sense conversion and sanitize_zeros are vectorised without large temporaries.

The chunk sizing of the sparse @ contraction was also tried but is redundant after #990 and was reverted.

Benchmarks (Model(sparse=True), 8.18M rows, 4.4M variables, 16.7M nonzeros, highspy 1.15, numpy 2.4)

Stage	master	this PR
LP file write, full	1.99 s	1.25 s
LP file write, constraints only	1.57–1.78 s	0.81–0.85 s
`model.matrices`	0.25 s, 771 MB peak	0.14 s, 571 MB peak
dual read-back (`assign_result`)	0.07 s, 165 MB	0.03 s, 133 MB
`sanitize_zeros` per solve	0.02 s, 72 MB	0.01 s, 16 MB
`merge` of 8 CSR operands, 8M nnz	0.37 s	0.13 s
`expr.solution`, CSR expression (40k x 200 vars)	1.35 s, 2.1 GB	0.14 s, 336 MB
`expr.solution`, dense expression, same model	1.44 s, 2.1 GB	0.30 s, 544 MB

The retained outputs of model.matrices alone are ~555 MB on this model; transient overhead is now ~16 MB.

Verification

LP output compared byte-for-byte against master over 256 model variants (frozen/mutable, variable/row/objective scaling, linear/quadratic objective, unsorted labels, explicit names, slice sizes).
model.matrices compared value-for-value against master over 25 models (masks, explicit zeros, NaN rhs, scaling, removed containers, integer/binary, indicator, empty).
Frozen duals compared against master and against the dense model over 16 scaling/freeze combinations.
N-ary merge and CSR solution compared against the dense path with absent cells, NaN solution values and auxiliary coordinates.
Full test suite: 6994 passed, 1315 skipped (remote tests excluded); mypy and ruff clean.

Checklist

AI-generated content is marked (see AGENTS.md).
Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
Unit tests for new features were added (if applicable).
A note for the release notes doc/release_notes.rst of the upcoming release is included.
I consent to the release of this PR's code under the MIT license.

…es via getModel MOSEK bound keys and names are built vectorised. The polars LP writer skips identity scaling, avoids re-sorting frozen rows and builds fewer string branches; output is byte-identical. HiGHS names, when requested, are attached through getModel/passModel so the Hessian survives.

…ntraction chunks Merge concatenates all CSR operands once instead of folding pairwise. Expression.solution maps labels directly and evaluates CSR rows without densifying or rebuilding the constraint matrix. Contraction chunks are sized by nonzeros instead of a fixed row count.

… read-back, one freeze gather Constraint blocks are gathered into preallocated CSR buffers without vstack or eliminate_zeros; scaling is skipped when identity and label lookups use the label dtype. Frozen constraints receive duals on active rows directly. Mask and non-empty filtering share one gather and sense conversion is vectorised.

…ion (#990)

codspeed · 2026-09-26T19:24:07Z

Merging this PR will improve performance by 44.76%

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 14 improved benchmarks
✅ 167 untouched benchmarks
⏩ 181 skipped benchmarks¹

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	`test_to_lp[nodal_balance_sparse-severity=0]`	3.9 MB	2.5 MB	+54.8%
⚡	`test_to_lp[nodal_balance_sparse-severity=50]`	3.9 MB	2.5 MB	+54.71%
⚡	`test_to_lp[nodal_balance_sparse-severity=100]`	3.9 MB	2.5 MB	+54.71%
⚡	`test_to_lp[knapsack-n=10000]`	2.1 MB	1.4 MB	+52.34%
⚡	`test_to_lp[storage-n=250]`	38.5 MB	25.8 MB	+49.12%
⚡	`test_to_lp[merge_balance-severity=0]`	2.7 MB	1.8 MB	+48.43%
⚡	`test_to_lp[rolling-severity=0]`	2 MB	1.4 MB	+48.07%
⚡	`test_to_lp[nodal_balance-severity=50]`	3.8 MB	2.6 MB	+45.33%
⚡	`test_to_lp[nodal_balance-severity=0]`	3.8 MB	2.6 MB	+45.27%
⚡	`test_to_lp[storage-n=10]`	1.6 MB	1.1 MB	+44.9%
⚡	`test_to_lp[expression_arithmetic-n=250]`	49.9 MB	36.3 MB	+37.44%
⚡	`test_to_lp[piecewise-n=1000]`	1.9 MB	1.4 MB	+37.3%
⚡	`test_to_lp[basic-n=250]`	29.9 MB	21.8 MB	+37.15%
⚡	`test_to_lp[sos-n=1000]`	1.4 MB	1.1 MB	+21.19%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing perf/sparse-pipeline (2e33cec) with master (1b2ea76)}

181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

github-actions · 2026-09-26T19:24:45Z

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy	time — v1 / legacy

Full table (time + peak, mean)

benchmarks/drivers/test_build.py::test_build[basic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.06376 (1.07)   │   15.03 (1.00) 
 (v1)        0.05943 (1.0)   │    15.00 (1.0) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.07303 (1.10)   │   12.04 (1.00) 
 (v1)        0.06668 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02776 (1.04)   │   15.20 (1.0) 
 (v1)        0.02663 (1.0)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04386 (1.10)   │   44.93 (1.0) 
 (v1)        0.03999 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.0313 (1.07)   │   11.51 (1.0) 
 (v1)       0.02933 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.07161 (1.05)   │   24.34 (1.06) 
 (v1)        0.06851 (1.0)   │    23.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.08332 (1.07)   │   16.12 (1.00) 
 (v1)        0.07778 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.01799 (1.10)   │   752.18 (1.10) 
 (v1)        0.01641 (1.0)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.01694 (1.05)   │   3.12 (1.33) 
 (v1)        0.01611 (1.0)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                 time (s)          peak (MiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0587 (1.39)   │   126.16 (1.44) 
 (v1)       0.04221 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                 time (s)          peak (MiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0579 (1.42)   │   126.16 (1.44) 
 (v1)       0.04092 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.05809 (1.45)   │   126.16 (1.44) 
 (v1)        0.04007 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03923 (1.00)   │    715.12 (1.0) 
 (v1)        0.03917 (1.0)   │   787.73 (1.10) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03835 (1.09)   │   4.54 (1.27) 
 (v1)        0.03521 (1.0)   │    3.57 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.2721 (1.04)   │   704.12 (1.09) 
 (v1)        0.2619 (1.0)   │    643.85 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.2882 (1.04)   │   18.34 (1.0) 
 (v1)         0.276 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.2827 (1.03)   │   9.54 (1.0) 
 (v1)        0.2738 (1.0)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05443 (1.07)   │   3.77 (1.12) 
 (v1)        0.05073 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.05584 (1.09)   │   216.59 (1.10) 
 (v1)        0.05121 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.02797 (1.08)   │   938.49 (1.0) 
 (v1)        0.02593 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02957 (1.07)   │   9.66 (1.0) 
 (v1)        0.02775 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02891 (1.07)   │   5.32 (1.0) 
 (v1)        0.02704 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01445 (1.00)   │   1.28 (1.0) 
 (v1)        0.01441 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)    0.01436 (1.0)   │   1.28 (1.0) 
 (v1)       0.01438 (1.00)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.0149 (1.03)   │   1.28 (1.0) 
 (v1)       0.01441 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.1358 (1.04)   │   946.85 (1.06) 
 (v1)        0.1306 (1.0)   │    891.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1349 (1.05)   │   12.01 (1.00) 
 (v1)        0.1281 (1.0)   │    11.99 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03464 (1.05)   │   147.70 (1.06) 
 (v1)        0.03287 (1.0)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03432 (1.05)   │   2.60 (1.09) 
 (v1)        0.03275 (1.0)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02857 (1.07)   │   696.75 (1.03) 
 (v1)        0.02666 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)    0.07178 (1.0)   │   137.97 (1.0) 
 (v1)       0.07299 (1.02)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04668 (1.04)   │   69.22 (1.0) 
 (v1)        0.04502 (1.0)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03304 (1.09)   │   402.33 (1.00) 
 (v1)        0.03022 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                 time (s)        peak (KiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.0328 (1.10)   │   3.19 (1.19) 
 (v1)       0.02986 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.03609 (1.07)   │   29.00 (1.54) 
 (v1)        0.03362 (1.0)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.04741 (1.14)   │   37.95 (1.43) 
 (v1)        0.04152 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)     0.0681 (1.0)   │    410.93 (1.0) 
 (v1)       0.06995 (1.03)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.07543 (1.0)   │     9.94 (1.0) 
 (v1)       0.0778 (1.03)   │   10.22 (1.03)

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

_{Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.}

FabianHofmann added 5 commits September 26, 2026 13:54

docs: release notes for sparse pipeline performance work

f315788

perf(csr): keep fixed contraction chunk, superseded by column compact…

1bee868

…ion (#990)

test(io): type slice_size explicitly in LP constraint section helper

2e33cec

Navigation Menu

perf(sparse): trim the stages around the CSR solver hand-off - #992

perf(sparse): trim the stages around the CSR solver hand-off#992
FabianHofmann wants to merge 6 commits into
masterfrom
perf/sparse-pipeline

FabianHofmann commented Sep 26, 2026

Uh oh!

codspeed Bot commented Sep 26, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 26, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

FabianHofmann commented Sep 26, 2026

Changes proposed in this Pull Request

Checklist

Uh oh!

codspeed Bot commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will improve performance by 44.76%

Performance Changes

Footnotes

Uh oh!

github-actions Bot commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build cost — v1 vs legacy

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codspeed Bot commented Sep 26, 2026 •

edited

Loading

github-actions Bot commented Sep 26, 2026 •

edited

Loading