Add ONNX Runtime backend support (OpenVINO / DirectML / TensorRT / CPU, Windows & Linux) by seniorfish · Pull Request #1222 · lightvector/KataGo

seniorfish · 2026-07-31T17:28:52Z

updated

Add ONNX Runtime neural-net backend

Feedback welcome — see notes at the bottom.

Summary

This PR adds a new USE_BACKEND=ONNX option that runs KataGo's neural-net inference through ONNX Runtime, with pluggable execution providers.

It targets current master including the transformer trunk models, and reuses master's existing OnnxModelBuilder (shared with the TensorRT backend) instead of a parallel graph emitter — so the new backend is a single file plus wiring and touches no other backend.

Verified in this PR:

Provider	Where	Status
OpenVINO	Intel GPU & NPU (Arc B580 desktop, Arc 140T laptop)	verified end-to-end, benchmarked
DirectML	Windows (B580)	benchmarked
TensorRT	Ubuntu 24.04 NVIDIA RTX 4090	benchmarked
CPU	Windows	benchmarked

Other providers (cuda / migraphx / coreml) are wired in and should work but are unverified. Note: upstream verification status (per Compiling.md) is limited to OpenVINO on Windows + Intel GPU; DirectML / TensorRT / CPU are covered by this PR's CI and manual testing but are not continuously tested upstream.

What changed

cpp/neuralnet/onnxbackend.cpp (new) — the full NeuralNet::* backend implementation behind #ifdef USE_ONNX_BACKEND. It serializes the model graph via the existing OnnxModelBuilder into an ONNX ModelProto and feeds it to an Ort::Session; input filling and output decode are copied line-for-line from trtbackend, so the IO protocol and post-processing are identical to TensorRT. onnxProvider selects among cpu / openvino / cuda / tensorrt / migraphx / coreml / directml.
cpp/neuralnet/onnxmodelbuilder.{h,cpp} — graph inputs are now always declared in OpenVINO EP required order.
DirectML EP wiring — __has_include-guarded dml_provider_factory.h plus DisableMemPattern + single-threaded execution, Windows only.
OpenVINO EP options — per-thread device_type selection (onnxOpenVINODeviceType + onnxOpenVINODeviceTypeThread<N>, device indices map to a .N suffix), NumStreams/Precision/CacheDir/DeviceConfig_*, and SetIntraOpNumThreads(1) scoped to this provider.
Scale-8 — applyScale8ToReduceActivations applied by default, with an onnxSkipScale8 opt-out (NPU models).
cpp/CMakeLists.txt — new ONNX branch linking ORT + protobuf + zlib.
cpp/program/setup.cpp — registers the onnx backend prefix; transformer trunks default to NHWC.
cpp/main.cpp — version-info #elif USE_ONNX_BACKEND.
cpp/program/gtpconfig.cpp — per-thread device config (onnxDeviceToUseThread<N> etc.).
cpp/configs/gtp_example.cfg — ONNX settings example block.
Compiling.md — per-EP build docs plus an execution-provider support matrix.
CI — a single .github/workflows/onnx-backend.yml matrix workflow (CPU Windows/Linux, DirectML, OpenVINO, TensorRT in an NGC container) with runtests, backend verification, caches, and self-contained artifacts.

Extensibility:

The architecture is provider-agnostic: onnxProvider selects the EP and adding a future EP requires no code change — only config. New onnx* config keys are auto-recognized by the onnx prefix in setup.cpp, so provider-specific options can be added without touching the backend core.

How To Use

Prebuilt binaries:

Windows x64 (OpenVINO EP + CPU, all runtime DLLs bundled): https://github.com/seniorfish/KataGo/releases/tag/onnx-backend-v1
CI artifacts for each EP/platform: download from the ONNX backend build & test workflow (katago-<OS>-onnx-<ep>, e.g. katago-Linux-onnx-tensorrt).

The backend adds a family of onnx* keys to the config file (see configs/gtp_example.cfg for the full list). The essential ones:

Key	Default	Purpose
`onnxProvider`	`cpu`	Execution provider: `cpu/openvino/cuda/tensorrt/migraphx/coreml/directml`
`onnxOpenVINODeviceType`	`GPU`	OpenVINO device: `GPU/CPU/NPU/AUTO:.../MULTI:...`
`onnxOpenVINODeviceTypeThread<N>`	—	Per-thread override of `onnxOpenVINODeviceType`
`onnxOpenVINOCacheDir`	empty	Compile cache dir — speeds up cold starts
`onnxOpenVINONumStreams`	1	OpenVINO execution streams
`onnxOpenVINOPrecision`	empty	`FP16/FP32/ACCURACY`
`onnxTransformerNHWC`	`true`	NHWC trunk for transformer models
`onnxSkipScale8`	`false`	true = Skip `applyScale8ToReduceActivations` for speed

To use an Intel GPU you must set both:

onnxProvider = openvino
onnxOpenVINODeviceType = GPU

Tip:

if you are on a discrete Intel GPU rather than an integrated one, setting onnxOpenVINONumStreams = 2 noticeably improves throughput.
If you have both an Intel GPU and an NPU, you can use the NPU and iGPU together.

Benchmark

katago benchmark, -visits 60, default config otherwise noted. EloDiff is relative to the first row of each group.

Desktop — Intel Arc B580

OpenCL (baseline). Model b11c768h12nbt3tflrs-fson-silu.bin.gz (transformer, v17):

numSearchThreads	visits/s	nnEvals/s	nnBatches/s	avgBatchSize	time	EloDiff
2	14.61	14.11	14.06	1.00	41.7s	baseline
4	15.39	15.12	7.76	1.95	40.9s	−4
8	16.74	16.72	4.67	3.58	40.0s	−20

ONNX Runtime (OpenVINO EP). Model b11c768h12nbt3tflrs-fson-silu.bin.gz, onnxOpenVINONumStreams = 2, onnxTransformerNHWC = true:

numSearchThreads	visits/s	nnEvals/s	nnBatches/s	avgBatchSize	time	EloDiff
4	273.22	216.37	108.51	1.99	29.4s	baseline
8	362.81	290.80	73.26	3.97	22.2s	+87
12	394.78	321.04	54.35	5.91	20.5s	+102

Model kata1-zhizi-b40c768nbt-s11472M-d5982M.bin.gz (convnet, v15), onnxOpenVINONumStreams = 2:

numSearchThreads	visits/s	nnEvals/s	nnBatches/s	avgBatchSize	time	EloDiff
4	122.82	96.05	48.20	1.99	65.4s	baseline
8	156.49	125.23	31.55	3.97	51.6s	+62
12	169.99	134.65	22.75	5.92	47.7s	+66

Ablation (b11 transformer, numSearchThreads = 12), isolating NumStreams / transformerNHWC:

onnxOpenVINONumStreams	onnxTransformerNHWC	visits/s	nnEvals/s	avgBatchSize	time
1	true	126.72	103.04	5.91	63.9s
2	true	394.78	321.04	5.91	20.5s
2	false	185.39	150.18	5.91	43.7s

Both knobs matter: NumStreams = 2 ≈ 3× throughput vs 1, and transformerNHWC = true ≈ 2× vs false on this model.

Laptop — Intel Core Ultra 9 285H, Intel Arc 140T

ONNX Runtime (OpenVINO EP), numSearchThreads = 8:

numSearchThreads	visits/s	nnEvals/s	nnBatches/s	avgBatchSize	time	EloDiff
8	58.22	57.88	11.80	4.90	14.9s	baseline

More conditions and benchmark results are below.

Add a new USE_BACKEND=ONNX option that runs inference through ONNX Runtime, reusing the existing OnnxModelBuilder graph emitter (shared with the TensorRT backend) so the IO protocol and output decode are identical to TensorRT. Only onnxbackend.cpp is new; the rest is wiring (CMake, setup, version info, config keys, example config). The backend supports any ONNX Runtime execution provider. It is primarily useful for running KataGo on non-NVIDIA accelerators that have an EP, e.g. Intel GPUs/NPUs via the OpenVINO EP. Because the official prebuilt ONNX Runtime packages do not ship those EPs, the Compiling.md section documents building ONNX Runtime from source with the desired provider enabled. onnxmodelbuilder.cpp: declare graph inputs in consumption order (InputSpatial, InputGlobal, InputMask) rather than the previous InputMask-first order. This is purely cosmetic for backends that bind inputs by name (TensorRT), but is required for the OpenVINO EP, which builds its name->index map from declaration order while the ORT runtime feeds the EP kernel inputs in consumption order -- with InputMask first the EP misroutes the mask tensor into the InputSpatial port.

onnxmodelbuilder: use Pow(x,2.0) instead of Mul(x,x) for RMSNorm square OpenVINO RMSFusion (rms_fusion.cpp:38) matches Power(x, const(2)) but not Mul(x,x). Without this, all 66 RMSNorm nodes in the b11c768 transformer run as unfused ReduceMean->Sqrt->Div chains. Benchmark (Arc B580, NHWC, numStreams=2, 10 threads, 800 visits): Before: 347.49 visits/s 296.04 nnEvals/s After: 416.52 visits/s 353.14 nnEvals/s (+19.9%) Also add KATAGO_DUMP_ONNX env-var debug aid to dump the serialized ONNX model before session creation. Co-Authored-By: Claude <noreply@anthropic.com> @

seniorfish · 2026-08-01T04:52:14Z

Prebuilt binaries with ONNX Runtime + OpenVINO 2026.2.1 for Intel GPUs: https://github.com/seniorfish/KataGo/releases/tag/onnx-backend-v2 ~20% faster than v1 on transformer models (numStreams=2, 10 threads, 800 visits, 347.49 visits/s -> 416.52 visits/s).

kaorahi · 2026-08-01T07:26:36Z

Thank you for sharing your work! For some reason, it runs at only about 36% of the speed of #1171 in my environment (Intel Core Ultra 7 255U, Linux).

OpenCL: 25.16 visits/s with kata1-b18c384nbt
PR 1171 (iGPU): 76.33
PR 1171 (NPU): 87.30
PR 1171 + 3ed8922 (NPU + iGPU): 142.14
PR 1222 (NPU): 31.52

Also, what do you think about supporting the combined use of the NPU and iGPU? Simply setting onnxOpenVINODeviceType = MULTI:NPU,GPU didn't make much difference. For a significant speedup, we may need "per-thread OpenVINO device and batch size configuration" like the patch linked above.

kaorahi · 2026-08-01T12:13:38Z

(1) katago benchmark fails with the older small model g170-b6c96-s175395328-d26788732.txt.gz when onnxOpenVINODeviceType = NPU. It works with CPU and GPU.

terminate called after throwing an instance of 'StringError'
  what():  Got nonfinite for policy sum

(2) It also fails with the model kata1-b18c384nbt-s9996604416-d4316597426.bin.gz when onnxOpenVINODeviceType = GPU. It works with CPU and NPU.

ERROR: Async bot thread failed: Got nonfinite for policy sum

(3) Removing modelDesc.applyScale8ToReduceActivations(); from cpp/neuralnet/onnxbackend.cpp makes NPU inference run at 4× the original speed (= 1.5× the speed of #1171).

PR 1222 - Scale8 (NPU): 129.25 visits/s with kata1-b18c384nbt

…LE8 subgraphs hurting NPU performance

seniorfish · 2026-08-01T14:47:26Z

@kaorahi, thank you. That's a very insightful observation. The patch has been committed. The new version gives me roughly a 4x performance improvement here as well (Core Ultra 9 285H reaching 54 visits/s, NPU).

Regarding your second issue, kata1-b18c384nbt-s9996604416-d4316597426 seems to run fine on my end, both on the B580 and the 285H, with onnxOpenVINODeviceType = GPU.
I'll test the first issue later.

As for your suggestion about supporting the combined use of the NPU and iGPU, I'll try to implement it.

Add per-thread OpenVINO device type and batch size config for multi-device inference - onnxOpenVINODeviceTypeThread<N>: assign different device types (CPU/GPU/NPU) to individual server threads, enabling simultaneous heterogeneous inference - onnxOpenVINODeviceConfig_<DEV>_<Option>: per-device-type EP option overrides (num_streams, precision, etc.) for fine-grained device tuning - nnMaxBatchSizeThread<N>: per-thread max batch size to prevent fast devices from being starved by slow devices sharing the same eval queue - extractShortDeviceName(): maps GPU.0/GPU.1/AUTO:MULTI:HETERO: strings to short device names for config lookup - Enhanced printDevices() to document the new multi-device config options Co-Authored-By: Claude <noreply@anthropic.com> @

seniorfish · 2026-08-02T16:01:12Z

@kaorahi, I referred to your implementation in 3ed8922 and implemented the combined use of the NPU and iGPU (even the CPU, but tests showed that its benefit was always negative).
The benchmark data is as follows.

for all
-model kata1-b18c384nbt-s9996604416-d4316597426.bin.gz, -t 4,6,8(or 6,8,10) -v 200(or 300)
Intel Core Ultra 285H(140T)
GPU only:
numSearchThreads =  4: 10 / 10 positions, visits/s = 109.65 nnEvals/s = 101.06 nnBatches/s = 50.34 avgBatchSize = 2.01 (18.5 secs) (EloDiff baseline)
numSearchThreads =  6: 10 / 10 positions, visits/s = 144.35 nnEvals/s = 132.80 nnBatches/s = 45.35 avgBatchSize = 2.93 (14.2 secs) (EloDiff +88)
numSearchThreads =  8: 10 / 10 positions, visits/s = 145.33 nnEvals/s = 135.35 nnBatches/s = 46.08 avgBatchSize = 2.94 (14.2 secs) (EloDiff +76)

NPU only:
numSearchThreads =  6: 10 / 10 positions, visits/s = 73.42 nnEvals/s = 67.51 nnBatches/s = 34.13 avgBatchSize = 1.98 (27.9 secs) (EloDiff baseline)
numSearchThreads =  8: 10 / 10 positions, visits/s = 80.82 nnEvals/s = 75.24 nnBatches/s = 37.95 avgBatchSize = 1.98 (25.6 secs) (EloDiff +18)
numSearchThreads = 10: 10 / 10 positions, visits/s = 79.81 nnEvals/s = 75.38 nnBatches/s = 38.07 avgBatchSize = 1.98 (26.2 secs) (EloDiff -4)

GPU + NPU with nnMaxBatchSize 6:2:
numSearchThreads =  4: 10 / 10 positions, visits/s = 139.54 nnEvals/s = 129.91 nnBatches/s = 113.74 avgBatchSize = 1.14 (14.5 secs) (EloDiff baseline)
numSearchThreads =  6: 10 / 10 positions, visits/s = 187.25 nnEvals/s = 170.89 nnBatches/s = 102.59 avgBatchSize = 1.67 (10.9 secs) (EloDiff +97)
numSearchThreads =  8: 10 / 10 positions, visits/s = 205.91 nnEvals/s = 194.57 nnBatches/s = 84.69 avgBatchSize = 2.30 (10.0 secs) (EloDiff +120)

GPU + NPU with nnMaxBatchSize 4:4:
numSearchThreads =  6: 10 / 10 positions, visits/s = 119.32 nnEvals/s = 107.19 nnBatches/s = 72.02 avgBatchSize = 1.49 (25.6 secs) (EloDiff baseline)
numSearchThreads =  8: 10 / 10 positions, visits/s = 237.83 nnEvals/s = 216.06 nnBatches/s = 86.79 avgBatchSize = 2.49 (12.9 secs) (EloDiff +249)
numSearchThreads = 10: 10 / 10 positions, visits/s = 259.90 nnEvals/s = 237.61 nnBatches/s = 73.09 avgBatchSize = 3.25 (11.9 secs) (EloDiff +272)

GPU + NPU + CPU 4:4:1:
numSearchThreads =  6: 10 / 10 positions, visits/s = 121.21 nnEvals/s = 109.64 nnBatches/s = 90.64 avgBatchSize = 1.21 (25.2 secs) (EloDiff baseline)
numSearchThreads =  8: 10 / 10 positions, visits/s = 128.33 nnEvals/s = 116.71 nnBatches/s = 64.17 avgBatchSize = 1.82 (23.9 secs) (EloDiff +6)
numSearchThreads = 10: 10 / 10 positions, visits/s = 181.46 nnEvals/s = 167.24 nnBatches/s = 69.23 avgBatchSize = 2.42 (17.0 secs) (EloDiff +126)

pre-builts: https://github.com/seniorfish/KataGo/releases/edit/onnx-backend-v4

kaorahi · 2026-08-02T18:30:53Z

I appreciate your onnx-backend-v4. It set a new record in my environment. katago benchmark recommended numSearchThreads = 5.

PR 1222 v4 (NPU + iGPU): 189.89 visits/s with kata1-b18c384nbt

config:

numNNServerThreadsPerModel = 2
nnMaxBatchSizeThread0 = 1
onnxOpenVINODeviceTypeThread0 = NPU
onnxOpenVINODeviceTypeThread1 = GPU

By the way, I noticed a few things.

Should the tag onnx-backend-v3 point to 4f86efa instead of 815378d ?
Is this the correct URL for the prebuilt binaries (without edit) ? https://github.com/seniorfish/KataGo/releases/tag/onnx-backend-v4
gpu_optimized.cfg and gtp_gpu_npu_example.cfg are missing from cpp/configs/. Is this intentional?

seniorfish · 2026-08-03T01:53:53Z

Fixed, it was just a tag pointing issue, the attachments are all correct. gpu_optimized.cfg and gtp_gpu_npu_example.cfg are merely two preset configurations for Intel hardware. To keep the commit clean, and since all newly added options are explained in gtp_example.cfg, I didn't include them in cpp/configs/.

kaorahi · 2026-08-04T09:39:38Z

This PR does not currently support HumanSL models, so I'm trying this patch to add support: d57c9bc. During analysis, I prefer to view a policy comparison between 5 kyu and 1 dan to learn from it.

lightvector · 2026-08-04T15:32:16Z

I added to the existing trt onnx builder support for HumanSL: 5ae42f8

Does that result in this having HumanSL support once it's rebased?

Starting to review a little bit! Regarding alignInputsToConsumptionOrder - why is this needed? If inputs in consumption order is now required, can we just simply document that it's required at the site that makes the inputs, and do the ordering unconditionally, rather than adding an extra argument?

Also, what is "consumption order" - for example if we enter the !requireExactNNLen branch (onnxmodelbuilder.cpp just after alignInputsToConsumptionOrder) will it mean that now the mask is actually referenced first in the graph and thus the new ordering will be bad again? Is this backend vendor doing something stupid and interface-violating that we need to work around Is there a way we can make this less fragile? Presumably it would be worth testing that all input order combinations work - when there is a mask, when there is no mask (require exact nn len), and with and without HumanSL's metadata encoder?

…VINOCacheDir

seniorfish · 2026-08-04T16:34:30Z

It has been confirmed that onnxOpenVINOEnableNPUFastCompile (enable_npu_fast_compile) has been deprecated by ONNX Runtime. This option has no effect, so it has been removed.

lightvector

Thanks for the work. Took a first initial pass at review, left comments.

lightvector · 2026-08-04T16:21:20Z

   usingFP16Mode(useFP16Mode),
   numThreads(numThr),
   gpuIdxByServerThread(gpuIdxByServerThr),
+   maxBatchSizeByServerThread(


If we're going to do per-server-thread batch sizes, please verify that this is threaded everywhere needed? For example, maybeWarmupComputeHandle uses batch sizes too and may need to be run with the same batch size as post-warmup if a backend relies on it.

lightvector · 2026-08-04T16:25:02Z

    throw StringError("NNEvaluator::setNumThreads called when threads were already running!");
  numThreads = (int)gpuIdxByServerThr.size();
  gpuIdxByServerThread = gpuIdxByServerThr;
+  maxBatchSizeByServerThread.assign(numThreads, maxBatchSize);


Is it a bit weird that setting the number of threads also resets all batch sizes, even for the server threads that were configured to demand a particular batch size?

Might deserve at least a comment or some documentation?

lightvector · 2026-08-04T16:32:29Z

+    // Skip applyScale8ToReduceActivations() for ONNX backend:
+    // NPU/ONNX Runtime execution providers don't benefit from the fp16
+    // dynamic-range workaround; removing this avoids MISH_SCALE8 subgraphs
+    // that block operator fusion and cost ~25% extra ops per activation.
+    // modelDesc.applyScale8ToReduceActivations();
+  }


Why don't they benefit from the fp16 range workaround?

Would you be willing to try a selfplay game on large boards like 39x39 or 50x50 using the bigger board sizes compile flag (with low visits so that it actually runs in a reasonable time) and see if it works or if it blows up or something, on each of a few different nets, like a big transformer and a convnet? It will play terribly poorly of course but the point is to check what the extremal behavior actually is. The fp16 range reductions from scale 8 were addressing over-large activations that used to happen on b28 on large boards and even happened on Friday9i's large board finetuned net https://katagotraining.org/extra_networks/ (although less frequently) at large sizes, and that IIRC also extremely rarely happened on 19x19 or barely-larger than 19x19 sizes (23x23, etc) on existing nets. This was very rare and hard to see in testing because it required especially unusual positions, but when it happened and crashed katago it was of course very annoying for users. (Testing on a large board size, even if the net plays horribly, is one way probing how much "headroom" there is).

lightvector · 2026-08-04T16:33:59Z

+    }
+
+    Ort::SessionOptions sessionOpts;
+    sessionOpts.SetIntraOpNumThreads(1);


What's this thread setting intended for? An ai-assisted pre-review pass for me flags this as potentially killing the performance of the CPU backend for onnx runtime (although I haven't verified this)

lightvector · 2026-08-04T16:36:36Z

+  // OpenVINO/CUDA pick FP16 themselves), so the global useFP16 flag is intentionally ignored.
+  (void)useFP16Mode;


This seems misleading. Can we wire it up so that if the user sets it to false it uses it as false?

Less ideally, if there is no easy way to wire it up, then it should perhaps fail loud if the user requests it as false. It shouldn't be the case that the user requests one thing and it silently gets ignored.

lightvector · 2026-08-04T16:41:16Z

+
+      // --- Build EP option map ---
+      std::unordered_map<std::string, std::string> openvinoOpts;
+      openvinoOpts["device_type"] = threadDeviceType;


Tossing over the wall a comment from an AI-assisted pre-review that I'm not an expert on but looks plausible to me that might need some consideration. If you think this is a false positive then feel free to refute/reject it.

Per the current OpenVINO EP docs, device_id isn't an accepted provider option any more — device selection is via device_type itself ("GPU.0", "GPU.1", etc.): https://onnxruntime.ai/docs/execution-providers/OpenVINO-ExecutionProvider.html. So on ORT builds that validate V2 provider options, any config that reaches this line (onnxOpenVINODeviceId, or onnxDeviceToUseThreadN with index > 0) would fail at session creation, and the retry at L411 wouldn't help since device_id isn't in the stripped-key set. Since this only triggers with multiple devices of the same type, it may never have been exercised in your testing. Suggest mapping the index into the device string instead — if threadDeviceType is a bare "GPU"/"NPU" without a .N suffix, emit device_type = "GPU." + idx — and dropping device_id (or keeping onnxOpenVINODeviceId as sugar that appends the suffix). Would be good to confirm with a two-GPU or GPU+iGPU config that per-thread selection actually lands on the intended devices.

lightvector · 2026-08-04T16:45:55Z

+  set_source_files_properties(${ONNX_PROTO_SRCS} PROPERTIES COMPILE_OPTIONS "-w")
+  target_sources(katago PRIVATE ${ONNX_PROTO_SRCS} neuralnet/onnxmodelbuilder.cpp)
+  target_include_directories(katago SYSTEM PRIVATE ${CMAKE_CURRENT_BINARY_DIR} ${Protobuf_INCLUDE_DIRS})
+  target_link_libraries(katago ${Protobuf_LIBRARIES})


Double check whether protobuf is linked in a portable way here? See TensorRT's protobuf linking for a case that required a fix for windows.

lightvector · 2026-08-04T16:52:09Z

And see of course the earlier comment I left about alignInputsToConsumptionOrder both code-style-wise and in terms of the potential fragility, if there's a way we can work around it better for the provider that's mis-conforming.

seniorfish · 2026-08-04T18:18:17Z

@lightvector thanks for the review comments on alignInputsToConsumptionOrder. I ran a dedicated investigation (drafted by an agent, then carefully checked myself), and the full findings with reproduction scripts are at: https://github.com/seniorfish/KataGo/tree/research/onnx-input-order/research/onnx-input-order

In short: this is an input-binding bug inside the ONNX Runtime OpenVINO EP, present since ORT v1.23.0 (introduced by commit dfc27cd7c7e). KataGo feeds inputs by name through the ORT Run API, which is the conforming interface; the mismatch reproduces on our stack (self-built dev ORT 1.29 + OpenVINO 2026.2). So I'll keep the workaround for now, but will address your code-style point by making the ordering unconditional and documenting the requirement at the input-construction site, and I am also considering preparing an upstream bug report to microsoft/onnxruntime with our repro.

Per review feedback, drop the opt-in boolean: the ONNX graph's inputs are always declared in the fixed order InputSpatial, InputGlobal, InputMask, which the OpenVINO EP under ONNX Runtime requires. The claim that this is a no-op for name-bound backends (TRT/CUDA/CoreML) was not verified, so it is removed from the comments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

seniorfish · 2026-08-04T18:49:06Z

Implementing per-server-thread batch sizes is relatively complex and requires further verification. plan to treat it as an independent feature and may submit a separate PR for it.

The nnMaxBatchSizeThread<N> feature (introduced in 815378d) modified the shared NNEvaluator core (nneval.cpp/h, setup.cpp) and was unrelated to the ONNX backend itself, widening the review surface of this PR. Remove it to keep the PR focused on the ONNX backend; a future dedicated PR can revisit per-thread batch sizing, addressing the warmup-consistency and setNumThreads reset semantics concerns raised in review. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- Fail loudly when the global useFP16 flag is requested (true): inference precision is controlled by the ONNX Runtime execution provider, so a true request cannot be honored and must not be silently ignored - Link protobuf via protobuf::libprotobuf for portability, defaulting Protobuf_USE_STATIC_LIBS to TRUE since a from-source ORT build bundles a static libprotobuf (avoids PROTOBUF_USE_DLLS breaking the Windows link) - Default onnxTransformerNHWC to true, matching the TensorRT backend's trtTransformerNHWC default; only affects transformer trunks, which convnets ignore - Document the useFP16 behavior and the new transformerNHWC default in gtp_example.cfg Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The OpenVINO execution provider runs inference in FP16 by default (the graph is emitted fp32, but the EP downcasts internally), so the FP16 dynamic-range workaround applies here. The previous skip left large-board convnet activations within ~2.2x of the FP16 overflow limit. Measured on M2 (28-block, 512-channel pure convnet) at 50x50, with and without scale8: - Main-path activation peak: 29284 -> 3660. Without scale8 that is 44.7% of the FP16 max (65504), i.e. only 2.24x headroom, and value_head.linear2 is the hotspot; with scale8 it is 5.6% of FP16 max, 17.9x headroom. - Real FP16 GPU inference: on value-range-20 random inputs the no-scale8 graph emits NaN across every output (5000 NaN elements in OutputPolicy); the scale8 graph stays finite and remains an exact 8x rescale of outputs. - Overflow-threshold scan: no-scale8 starts overflowing at ~15x input value range, i.e. only extreme positions, matching the real-world reports of b28 convnets overflowing and crashing on large boards, which is exactly what scale8 was introduced to prevent. Trade-off: MISH_SCALE8 subgraphs block OpenVINO's fused-Mish optimization (~2x slower FP16 inference on large-board convnets). Add onnxSkipScale8 (default false) to opt out for FP32 precision or small-board/transformer workloads where FP16 overflow is not a practical risk. Distributed selfplay (contribute) always forces onnxSkipScale8 to false: FP16-overflow NaN rows must never be uploaded to the shared training set. Match the TensorRT, CUDA, and OpenCL backends, which apply this unconditionally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The ONNX Runtime intra-op thread pool was pinned to a single thread unconditionally, for every execution provider. That significantly hurts the CPU provider, which relies on ORT's default multi-threaded intra-op execution. Only the OpenVINO provider wants the pin: the EP runs the graph nodes itself and manages its own inference threads via the num_of_threads provider option, so ORT's intra-op pool is left with only the few EP-external nodes. With one ORT session per nn-server thread, leaving the default intra-op thread count would oversubscribe the CPU with N x M worker pools. Other providers now use the ORT default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…device_type The OpenVINO EP's device_id provider option is deprecated in ONNX Runtime and only accepts the bare device names CPU/GPU/NPU - any other value throws at session creation. The backend was passing a numeric per-thread device index to it, so a multi-GPU setup (deviceIdxForThread > 0 via the gpuToUse*/deviceToUse* config keys) would fail, as would the documented onnxOpenVINODeviceId = 0 example. Now the per-thread device index is instead appended to device_type as an OpenVINO device suffix (GPU -> GPU.1), which is how the OpenVINO EP selects among multiple devices. The onnxOpenVINODeviceId config key and the device_id provider option are removed, and the config docs updated. Note: this was verified against the ONNX Runtime source (OpenVINO EP provider-option parsing) and with the standard build/tests only - I have no dual-GPU hardware here, so the multi-device path has not been exercised on real hardware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Adds .github/workflows/onnx-build.yml, which builds the ONNX backend (USE_BACKEND=ONNX) on Windows against a from-source ONNX Runtime carrying the OpenVINO execution provider (official prebuilt ORT packages do not ship the OpenVINO EP), runs `katago runtests`, and uploads a self-contained runnable directory as an artifact for easy download. Design notes: - Triggered only by push to feature/onnx-backend and by workflow_dispatch, so it does not run on master after the PR merges and never burns upstream CI minutes on the ~1h ORT build; maintainers may edit or delete the file freely. config/ and doc-only changes are filtered out via paths-ignore. - Runtime dependencies (ORT install tree + protobuf + zlib) are cached under deps/install; only the first run compiles ONNX Runtime. - Build recipes mirror the locally-verified scripts (build_ort.bat / build_katago_onnx.bat): OpenVINO 2026.2.1 toolkit download, ORT build.py with --use_openvino GPU, protobuf staged from the ORT build tree, and a static zlib. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The first CI run failed at "Checkout ONNX Runtime source": the workflow pinned onnxruntime to the ref v1.29.0, but that release tag does not exist yet (VERSION_NUMBER is 1.29.0 while upstream is still pre-release). Pin instead to the exact commit the ONNX backend was verified against locally (7e76a52398, verified to exist on upstream master via the GitHub API), so the CI source matches the locally-tested snapshot byte-for-byte rather than drifting with master. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Brings in upstream's three new commits: 978acec (serialize TRT engine builds), 5ae42f8 (HumanSL support in the onnx/trt path), 2292f25 (TRT automatic workspace sizing). Conflict resolution in onnxmodelbuilder.cpp: keep this branch's fixed input declaration order (InputSpatial, InputGlobal, InputMask) required by the OpenVINO EP, and slot HumanSL's InputMeta declaration between InputGlobal and InputMask (InputMeta is an NC11 vector input consumed at the trunk's initial bias, alongside InputGlobal). Position is empirical and pending verification against an actual HumanSL model. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Verified after merging upstream's HumanSL support: b18c384nbt-humanv0 (model v15, metaEncoderVersion > 0) runs end-to-end on the OpenVINO EP with inputs declared as InputSpatial, InputGlobal, InputMeta, InputMask (113 visits/s, no NaN). Record the verified order in the comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The second CI run built ORT and staged protobuf (both took ~1h) then failed at "Build zlib (static)": cmake --install resolved the install prefix to the literal string "D:/a/KataGo/KataGo/$env:GITHUB_WORKSPACE/deps/install/zlib" - unquoted $env:GITHUB_WORKSPACE was not expanded by pwsh here (quoted "$env:RUNNER_TEMP" and cmd's %CD% both worked). The same broken pattern was also present in the "Configure KataGo" step, which never got to run. Replace every $env:GITHUB_WORKSPACE in CMake -D values with the GitHub expression ${{ github.workspace }}, which the runner expands before handing the line to the shell, independent of the shell's variable semantics. Also extend on.push.branches to include ci/onnx-windows so this fix can be validated on its own branch without touching the PR branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

After validating the fix on ci/onnx-windows, scope on.push.branches back to feature/onnx-backend so the workflow does not fire on the temporary branch once it is merged back. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

seniorfish · 2026-08-05T09:09:50Z

@lightvector, Thanks for the initial review — all 7 comments are now addressed:

Per-thread max batch sizes — removed entirely to keep this PR focused.
scale8 — restored by default, with an opt-out onnxSkipScale8 (default off = scale8 applied). Verified on a 50×50 board with an M2 net, see c2e5552
SetIntraOpNumThreads(1) — now scoped to the OpenVINO EP only.
useFP16Mode — now fails loudly
device_id — dropped; the per-thread device index is now folded into device_type (e.g. "GPU.1"). Verified against the ORT/OpenVINO source (no dual-GPU hardware here to test live).
protobuf — linked portably via protobuf::libprotobuf + Protobuf_USE_STATIC_LIBS (same fix as the TensorRT path).

Also merged upstream master, the HumanSL model seems to work fine for me (513 visits/s on Arc B580), and added a CI workflow that builds the ONNX+OpenVINO backend on push and uploads a runnable artifact.

…on ignored device index - LoadedModel.scale8Resolved: plain bool under the existing mutex instead of std::atomic. All accesses were already under scale8Mutex, so the atomic added no synchronization; the mutex is what establishes the happens-before between the scale8 modelDesc write and each thread's subsequent graph build. Remove the now-unused <atomic> include. - ComputeContext.requireExactNNLenStored was a dead field (declared and initialized, never read); requireExactNNLen is already passed per-handle. Delete it. - OpenVINO device_type: when a nonzero device index is configured but the device_type is already a composite/qualified string (AUTO:/MULTI:/HETERO:), the index was silently ignored. Log a warning pointing at the per-thread onnxOpenVINODeviceTypeThread<N> override, and document the interaction in gtp_example.cfg. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…whitelist to a table Only the OpenVINO EP is verified upstream; the other onnxProvider options are experimental code paths. Make that explicit everywhere a provider is listed, with Compiling.md as the single source of truth: - Compiling.md: add an "Execution provider support matrix" (Verified / Experimental / Needs work) with platform, ORT build flag, and runtime deps per provider; reword the intro to say the backend "selects" rather than "supports" several providers. - gtp_example.cfg: annotate onnxProvider options with the same verification status and point at the matrix. - onnxbackend.cpp: hoist the hard-coded provider whitelist into a kKnownProviders table (shared by validation and the unknown-provider error, which now cites the matrix), note in the file header and printDevices that only OpenVINO is verified upstream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- onnx-cpu-build.yml: always-on regression job (pull_request + push to master + workflow_dispatch) that downloads the official prebuilt ONNX Runtime CPU package (no from-source ORT build: ~5 min vs ~1-3h for the OpenVINO job), builds static protobuf 3.21 (/MD) and zlib only on cache miss, builds with USE_BACKEND=ONNX, runs runtests, and verifies the backend via `katago version`. Complements the branch-scoped OpenVINO job with cheap always-on coverage. - CMakeLists.txt: ONNX include detection falls back to the flat include/ layout of official prebuilt ORT packages, alongside the include/onnxruntime/ layout of a from-source install. Note: ci/onnx-windows is temporarily in the push trigger so the workflow can be exercised before it exists on the default branch; drop it before merging to master. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- Include dml_provider_factory.h on Windows (headers come from the Microsoft.ML.OnnxRuntime.DirectML NuGet package), guarded by __has_include: the stock CPU prebuilt does not ship this header, so builds against it still compile and the DirectML provider fails at runtime with a clear message. - Add "directml" to the provider whitelist and to the error message listing. - In ComputeHandle, enable the DirectML EP via OrtDmlApi obtained from GetExecutionProviderApi("DML", ...): the plain export in dml_provider_factory.h is deprecated. Per the ORT DirectML docs the EP requires DisableMemPattern and sequential execution mode, both set here. device_id maps from the per-thread GPU index. - DirectML EP is experimental (unverified upstream); on non-Windows platforms the provider fails loudly, matching the CoreML pattern. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…actions Replace onnx-build.yml (OpenVINO, from-source ORT) and onnx-cpu-build.yml (CPU, prebuilt ORT) with a single onnx-backend.yml that drives a matrix of execution providers. The only real difference between providers - how ONNX Runtime is obtained - is encapsulated in .github/actions/onnx-prepare-ort (prebuilt zip / DirectML NuGet / from-source build), and the shared configure-build-test-stage pipeline lives in .github/actions/onnx-build-katago. - Matrix rows today: cpu + directml (fast, prebuilt/nuget, minutes) under build-fast, openvino (from-source, 1-3h) under build-slow. - Trigger policy by tier: fast jobs run on PR + master push + dispatch so the ONNX backend keeps a cheap always-on regression guard; slow from-source jobs run only on workflow_dispatch so they never burn upstream CI minutes. - Adding a backend = one matrix row + a fetch/build recipe in onnx-prepare-ort. - GitHub-hosted runners have no GPU, so from-source jobs verify build + EP wiring only; real GPU inference must be validated on a GPU machine. - ci/onnx-windows stays in push.branches as a temporary trigger (dispatch needs the file on the default branch); remove it before merging to master. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- build-fast: + ubuntu-latest CPU (prebuilt ORT tgz) - onnx-prepare-ort: prebuilt mode now fetches the Linux tgz (zip is Windows-only) - onnx-build-katago: Linux staging rewrites DT_RUNPATH to $ORIGIN via patchelf so the downloaded binary finds its sibling libonnxruntime.so*, matching the Windows DLL layout Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- build-tensorrt: dispatch-only job in nvcr.io/nvidia/tensorrt:25.03-py3 (CUDA 12.8 + TensorRT 10.9, the combo ORT is tested against). Bootstrap installs git/node20/ninja, cmake >= 3.28 (NGC ships 3.27), and git safe.directory for the docker-mounted workspace; then onnx-prepare-ort builds ORT with --use_tensorrt --use_cuda, limited to sm_89 (RTX 4090) instead of ORT's 10-arch default. - onnx-build-katago: use $GITHUB_WORKSPACE (resolves in container jobs where github.workspace points at the runner path), and don't assume sudo for patchelf. - cache zlib/protobuf + ORT explicitly (actions/cache@v4's post-save is unreliable in composite actions), keyed identically to the restore steps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

seniorfish · 2026-08-06T01:55:45Z

I've successfully added support for several execution-providers: OpenVINO, DirectML (~20 v/s on transformer models, outperforming OpenCL), and CPU on Windows, as well as TensorRT (though performance is currently underperforming at ~17 v/s on a 4090) and CPU on Linux.

All the binaries can be found in the artifacts. This demonstrates the great flexibility ONNX Runtime offers in integrating various backends. I'm currently working on adding Android support.

seniorfish · 2026-08-06T13:56:11Z

Added experimental WebAssembly (Emscripten) support to the ONNX backend to run KataGo's transformer network in the browser. Verified decent execution speeds using a local Go board web demo. Further development is ongoing in the feature branch: https://github.com/seniorfish/KataGo/tree/feature/onnx-wasm

seniorfish · 2026-08-06T17:46:02Z

I managed to get KataGo running in the browser by combining onnx-wasm with Sabaki. The code has been pushed to the feature/web branch. Please refer to README-web.md for build steps—feedback is welcome!

seniorfish changed the title ~~Add ONNX Runtime neural-net backend (OpenVINO EP)~~ Add ONNX Runtime neural-net backend (OpenVINO EP, v1.17.0) Jul 31, 2026

Merge remote-tracking branch 'origin/master' into feature/onnx-backend

30a01b8

seniorfish changed the title ~~Add ONNX Runtime neural-net backend (OpenVINO EP, v1.17.0)~~ Add ONNX Runtime neural-net backend (OpenVINO EP, v1.17.1) Jul 31, 2026

Skip applyScale8ToReduceActivations in ONNX backend to avoid MISH_SCA…

4f86efa

…LE8 subgraphs hurting NPU performance

seniorfish changed the title ~~Add ONNX Runtime neural-net backend (OpenVINO EP, v1.17.1)~~ Add ONNX Runtime backend support (OpenVINO EP, Intel GPU & NPU, v1.17.1) Aug 2, 2026

seniorfish changed the title ~~Add ONNX Runtime backend support (OpenVINO EP, Intel GPU & NPU, v1.17.1)~~ Add ONNX Runtime backend support (OpenVINO EP, Intel GPU & NPU, Windows, v1.17.1) Aug 2, 2026

Remove deprecated onnxOpenVINOEnableNPUFastCompile; document onnxOpen…

cac464b

…VINOCacheDir

seniorfish marked this pull request as ready for review August 4, 2026 16:16

lightvector reviewed Aug 4, 2026

View reviewed changes

seniorfish and others added 4 commits August 5, 2026 09:38

seniorfish and others added 7 commits August 5, 2026 11:58

Restrict CI trigger back to the PR branch

0909331

After validating the fix on ci/onnx-windows, scope on.push.branches back to feature/onnx-backend so the workflow does not fire on the temporary branch once it is merged back. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Looong01 mentioned this pull request Aug 5, 2026

[Intel NPU/GPU] Add Windows & Linux Intel NPU & GPU support #1171

Open

seniorfish and others added 3 commits August 5, 2026 17:53

seniorfish force-pushed the feature/onnx-backend branch from 575d797 to 22960c0 Compare August 5, 2026 10:49

seniorfish and others added 4 commits August 5, 2026 21:05

seniorfish changed the title ~~Add ONNX Runtime backend support (OpenVINO EP, Intel GPU & NPU, Windows, v1.17.1)~~ Add ONNX Runtime backend support (OpenVINO / DirectML / TensorRT / CPU, Windows & Linux) Aug 6, 2026

seniorfish requested a review from lightvector August 6, 2026 05:13

Merge upstream master (v1.17.2) into feature/onnx-backend

db0a622

		// OpenVINO/CUDA pick FP16 themselves), so the global useFP16 flag is intentionally ignored.
		(void)useFP16Mode;

Conversation

seniorfish commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add ONNX Runtime neural-net backend

Summary

What changed

How To Use

Benchmark

Desktop — Intel Arc B580

Laptop — Intel Core Ultra 9 285H, Intel Arc 140T

Uh oh!

seniorfish commented Aug 1, 2026

Uh oh!

kaorahi commented Aug 1, 2026

Uh oh!

kaorahi commented Aug 1, 2026

Uh oh!

seniorfish commented Aug 1, 2026

Uh oh!

seniorfish commented Aug 2, 2026

Uh oh!

kaorahi commented Aug 2, 2026

Uh oh!

seniorfish commented Aug 3, 2026

Uh oh!

kaorahi commented Aug 4, 2026

Uh oh!

lightvector commented Aug 4, 2026

Uh oh!

seniorfish commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lightvector left a comment

Choose a reason for hiding this comment

Uh oh!

lightvector Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

lightvector Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

lightvector Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

lightvector Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

lightvector Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

lightvector Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

lightvector Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

lightvector commented Aug 4, 2026

Uh oh!

seniorfish commented Aug 4, 2026

Uh oh!

seniorfish commented Aug 4, 2026

Uh oh!

seniorfish commented Aug 5, 2026

Uh oh!

seniorfish commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seniorfish commented Aug 6, 2026

Uh oh!

seniorfish commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

seniorfish commented Jul 31, 2026 •

edited

Loading

seniorfish commented Aug 4, 2026 •

edited

Loading

seniorfish commented Aug 6, 2026 •

edited

Loading