Improve ONNX inference performance by adopting the OrtValue API

[rosebyte]

OnnxTransformer still invokes ONNX Runtime through the legacy NamedOnnxValue API. ONNX Runtime now recommends its OrtValue API, which produces less garbage, supports reusable input and output buffers, and can improve inference performance.

ML.NET currently references ONNX Runtime 1.23.2. The required C# API was introduced in 1.16, so upgrading ONNX Runtime is not a prerequisite for this work.

Motivation

The current inference path:

• Creates NamedOnnxValue inputs for every row.
• Copies numeric inputs using data.ToArray().
• Creates intermediate string[] and DenseTensor<string> instances for string inputs.
• Lets ONNX Runtime allocate output values for every invocation.
• Converts and copies results into ML.NET VBuffer instances.
• Performs one ONNX Runtime invocation per IDataView row.

These allocations can increase GC pressure and latency, particularly for small models and variable-length string inputs. This is related to #6620.

ONNX Runtime describes NamedOnnxValue as a legacy API and reports that OrtValue can provide substantial performance and allocation improvements in some scenarios.

Proposal

Benchmark and, where beneficial, migrate OnnxTransformer to the OrtValue API:

1. Use OrtValue.CreateTensorValueFromMemory with reusable managed buffers for numeric inputs.
2. Preallocate and reuse numeric output tensors when their shapes are fixed.
3. Access dynamically allocated outputs through GetTensorDataAsSpan<T>().
4. Populate native string tensors directly from ML.NET values, avoiding intermediate string[] and DenseTensor<string> allocations where possible.
5. Reuse input names, output names, shapes, and RunOptions rather than recreating them for every row.
6. Dispose native values deterministically and ensure managed buffers are not left pinned.
7. Preserve the existing behaviour for dynamic shapes, strings, maps, sequences, and other supported ONNX value types.

String tensors will still require UTF-16 to UTF-8 conversion and copying, but the new API should allow us to remove several intermediate managed allocations.

Benchmark plan

Compare the existing and OrtValue paths using ONNX Runtime 1.23.2 first, so the effect of the API migration can be measured independently.

Cover at least:

• Fixed-shape numeric inputs and outputs.
• Dynamic-shape numeric inputs or outputs.
• Fixed and variable-length string inputs.
• Single and multiple input/output models.
• Small models where managed overhead is significant.
• Longer-running models to check for regressions.

Measure:

• Mean inference latency and throughput.
• Allocated bytes per inference.
• Gen 0, Gen 1, and Gen 2 collections.
• Performance under repeated scoring.
• Memory and handle stability over prolonged execution.

The latest stable ONNX Runtime version should be evaluated separately. Any dependency update should ideally be made in a separate change so its kernel and execution-provider improvements can be measured independently.

Acceptance criteria

• Benchmarks document the performance and allocation differences.
• The OrtValue path is used where it improves performance without breaking existing behaviour.
• Reusable buffers are used for fixed-shape numeric tensors.
• Intermediate string arrays and tensors are removed where the API permits.
• Existing ONNX transformer tests continue to pass.
• Dynamic shapes and all currently supported input and output types remain compatible.
• Repeated inference does not leak native memory or leave managed buffers pinned.

Out of scope

• Changing IDataView to perform automatic batching.
• Introducing device-backed VBuffer values.
• Keeping complete pipelines resident in GPU memory.

Those changes may provide larger GPU throughput improvements, but require broader ML.NET API and pipeline design work. OrtIoBinding already has an official C# binding and can be investigated separately for such scenarios.

References

• ONNX Runtime C# OrtValue guidance
• ONNX Runtime C# OrtValue implementation
• NamedOnnxValue API documentation
• ML.NET GC pressure report
• Related ONNX Runtime investigation

[github-actions]

🔍 No similar issues found with high confidence.

---

AI-powered similar issue detection