Skip to content

Commit e0954da

Browse files
ChSonnabendChSonnabend
committed
Init benchmark task for O2AT6
1 parent 177407e commit e0954da

3 files changed

Lines changed: 342 additions & 0 deletions

File tree

‎Tutorials/ML/CMakeLists.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,3 +19,8 @@ o2physics_add_dpl_workflow(apply-ml-selection
1919
SOURCES applyMlSelection.cxx
2020
PUBLIC_LINK_LIBRARIES O2Physics::AnalysisCore O2Physics::MLCore
2121
COMPONENT_NAME AnalysisTutorial)
22+
23+
o2physics_add_dpl_workflow(benchmark-onnx-model
24+
SOURCES benchmarkOnnxModel.cxx
25+
PUBLIC_LINK_LIBRARIES O2::Framework O2::ML O2::GPUCommon ONNXRuntime::ONNXRuntime ${CMAKE_DL_LIBS}
26+
COMPONENT_NAME AnalysisTutorial)

‎Tutorials/ML/README-benchmark.md‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
# ONNX inference benchmark tutorial
2+
3+
`benchmarkOnnxModel.cxx` registers `o2-analysis-benchmark-onnx-model`.
4+
It runs once with deterministic synthetic data, writes one CSV row, and exits.
5+
It accepts one dense floating-point input of shape `[batch, features]` and one
6+
or more outputs. Static dimensions must match the options; symbolic dimensions
7+
are filled from the options. Output types are left to ONNX Runtime.
8+
9+
## Build and AMD prerequisites
10+
11+
Build O2Physics normally after loading its O2 environment. This target links
12+
`O2::ML`, `O2::GPUCommon`, `ONNXRuntime::ONNXRuntime`, and the platform dynamic
13+
loader. No HIP headers are required for building the task. The O2 installation
14+
must contain `ML/3rdparty/GPUORTFloat16.h` and its GPUCommon headers.
15+
16+
For GPU runs, ONNX Runtime must contain **MIGraphXExecutionProvider**, and its
17+
MIGraphX/ROCm shared libraries, including `libamdhip64.so`, must be discoverable
18+
at runtime. The task checks provider availability and disables CPU fallback:
19+
unsupported graphs fail rather than producing a misleading GPU benchmark.
20+
No changes to O2's OrtInterface are needed for this task; the current checked-out
21+
OrtInterface does not attach an AMD execution provider, so this example uses
22+
ORT directly.
23+
24+
Check the actual hydra driver, ROCm, MIGraphX and ORT combination first. MI100
25+
(gfx908) and MI50 (gfx906) do not have identical software support. MI50 is absent
26+
from current supported ROCm hardware lists, so it needs a compatible older
27+
stack or a separately validated build. The old ROCm execution provider was
28+
removed from ONNX Runtime starting with 1.23; this task uses MIGraphX.
29+
30+
References:
31+
- https://onnxruntime.ai/docs/execution-providers/MIGraphX-ExecutionProvider.html
32+
- https://onnxruntime.ai/docs/execution-providers/ROCm-ExecutionProvider.html
33+
- https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html
34+
35+
## Run the three comparisons
36+
37+
Use equivalent FP32 and FP16 exports of the same network. Replace the example
38+
feature count with the actual model input width. Use separate output files;
39+
each invocation overwrites its output CSV.
40+
41+
```bash
42+
o2-analysis-benchmark-onnx-model -b --model-path model-fp32.onnx \
43+
--provider cpu --fp16 false --batch-size 1024 --features 5 \
44+
--cpu-threads 1 --warmup 20 --iterations 200 --output-csv cpu-fp32.csv
45+
46+
o2-analysis-benchmark-onnx-model -b --model-path model-fp32.onnx \
47+
--provider migraphx --gpu-id 0 --fp16 false --batch-size 1024 --features 5 \
48+
--warmup 20 --iterations 200 --output-csv gpu-fp32.csv
49+
50+
o2-analysis-benchmark-onnx-model -b --model-path model-fp16.onnx \
51+
--provider migraphx --gpu-id 0 --fp16 true --batch-size 1024 --features 5 \
52+
--warmup 20 --iterations 200 --output-csv gpu-fp16.csv
53+
```
54+
55+
`gpu-id` is the HIP logical ID after any `HIP_VISIBLE_DEVICES` or
56+
`ROCR_VISIBLE_DEVICES` masking. Run one benchmark process at a time on an idle
57+
GPU. Repeat with the same batch sizes and CPU thread count, and compare several
58+
runs. Small batches can be dominated by host overhead and transfers.
59+
60+
`--fp16 true` selects O2's `o2::OrtDataType::Float16_t` input storage and checks
61+
that the model input is FLOAT16. It does not convert model weights or prove that
62+
every internal operation is FP16. Export/inspect the model accordingly. FP16
63+
models retaining FP32 input interfaces need an adapted input path; this small
64+
tutorial intentionally rejects them. CPU runs require FP32 input.
65+
66+
## What the CSV measures
67+
68+
- `samples_per_second = batch_size * iterations / seconds`.
69+
- Timing covers synchronous `Session::Run`, input/output transfers, and output
70+
allocation/destruction. It excludes session creation, MIGraphX compilation,
71+
warmup, input generation, FP16 conversion, telemetry, and CSV I/O. This is
72+
host-to-host inference throughput, not GPU kernel-only throughput.
73+
- GPU memory fields are **device-wide used bytes**, sampled through
74+
`hipMemGetInfo` before session creation, after warmup, and after timed runs.
75+
They include other processes and runtime allocations; they are neither
76+
model-only memory nor a transient peak. The baseline follows HIP context
77+
initialization. CPU runs leave these fields empty. GPU telemetry failures
78+
fail the run rather than reporting zero usage.
79+
- Precision labels describe the requested model/input variant, not measured
80+
arithmetic precision. Strict CPU fallback rejection can reject graphs that
81+
would otherwise run with some CPU operators.
82+
83+
This benchmark does not check prediction accuracy. Compare real model outputs
84+
between the exports separately before interpreting performance gains.
85+
86+
## Adding the AO2D input later
87+
88+
Planned input on hydra:
89+
90+
```text
91+
/lustre/alice/users/marin/R3A/TPCTrees/TPCTrees_LHC24as_250224/AO2D_merge_LHC24as_updatedBB_250224.root
92+
```
93+
94+
The file has not been inspected here. Do not pass `--aod-file` to this standalone
95+
synthetic task yet. First establish the AO2D tables, feature columns and order,
96+
training normalization, selections, and missing-value policy. Then add an
97+
analysis `process(...)` input with the appropriate table definitions, build
98+
contiguous batches from its rows, and reuse the inference path. Keep session
99+
creation in `init`, aggregate timing/sample counts across batches, and write
100+
results at end of stream. Handle the final partial batch explicitly (dynamic
101+
batch, or pad and count only valid samples). Keep data reading/preprocessing
102+
outside inference timing or report a separate full-pipeline metric.

‎Tutorials/ML/benchmarkOnnxModel.cxx‎

Lines changed: 235 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,235 @@
1+
// Copyright 2019-2020 CERN and copyright holders of ALICE O2.
2+
// See https://alice-o2.web.cern.ch/copyright for details of the copyright holders.
3+
// All rights not expressly granted are reserved.
4+
//
5+
// This software is distributed under the terms of the GNU General Public
6+
// License v3 (GPL Version 3), copied verbatim in the file "COPYING".
7+
//
8+
// In applying this license CERN does not waive the privileges and immunities
9+
// granted to it by virtue of its status as an Intergovernmental Organization
10+
// or submit itself to any jurisdiction.
11+
12+
/// Benchmark one dense ONNX input [batch, features] using CPU or AMD MIGraphX.
13+
#include <Framework/AnalysisTask.h>
14+
#include <Framework/Configurable.h>
15+
#include <Framework/ControlService.h>
16+
#include <Framework/ProcessingContext.h>
17+
#include <Framework/runDataProcessing.h>
18+
19+
#include "ML/3rdparty/GPUORTFloat16.h"
20+
#include <onnxruntime_cxx_api.h>
21+
22+
#include <algorithm>
23+
#include <chrono>
24+
#include <cstdint>
25+
#include <fstream>
26+
#include <iomanip>
27+
#include <limits>
28+
#include <memory>
29+
#include <stdexcept>
30+
#include <string>
31+
#include <vector>
32+
33+
#include <dlfcn.h>
34+
35+
namespace
36+
{
37+
// Optional HIP telemetry without introducing a ROCm build dependency for CPU users.
38+
// Memory is device-wide, not process-local; samples are not a transient peak.
39+
class HipMemory
40+
{
41+
public:
42+
explicit HipMemory(int device)
43+
{
44+
library = dlopen("libamdhip64.so", RTLD_LAZY | RTLD_LOCAL);
45+
if (!library) {
46+
throw std::runtime_error("GPU telemetry requires libamdhip64.so on LD_LIBRARY_PATH");
47+
}
48+
auto setDevice = reinterpret_cast<int (*)(int)>(dlsym(library, "hipSetDevice"));
49+
getMemory = reinterpret_cast<int (*)(size_t*, size_t*)>(dlsym(library, "hipMemGetInfo"));
50+
if (!setDevice || !getMemory || setDevice(device) != 0) {
51+
dlclose(library);
52+
library = nullptr;
53+
throw std::runtime_error("Cannot select HIP device or resolve HIP memory API");
54+
}
55+
}
56+
~HipMemory()
57+
{
58+
if (library) {
59+
dlclose(library);
60+
}
61+
}
62+
HipMemory(HipMemory const&) = delete;
63+
HipMemory& operator=(HipMemory const&) = delete;
64+
uint64_t used() const
65+
{
66+
size_t free = 0, total = 0;
67+
if (getMemory(&free, &total) != 0 || free > total) {
68+
throw std::runtime_error("hipMemGetInfo failed");
69+
}
70+
return total - free;
71+
}
72+
73+
private:
74+
void* library = nullptr;
75+
int (*getMemory)(size_t*, size_t*) = nullptr;
76+
};
77+
78+
std::string csvQuote(std::string const& value)
79+
{
80+
std::string result = "\"";
81+
for (char c : value) {
82+
result += c;
83+
if (c == '"') {
84+
result += '"';
85+
}
86+
}
87+
return result + '"';
88+
}
89+
} // namespace
90+
91+
using namespace o2::framework;
92+
93+
struct BenchmarkOnnxModel {
94+
Configurable<std::string> modelPath{"model-path", "model.onnx", "Local ONNX model"};
95+
Configurable<std::string> provider{"provider", "cpu", "cpu or migraphx"};
96+
Configurable<int> gpuId{"gpu-id", 0, "HIP logical device ID (after visibility masking)"};
97+
Configurable<bool> fp16{"fp16", false, "Require FP16 input; does not convert model weights"};
98+
Configurable<int> batchSize{"batch-size", 1024, "Samples per inference"};
99+
Configurable<int> features{"features", 5, "Features per sample, in model order"};
100+
Configurable<int> warmup{"warmup", 10, "Untimed warm-up calls"};
101+
Configurable<int> iterations{"iterations", 100, "Timed inference calls"};
102+
Configurable<int> cpuThreads{"cpu-threads", 1, "ORT intra-op CPU threads"};
103+
Configurable<std::string> outputCsv{"output-csv", "onnx-benchmark.csv", "CSV output (overwritten)"};
104+
105+
void run(ProcessingContext& pc)
106+
{
107+
benchmark();
108+
pc.services().get<ControlService>().endOfStream();
109+
pc.services().get<ControlService>().readyToQuit(QuitRequest::Me);
110+
}
111+
112+
void benchmark()
113+
{
114+
if (batchSize.value <= 0 || features.value <= 0 || iterations.value <= 0 || warmup.value < 1 || cpuThreads.value <= 0 || gpuId.value < 0) {
115+
throw std::runtime_error("Require positive batch, features, iterations, warmup, threads and nonnegative GPU ID");
116+
}
117+
bool gpu = provider.value == "migraphx";
118+
if (!gpu && provider.value != "cpu") {
119+
throw std::runtime_error("provider must be cpu or migraphx");
120+
}
121+
if (!gpu && fp16.value) {
122+
throw std::runtime_error("Use FP32 for the CPU baseline");
123+
}
124+
std::unique_ptr<HipMemory> memory;
125+
uint64_t before = 0, afterWarmup = 0, afterRun = 0;
126+
Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "tutorial-benchmark");
127+
Ort::SessionOptions options;
128+
options.SetIntraOpNumThreads(cpuThreads.value);
129+
options.SetInterOpNumThreads(1);
130+
options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL);
131+
if (gpu) {
132+
auto providers = Ort::GetAvailableProviders();
133+
if (std::find(providers.begin(), providers.end(), "MIGraphXExecutionProvider") == providers.end()) {
134+
throw std::runtime_error("This ONNX Runtime has no MIGraphXExecutionProvider");
135+
}
136+
memory = std::make_unique<HipMemory>(gpuId.value);
137+
before = memory->used();
138+
OrtMIGraphXProviderOptions amd{};
139+
amd.device_id = gpuId.value;
140+
amd.migraphx_mem_limit = std::numeric_limits<size_t>::max();
141+
// Keep the supplied model precision; do not automatically quantize FP32.
142+
amd.migraphx_fp16_enable = 0;
143+
options.AppendExecutionProvider_MIGraphX(amd);
144+
options.AddConfigEntry("session.disable_cpu_ep_fallback", "1");
145+
}
146+
Ort::Session session(env, modelPath.value.c_str(), options);
147+
if (session.GetInputCount() != 1 || session.GetOutputCount() < 1) {
148+
throw std::runtime_error("Tutorial expects exactly one dense input and at least one output");
149+
}
150+
auto inputInfo = session.GetInputTypeInfo(0);
151+
auto tensorInfo = inputInfo.GetTensorTypeAndShapeInfo();
152+
auto shape = tensorInfo.GetShape();
153+
auto type = fp16.value ? ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT16 : ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT;
154+
if (tensorInfo.GetElementType() != type || shape.size() != 2 ||
155+
(shape[0] >= 0 && shape[0] != batchSize.value) || (shape[1] >= 0 && shape[1] != features.value)) {
156+
throw std::runtime_error("Input must match fp16 flag and [batch-size, features]; only dynamic dimensions may be overridden");
157+
}
158+
shape = {batchSize.value, features.value};
159+
const size_t count = static_cast<size_t>(batchSize.value) * features.value;
160+
if (count > std::numeric_limits<size_t>::max() / sizeof(float)) {
161+
throw std::runtime_error("Input size overflow");
162+
}
163+
// Replace this feature filling with AO2D extraction and training preprocessing later.
164+
std::vector<float> floats(fp16.value ? 0 : count);
165+
std::vector<o2::OrtDataType::Float16_t> halves(fp16.value ? count : 0);
166+
static_assert(sizeof(o2::OrtDataType::Float16_t) == 2);
167+
for (size_t i = 0; i < count; ++i) {
168+
float value = static_cast<float>(i % 101) / 100.f - 0.5f;
169+
if (fp16.value) {
170+
halves[i] = o2::OrtDataType::Float16_t(value);
171+
} else {
172+
floats[i] = value;
173+
}
174+
}
175+
auto host = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
176+
void* data = fp16.value ? static_cast<void*>(halves.data()) : static_cast<void*>(floats.data());
177+
size_t bytes = count * (fp16.value ? sizeof(halves[0]) : sizeof(floats[0]));
178+
auto input = Ort::Value::CreateTensor(host, data, bytes, shape.data(), shape.size(), type);
179+
Ort::AllocatorWithDefaultOptions allocator;
180+
auto inputName = session.GetInputNameAllocated(0, allocator);
181+
const char* inputNames[] = {inputName.get()};
182+
std::vector<std::string> names;
183+
for (size_t i = 0; i < session.GetOutputCount(); ++i) {
184+
names.emplace_back(session.GetOutputNameAllocated(i, allocator).get());
185+
}
186+
std::vector<const char*> outputNames;
187+
for (auto const& name : names) {
188+
outputNames.push_back(name.c_str());
189+
}
190+
Ort::RunOptions runOptions;
191+
auto infer = [&]() {
192+
// Default synchronous Run with CPU input/output includes host/device transfers
193+
// and output allocation/destruction. No asynchronous Run option is enabled.
194+
auto outputs = session.Run(runOptions, inputNames, &input, 1, outputNames.data(), outputNames.size());
195+
};
196+
for (int i = 0; i < warmup.value; ++i) {
197+
infer();
198+
}
199+
if (memory) {
200+
afterWarmup = memory->used();
201+
}
202+
auto start = std::chrono::steady_clock::now();
203+
for (int i = 0; i < iterations.value; ++i) {
204+
infer();
205+
}
206+
double seconds = std::chrono::duration<double>(std::chrono::steady_clock::now() - start).count();
207+
if (memory) {
208+
afterRun = memory->used();
209+
}
210+
std::ofstream csv(outputCsv.value, std::ios::trunc);
211+
if (!csv) {
212+
throw std::runtime_error("Cannot open output-csv");
213+
}
214+
csv << "model,ort_version,provider,precision,gpu_id,batch_size,features,warmup,iterations,cpu_threads,seconds,samples_per_second,gpu_used_before_bytes,gpu_used_after_warmup_bytes,gpu_used_after_run_bytes\n";
215+
csv << csvQuote(modelPath.value) << ',' << csvQuote(Ort::GetVersionString()) << ',' << provider.value << ','
216+
<< (fp16.value ? "fp16" : "fp32") << ',' << (gpu ? gpuId.value : -1) << ','
217+
<< batchSize.value << ',' << features.value << ',' << warmup.value << ',' << iterations.value << ',' << cpuThreads.value << ','
218+
<< std::setprecision(12) << seconds << ',' << (static_cast<double>(batchSize.value) * iterations.value / seconds) << ',';
219+
if (memory) {
220+
csv << before << ',' << afterWarmup << ',' << afterRun;
221+
} else {
222+
csv << ",,";
223+
}
224+
csv << '\n';
225+
csv.close();
226+
if (!csv) {
227+
throw std::runtime_error("Failed writing output-csv");
228+
}
229+
}
230+
};
231+
232+
WorkflowSpec defineDataProcessing(ConfigContext const& cfgc)
233+
{
234+
return WorkflowSpec{adaptAnalysisTask<BenchmarkOnnxModel>(cfgc)};
235+
}

0 commit comments

Comments
 (0)