Intermediate results with (ONNX) ReferenceEvaluator

Let’s assume onnxruntime crashes without telling why or where. The first thing is do is to locate where. For that, we run a python runtime which is going to run until it fails.

A failing model

The issue here is a an operator Cast trying to convert a result into a non-existing type.

import numpy as np
import onnx
import onnx.helper as oh
import onnxruntime
from onnx_diagnostic import doc
from onnx_diagnostic.helpers.onnx_helper import from_array_extended
from onnx_diagnostic.reference import ExtendedReferenceEvaluator

TFLOAT = onnx.TensorProto.FLOAT

model = oh.make_model(
    oh.make_graph(
        [
            oh.make_node("Mul", ["X", "Y"], ["xy"], name="n0"),
            oh.make_node("Sigmoid", ["xy"], ["sy"], name="n1"),
            oh.make_node("Add", ["sy", "one"], ["C"], name="n2"),
            oh.make_node("Cast", ["C"], ["X999"], to=999, name="failing"),
            oh.make_node("CastLike", ["X999", "Y"], ["Z"], name="n4"),
        ],
        "-nd-",
        [
            oh.make_tensor_value_info("X", TFLOAT, ["a", "b", "c"]),
            oh.make_tensor_value_info("Y", TFLOAT, ["a", "b", "c"]),
        ],
        [oh.make_tensor_value_info("Z", TFLOAT, ["a", "b", "c"])],
        [from_array_extended(np.array([1], dtype=np.float32), name="one")],
    ),
    opset_imports=[oh.make_opsetid("", 18)],
    ir_version=9,
)

We check it is failing.

try:
    onnxruntime.InferenceSession(model.SerializeToString(), providers=["CPUExecutionProvider"])
except onnxruntime.capi.onnxruntime_pybind11_state.Fail as e:
    print(e)

[ONNXRuntimeError] : 1 : FAIL : Node (failing) Op (Cast) [TypeInferenceError] Attribute to does not specify a valid type in .

ExtendedReferenceEvaluator

This class extends onnx.reference.ReferenceEvaluator with operators outside the standard but defined by onnxruntime. verbose=10 tells the class to print as much as possible, verbose=0 prints nothing. Intermediate values for more or less verbosity.

ref = ExtendedReferenceEvaluator(model, verbose=10)
feeds = dict(
    X=np.random.rand(3, 4).astype(np.float32), Y=np.random.rand(3, 4).astype(np.float32)
)
try:
    ref.run(None, feeds)
except Exception as e:
    print("ERROR", type(e), e)

 +C one: float32:(1,):[1.0]
 +I X: float32:(3, 4):0.17795032262802124,0.7188646197319031,0.5659847855567932,0.6799622774124146,0.7473636865615845...
 +I Y: float32:(3, 4):0.6002911925315857,0.3609388470649719,0.3137604892253876,0.6788439154624939,0.25307998061180115...
Mul(X, Y) -> xy
 + xy: float32:(3, 4):0.10682201385498047,0.25946617126464844,0.17758366465568542,0.4615882635116577,0.18914279341697693...
Sigmoid(xy) -> sy
 + sy: float32:(3, 4):0.5266801118850708,0.5645050406455994,0.5442796349525452,0.6133908629417419,0.5471452474594116...
Add(sy, one) -> C
 + C: float32:(3, 4):1.5266801118850708,1.5645051002502441,1.5442795753479004,1.6133909225463867,1.5471452474594116...
Cast(C) -> X999
ERROR <class 'KeyError'> 999

We can see it run until it reaches Cast and stops. The error message is not always obvious to interpret. It gets improved every time from time to time. This runtime is useful when it fails for a numerical reason. It is possible to insert prints in the python code to print more information or debug if needed.

doc.plot_legend("Python Runtime\nfor ONNX", "ExtendedReferenceEvalutor", "lightgrey")

Related examples

Find where a model is failing by running submodels

Find where a model is failing by running submodels

Intermediate results with onnxruntime

Intermediate results with onnxruntime

Find and fix an export issue due to dynamic shapes

Find and fix an export issue due to dynamic shapes