onnx_diagnostic.helpers.rt_helper¶

onnx_diagnostic.helpers.rt_helper.make_feeds(proto: ModelProto | List[str], inputs: Any, use_numpy: bool = False, copy: bool = False, check_flatten: bool = True, is_modelbuilder: bool = False) → Dict[str, ndarray | Tensor][source][source]¶

Serializes the inputs to produce feeds expected by onnxruntime.InferenceSession.

Parameters:

proto – onnx model or list of names
inputs – any kind of inputs
use_numpy – if True, converts torch tensors into numpy arrays
copy – a copy is made, this should be the case if the inputs is ingested by OrtValue
check_flatten – if True, checks the torch.utils._pytree.tree_flatten returns the same number of outputs
is_modelbuilder – if True, the exporter is ModelBuilder, and we need to reorder the past_key_values inputs to match the expected order, and get rid of position_ids.

Returns:

feeds dictionary

onnx_diagnostic.helpers.rt_helper.reorder_modelbuilder_cache_to_torch(past_kv: List[Any]) → List[Any][source][source]¶

Reorders the past_kvs for ModelBuilder to match the expected order by PyTorch exported models.

Note

This function can take either the names or the actual tensors as long as they are in a list.

Conceptually,

From:

[past_key_values.0.key, past_key_values.0.value,
past_key_values.1.key, past_key_values.1.value, ...]

To:

[past_key_values.0.key, past_key_values.1.key,
..., past_key_values.0.value, past_key_values.1.value, ...]