Testing¶

experimental_experiment.ext_test_case¶

class experimental_experiment.ext_test_case.ExtTestCase(methodName='runTest')[source]¶

assertAlmostEqual(expected: ndarray, value: ndarray, atol: float = 0, rtol: float = 0)[source]¶

Fail if the two objects are unequal as determined by their difference rounded to the given number of decimal places (default 7) and comparing to zero, or by comparing that the difference between the two objects is more than the given delta.

Note that decimal places (from zero) are usually not the same as significant digits (measured from the most significant digit).

If the two objects compare equal then they will automatically compare almost equal.

capture(fct: Callable)[source]¶

Runs a function and capture standard output and error.

Parameters:: fct – function to run
Returns:: result of fct, output, error

dump_onnx(name: str, proto: Any, folder: str | None = None) → str[source]¶: Dumps an onnx file.

get_dump_file(name: str, folder: str | None = None) → str[source]¶: Returns a filename to dump a model.

classmethod tearDownClass()[source]¶: Hook method for deconstructing the class fixture after running all tests in the class.

tryCall(fct: Callable, msg: str | None = None, none_if: str | None = None) → Any | None[source]¶

Calls the function, catch any error.

Parameters:

fct – function to call
msg – error message to display if failing
none_if – returns None if this substring is found in the error message

Returns:

output of fct

experimental_experiment.ext_test_case.get_figure(ax)[source]¶: Returns the figure of a matplotlib figure.

experimental_experiment.ext_test_case.has_cuda() → bool[source]¶: Returns torch.cuda.is_available().

experimental_experiment.ext_test_case.hide_stdout(f: Callable | None = None) → Callable[source]¶

Catches warnings.

Parameters:: f – the function is called with the stdout as an argument

experimental_experiment.ext_test_case.ignore_warnings(warns: List[Warning]) → Callable[source]¶

Catches warnings.

Parameters:: warns – warnings to ignore

experimental_experiment.ext_test_case.is_azure() → bool[source]¶: Tells if the job is running on Azure DevOps.

experimental_experiment.ext_test_case.measure_time(stmt: str | Callable, context: Dict[str, Any] | None = None, repeat: int = 10, number: int = 50, warmup: int = 1, div_by_number: bool = True, max_time: float | None = None) → Dict[str, str | int | float][source]¶

Measures a statement and returns the results as a dictionary.

Parameters:

stmt – string or callable
context – variable to know in a dictionary
repeat – average over repeat experiment
number – number of executions in one row
warmup – number of iteration to do before starting the real measurement
div_by_number – divide by the number of executions
max_time – execute the statement until the total goes beyond this time (approximatively), repeat is ignored, div_by_number must be set to True

Returns:

dictionary

<<<

from pprint import pprint
from math import cos
from experimental_experiment.ext_test_case import measure_time

res = measure_time(lambda: cos(0.5))
pprint(res)

>>>

    {'average': 8.080001862253994e-08,
     'context_size': 64,
     'deviation': 2.4000231258147997e-09,
     'max_exec': 8.800008799880743e-08,
     'min_exec': 7.999988156370819e-08,
     'number': 50,
     'repeat': 10,
     'ttime': 8.080001862253994e-07,
     'warmup_time': 2.6999987312592566e-06}

See Timer.repeat for a better understanding of parameter repeat and number. The function returns a duration corresponding to number times the execution of the main statement.

experimental_experiment.ext_test_case.requires_onnx(version: str, msg: str = '') → Callable[source]¶: Skips a unit test if onnx is not recent enough.

experimental_experiment.ext_test_case.requires_onnxruntime(version: str, msg: str = '') → Callable[source]¶: Skips a unit test if onnxruntime is not recent enough.

experimental_experiment.ext_test_case.requires_onnxruntime_training(push_back_batch: bool = False, msg: str = '') → Callable[source]¶: Skips a unit test if onnxruntime is not onnxruntime_training.

experimental_experiment.ext_test_case.requires_onnxscript(version: str, msg: str = '') → Callable[source]¶: Skips a unit test if onnxscript is not recent enough.

experimental_experiment.ext_test_case.requires_torch(version: str, msg: str = '') → Callable[source]¶: Skips a unit test if pytorch is not recent enough.

experimental_experiment.ext_test_case.requires_transformers(version: str, msg: str = '') → Callable[source]¶: Skips a unit test if transformers is not recent enough.

experimental_experiment.ext_test_case.requires_zoo(msg: str = '') → Callable[source]¶: Skips a unit test if environment variable ZOO is not equal to 1.

experimental_experiment.ext_test_case.skipif_ci_apple(msg) → Callable[source]¶: Skips a unit test if it runs on azure pipeline on Windows.

experimental_experiment.ext_test_case.skipif_ci_windows(msg) → Callable[source]¶: Skips a unit test if it runs on azure pipeline on Windows.

experimental_experiment.ext_test_case.skipif_not_onnxrt(msg) → Callable[source]¶: Skips a unit test if it runs on azure pipeline on Windows.

experimental_experiment.ext_test_case.skipif_transformers(version_to_skip: str | Set[str], msg: str) → Callable[source]¶: Skips a unit test if transformers has a specific version.

experimental_experiment.ext_test_case.statistics_on_file(filename: str) → Dict[str, str | int | float][source]¶

Computes statistics on a file.

<<<

import pprint
from experimental_experiment.ext_test_case import statistics_on_file, __file__

pprint.pprint(statistics_on_file(__file__))

>>>

    {'chars': 14461, 'ext': '.py', 'lines': 522}

experimental_experiment.ext_test_case.statistics_on_folder(folder: str | List[str], pattern: str = '.*[.]((py|rst))$', aggregation: int = 0) → List[Dict[str, str | int | float]][source]¶

Computes statistics on files in a folder.

Parameters:

folder – folder or folders to investigate
pattern – file pattern
aggregation – show the first subfolders

Returns:

list of dictionaries

<<<

import os
import pprint
from experimental_experiment.ext_test_case import statistics_on_folder, __file__

pprint.pprint(statistics_on_folder(os.path.dirname(__file__)))

>>>

    [{'chars': 65, 'ext': '.py', 'lines': 3, 'name': '__main__.py'},
     {'chars': 3766, 'ext': '.py', 'lines': 167, 'name': 'memory_peak.py'},
     {'chars': 2809, 'ext': '.py', 'lines': 93, 'name': 'torch_test_helper.py'},
     {'chars': 66, 'ext': '.py', 'lines': 3, 'name': '_bench_test.py'},
     {'chars': 1807, 'ext': '.py', 'lines': 51, 'name': 'checks.py'},
     {'chars': 2722, 'ext': '.py', 'lines': 98, 'name': 'bench_run.py'},
     {'chars': 2611, 'ext': '.py', 'lines': 96, 'name': 'args.py'},
     {'chars': 14461, 'ext': '.py', 'lines': 522, 'name': 'ext_test_case.py'},
     {'chars': 3109,
      'ext': '.py',
      'lines': 126,
      'name': '_command_lines_parser.py'},
     {'chars': 2725, 'ext': '.py', 'lines': 102, 'name': 'onnx_tools.py'},
     {'chars': 72, 'ext': '.py', 'lines': 4, 'name': '__init__.py'},
     {'chars': 4454,
      'ext': '.py',
      'lines': 138,
      'name': 'convert/convert_helper.py'},
     {'chars': 851, 'ext': '.py', 'lines': 22, 'name': 'convert/ort_helper.py'},
     {'chars': 0, 'ext': '.py', 'lines': 0, 'name': 'convert/__init__.py'},
     {'chars': 4060,
      'ext': '.py',
      'lines': 129,
      'name': 'torch_dynamo/partition.py'},
     {'chars': 2338,
      'ext': '.py',
      'lines': 69,
      'name': 'torch_dynamo/_dynamo_exporter.py'},
     {'chars': 568,
      'ext': '.py',
      'lines': 19,
      'name': 'torch_dynamo/dynger_backend.py'},
     {'chars': 491,
      'ext': '.py',
      'lines': 16,
      'name': 'torch_dynamo/backend_helper.py'},
     {'chars': 17137,
      'ext': '.py',
      'lines': 574,
      'name': 'torch_dynamo/fast_backend.py'},
     {'chars': 8147,
      'ext': '.py',
      'lines': 278,
      'name': 'torch_dynamo/debug_backend.py'},
     {'chars': 5110,
      'ext': '.py',
      'lines': 173,
      'name': 'torch_dynamo/__init__.py'},
     {'chars': 178, 'ext': '.py', 'lines': 5, 'name': 'xbuilder/_helper.py'},
     {'chars': 4121,
      'ext': '.py',
      'lines': 173,
      'name': 'xbuilder/graph_builder_opset.py'},
     {'chars': 19381,
      'ext': '.py',
      'lines': 631,
      'name': 'xbuilder/shape_type_compute.py'},
     {'chars': 3871,
      'ext': '.py',
      'lines': 143,
      'name': 'xbuilder/type_inference.py'},
     {'chars': 5232,
      'ext': '.py',
      'lines': 178,
      'name': 'xbuilder/model_container.py'},
     {'chars': 3139,
      'ext': '.py',
      'lines': 93,
      'name': 'xbuilder/optimization_options.py'},
     {'chars': 4312,
      'ext': '.py',
      'lines': 207,
      'name': 'xbuilder/_onnx_helper.py'},
     {'chars': 2534, 'ext': '.py', 'lines': 85, 'name': 'xbuilder/shape_helper.py'},
     {'chars': 872,
      'ext': '.py',
      'lines': 30,
      'name': 'xbuilder/expression_dimension.py'},
     {'chars': 1136,
      'ext': '.py',
      'lines': 39,
      'name': 'xbuilder/_dtype_helper.py'},
     {'chars': 96778,
      'ext': '.py',
      'lines': 3344,
      'name': 'xbuilder/graph_builder.py'},
     {'chars': 86, 'ext': '.py', 'lines': 2, 'name': 'xbuilder/__init__.py'},
     {'chars': 13170,
      'ext': '.py',
      'lines': 449,
      'name': 'reference/ort_evaluator.py'},
     {'chars': 1111, 'ext': '.py', 'lines': 39, 'name': 'reference/__init__.py'},
     {'chars': 3872, 'ext': '.py', 'lines': 146, 'name': 'reference/evaluator.py'},
     {'chars': 1915,
      'ext': '.py',
      'lines': 73,
      'name': 'reference/ops/op_scatter_elements.py'},
     {'chars': 295,
      'ext': '.py',
      'lines': 10,
      'name': 'reference/ops/op_transpose_cast.py'},
     {'chars': 455,
      'ext': '.py',
      'lines': 17,
      'name': 'reference/ops/op_mul_sigmoid.py'},
     {'chars': 1091,
      'ext': '.py',
      'lines': 44,
      'name': 'reference/ops/op_add_add_mul_mul.py'},
     {'chars': 140,
      'ext': '.py',
      'lines': 7,
      'name': 'reference/ops/op_memcpy_host.py'},
     {'chars': 667,
      'ext': '.py',
      'lines': 16,
      'name': 'reference/ops/op_scatternd_of_shape.py'},
     {'chars': 458,
      'ext': '.py',
      'lines': 17,
      'name': 'reference/ops/op_quick_gelu.py'},
     {'chars': 147,
      'ext': '.py',
      'lines': 5,
      'name': 'reference/ops/op_negxplus1.py'},
     {'chars': 651,
      'ext': '.py',
      'lines': 31,
      'name': 'reference/ops/op_fused_matmul.py'},
     {'chars': 907,
      'ext': '.py',
      'lines': 48,
      'name': 'reference/ops/op_constant_of_shape.py'},
     {'chars': 990,
      'ext': '.py',
      'lines': 31,
      'name': 'reference/ops/op_cast_like.py'},
     {'chars': 415,
      'ext': '.py',
      'lines': 16,
      'name': 'reference/ops/op_rotary.py'},
     {'chars': 380,
      'ext': '.py',
      'lines': 14,
      'name': 'reference/ops/op_tri_matrix.py'},
     {'chars': 383,
      'ext': '.py',
      'lines': 11,
      'name': 'reference/ops/op_concat.py'},
     {'chars': 220,
      'ext': '.py',
      'lines': 6,
      'name': 'reference/ops/op_simplified_layer_normalization.py'},
     {'chars': 0, 'ext': '.py', 'lines': 0, 'name': 'reference/ops/__init__.py'},
     {'chars': 224,
      'ext': '.py',
      'lines': 10,
      'name': 'reference/ops/op_replace_zero.py'},
     {'chars': 531, 'ext': '.py', 'lines': 15, 'name': 'reference/ops/op_slice.py'},
     {'chars': 317,
      'ext': '.py',
      'lines': 9,
      'name': 'reference/ops/op_gather_grad.py'},
     {'chars': 776,
      'ext': '.py',
      'lines': 29,
      'name': 'torch_interpreter/_torch_helper.py'},
     {'chars': 3860,
      'ext': '.py',
      'lines': 178,
      'name': 'torch_interpreter/oxs_opset.py'},
     {'chars': 76679,
      'ext': '.py',
      'lines': 3165,
      'name': 'torch_interpreter/_aten_functions.py'},
     {'chars': 9631,
      'ext': '.py',
      'lines': 307,
      'name': 'torch_interpreter/onnx_export.py'},
     {'chars': 1629,
      'ext': '.py',
      'lines': 54,
      'name': 'torch_interpreter/aten_functions.py'},
     {'chars': 5704,
      'ext': '.py',
      'lines': 204,
      'name': 'torch_interpreter/oxs_dispatcher.py'},
     {'chars': 10175,
      'ext': '.py',
      'lines': 460,
      'name': 'torch_interpreter/_prims_functions.py'},
     {'chars': 6737,
      'ext': '.py',
      'lines': 244,
      'name': 'torch_interpreter/dispatcher.py'},
     {'chars': 528,
      'ext': '.py',
      'lines': 21,
      'name': 'torch_interpreter/aten_methods.py'},
     {'chars': 87,
      'ext': '.py',
      'lines': 2,
      'name': 'torch_interpreter/_exceptions.py'},
     {'chars': 27772,
      'ext': '.py',
      'lines': 926,
      'name': 'torch_interpreter/interpreter.py'},
     {'chars': 119,
      'ext': '.py',
      'lines': 3,
      'name': 'torch_interpreter/__init__.py'},
     {'chars': 8437,
      'ext': '.py',
      'lines': 335,
      'name': 'torch_interpreter/_aten_methods.py'},
     {'chars': 22073, 'ext': '.py', 'lines': 723, 'name': 'xoptim/patterns_api.py'},
     {'chars': 4164, 'ext': '.py', 'lines': 172, 'name': 'xoptim/order_optim.py'},
     {'chars': 26686,
      'ext': '.py',
      'lines': 948,
      'name': 'xoptim/graph_builder_optim.py'},
     {'chars': 2416, 'ext': '.py', 'lines': 80, 'name': 'xoptim/__init__.py'},
     {'chars': 8684,
      'ext': '.py',
      'lines': 294,
      'name': 'xoptim/patterns/onnx_cast.py'},
     {'chars': 2435,
      'ext': '.py',
      'lines': 73,
      'name': 'xoptim/patterns/onnx_reduce.py'},
     {'chars': 4101,
      'ext': '.py',
      'lines': 152,
      'name': 'xoptim/patterns/onnx_any.py'},
     {'chars': 15707,
      'ext': '.py',
      'lines': 497,
      'name': 'xoptim/patterns/onnx_rotary.py'},
     {'chars': 2571,
      'ext': '.py',
      'lines': 85,
      'name': 'xoptim/patterns/onnx_sub.py'},
     {'chars': 9515,
      'ext': '.py',
      'lines': 293,
      'name': 'xoptim/patterns/onnx_reshape.py'},
     {'chars': 9714,
      'ext': '.py',
      'lines': 334,
      'name': 'xoptim/patterns/onnx_mul.py'},
     {'chars': 1609,
      'ext': '.py',
      'lines': 47,
      'name': 'xoptim/patterns/onnx_equal.py'},
     {'chars': 1455,
      'ext': '.py',
      'lines': 41,
      'name': 'xoptim/patterns/onnx_unsqueeze.py'},
     {'chars': 17710,
      'ext': '.py',
      'lines': 577,
      'name': 'xoptim/patterns/onnx_matmul.py'},
     {'chars': 5328,
      'ext': '.py',
      'lines': 158,
      'name': 'xoptim/patterns/onnx_expand.py'},
     {'chars': 2309,
      'ext': '.py',
      'lines': 79,
      'name': 'xoptim/patterns/onnx_transpose.py'},
     {'chars': 3052,
      'ext': '.py',
      'lines': 96,
      'name': 'xoptim/patterns/__init__.py'},
     {'chars': 2875,
      'ext': '.py',
      'lines': 94,
      'name': 'xoptim/patterns/onnx_split.py'},
     {'chars': 1213,
      'ext': '.py',
      'lines': 44,
      'name': 'xoptim/patterns_investigation/element_wise.py'},
     {'chars': 459,
      'ext': '.py',
      'lines': 15,
      'name': 'xoptim/patterns_investigation/__init__.py'},
     {'chars': 3967,
      'ext': '.py',
      'lines': 120,
      'name': 'xoptim/patterns_exp/constant_of_shape_scatter_nd.py'},
     {'chars': 2543,
      'ext': '.py',
      'lines': 76,
      'name': 'xoptim/patterns_exp/unary_operators.py'},
     {'chars': 3151,
      'ext': '.py',
      'lines': 93,
      'name': 'xoptim/patterns_exp/constants.py'},
     {'chars': 1493,
      'ext': '.py',
      'lines': 47,
      'name': 'xoptim/patterns_exp/where_replace.py'},
     {'chars': 2320,
      'ext': '.py',
      'lines': 76,
      'name': 'xoptim/patterns_exp/simple_rotary.py'},
     {'chars': 14112,
      'ext': '.py',
      'lines': 477,
      'name': 'xoptim/patterns_exp/binary_operators.py'},
     {'chars': 1564,
      'ext': '.py',
      'lines': 48,
      'name': 'xoptim/patterns_exp/__init__.py'},
     {'chars': 1282,
      'ext': '.py',
      'lines': 42,
      'name': 'xoptim/patterns_fix/add_reduction_scatter_nd.py'},
     {'chars': 429,
      'ext': '.py',
      'lines': 14,
      'name': 'xoptim/patterns_fix/__init__.py'},
     {'chars': 1929,
      'ext': '.py',
      'lines': 57,
      'name': 'xoptim/patterns_ort/activation_grad.py'},
     {'chars': 8846,
      'ext': '.py',
      'lines': 326,
      'name': 'xoptim/patterns_ort/fused_matmul.py'},
     {'chars': 4071,
      'ext': '.py',
      'lines': 116,
      'name': 'xoptim/patterns_ort/simplified_layer_normalization.py'},
     {'chars': 1455,
      'ext': '.py',
      'lines': 46,
      'name': 'xoptim/patterns_ort/gather_grad.py'},
     {'chars': 912,
      'ext': '.py',
      'lines': 27,
      'name': 'xoptim/patterns_ort/__init__.py'},
     {'chars': 1889, 'ext': '.py', 'lines': 56, 'name': 'plotting/memory.py'},
     {'chars': 6285, 'ext': '.py', 'lines': 377, 'name': 'plotting/data.py'},
     {'chars': 0, 'ext': '.py', 'lines': 0, 'name': 'plotting/__init__.py'},
     {'chars': 4281,
      'ext': '.py',
      'lines': 159,
      'name': 'torch_bench/_dort_cmd_common_models.py'},
     {'chars': 4131,
      'ext': '.py',
      'lines': 120,
      'name': 'torch_bench/export_model.py'},
     {'chars': 1149,
      'ext': '.py',
      'lines': 37,
      'name': 'torch_bench/check_model.py'},
     {'chars': 15115,
      'ext': '.py',
      'lines': 485,
      'name': 'torch_bench/_dort_cmd_common.py'},
     {'chars': 5757,
      'ext': '.py',
      'lines': 193,
      'name': 'torch_bench/dort_bench_profile.py'},
     {'chars': 8514,
      'ext': '.py',
      'lines': 252,
      'name': 'torch_bench/dort_bench.py'},
     {'chars': 5678,
      'ext': '.py',
      'lines': 164,
      'name': 'torch_bench/dort_profile.py'},
     {'chars': 0, 'ext': '.py', 'lines': 0, 'name': 'torch_bench/__init__.py'},
     {'chars': 17624,
      'ext': '.py',
      'lines': 498,
      'name': 'gradient/loss_helper.py'},
     {'chars': 11622,
      'ext': '.py',
      'lines': 383,
      'name': 'gradient/grad_helper.py'},
     {'chars': 0, 'ext': '.py', 'lines': 0, 'name': 'gradient/__init__.py'},
     {'chars': 89, 'ext': '.py', 'lines': 2, 'name': 'gradient/ops/__init__.py'},
     {'chars': 671,
      'ext': '.py',
      'lines': 42,
      'name': 'gradient/ops/op_broadcast_gradient_args.py'},
     {'chars': 5779,
      'ext': '.py',
      'lines': 202,
      'name': 'torch_models/dump_helper.py'},
     {'chars': 3382,
      'ext': '.py',
      'lines': 118,
      'name': 'torch_models/mistral_helper.py'},
     {'chars': 5687,
      'ext': '.py',
      'lines': 158,
      'name': 'torch_models/llama_helper.py'},
     {'chars': 3770,
      'ext': '.py',
      'lines': 131,
      'name': 'torch_models/phi_helper.py'},
     {'chars': 0, 'ext': '.py', 'lines': 0, 'name': 'torch_models/__init__.py'},
     {'chars': 7386,
      'ext': '.py',
      'lines': 218,
      'name': 'torch_models/training_helper.py'}]

experimental_experiment.ext_test_case.unit_test_going()[source]¶: Enables a flag telling the script is running while testing it. Avois unit tests to be very long.

experimental_experiment.ext_test_case.with_path_append(path_to_add: str | List[str]) → Callable[source]¶: Adds a path to sys.path to check.

experimental_experiment.torch_test_helper¶

experimental_experiment.torch_test_helper.check_model_ort(onx: ModelProto, providers: str | List[str] | None = None) → onnxruntime.InferenceSession[source]¶

Loads a model with onnxruntime.

Parameters:

onx – ModelProto
providers – list of providers, None fur CPU, cpu for CPU, cuda for CUDA

Returns:

InferenceSession

experimental_experiment.torch_test_helper.export_to_onnx(model: Any, *args: List[Any], verbose: int = 0, return_builder: bool = False, torch_script: bool = True, target_opset: int = 18, prefix: str | None = None, rename_inputs: bool = False, optimize: str | bool = True, folder: str | None = 'dump_test') → Dict[str, str | ModelProto | GraphBuilder][source]¶

Exports a model to ONNX.

Parameters:

model – model to export
args – arguments
verbose – verbosity
return_builder – returns the builder
torch_script – export with torch.script as well
target_opset – opset to export into
prefix – prefix to choose to export into
rename_inputs – rename the inputs into input_{i}
optimize – enable, disable optimizations of pattern to test
folder – where to dump the model, creates it if it does not exist

Returns:

dictionary with ModelProto, builder, filenames