onnx_diagnostic.ext_test_case

The module contains the main class ExtTestCase which adds specific functionalities to this project.

class onnx_diagnostic.ext_test_case.ExtTestCase(methodName='runTest')

Inherits from unittest.TestCase and adds specific comprison functions and other helper.

assertAlmostEqual(expected: ndarray, value: ndarray, atol: float = 0, rtol: float = 0)

In the name

assertEmpty(value: Any)

In the name

assertEndsWith(suffix: str, full: str)

In the name

assertEqual(expected: Any, value: Any, msg: str = '')

Overwrites the error message to get a more explicit message about what is what.

assertEqualArray(expected: Any, value: Any, atol: float = 0, rtol: float = 0, msg: str | None = None)

In the name

assertEqualArrays(expected: Sequence[ndarray], value: Sequence[ndarray], atol: float = 0, rtol: float = 0, msg: str | None = None)

In the name

assertEqualDataFrame(d1, d2, **kwargs)

Checks that two dataframes are equal. Calls pandas.testing.assert_frame_equal().

assertExists(name)

Checks the existing of a file.

assertGreaterOrEqual(a, b, msg=None)

In the name

assertIn(tofind: str, text: str, msg: str = '')

Just like self.assertTrue(a in b), but with a nicer default message.

assertNotEmpty(value: Any)

In the name

assertRaise(fct: Callable, exc_type: type[Exception], msg: str | None = None)

In the name

assertSetContained(set1, set2)

Checks that set1 is contained in set2.

assertStartsWith(prefix: str, full: str)

In the name

assert_onnx_disc(test_name: str, proto: onnx.ModelProto, model: torch.nn.Module, inputs: Tuple[Any] | Dict[str, Any], verbose: int = 0, atol: float = 1e-05, rtol: float = 0.001, copy_inputs: bool = True, expected: Any | None = None, use_ort: bool = False, **kwargs)

Checks for discrepancies. Runs the onnx models, computes expected outputs, in that order. The inputs may be modified by this functions if the torch model modifies them inplace.

Parameters:

• test_name – test name, dumps the model if not empty

• proto – onnx model

• model – torch model

• inputs – inputs

• verbose – verbosity

• atol – absolute tolerance

• rtol – relative tolerance

• expected – expected values

• copy_inputs – to copy the inputs

• use_ort – use onnxruntime.InferenceSession

• kwargs – arguments sent to onnx_diagnostic.helpers.ort_session.InferenceSessionForTorch

capture(fct: Callable)

Runs a function and capture standard output and error.

Parameters:

fct – function to run

Returns:

result of fct, output, error

dump_onnx(name: str, proto: Any, folder: str | None = None) → str

Dumps an onnx file.

get_dump_file(name: str, folder: str | None = None) → str

Returns a filename to dump a model.

get_dump_folder(folder: str) → str

Returns a folder.

print_model(model: ModelProto)

Prints a ModelProto

print_onnx(model: ModelProto)

Prints a ModelProto

classmethod setUpClass()

Hook method for setting up class fixture before running tests in the class.

subloop(*args, verbose: int = 0)

Loops over elements and calls unittests.TestCase.subTest().

classmethod tearDownClass()

Hook method for deconstructing the class fixture after running all tests in the class.

classmethod todo(f: Callable, msg: str)

Adds a todo printed when all test are run.

tryCall(fct: Callable, msg: str | None = None, none_if: str | None = None) → Any | None

Calls the function, catch any error.

Parameters:

• fct – function to call

• msg – error message to display if failing

• none_if – returns None if this substring is found in the error message

Returns:

output of fct

property verbose

Returns the the value of environment variable VERBOSE.

onnx_diagnostic.ext_test_case.get_figure(ax)

Returns the figure of a matplotlib figure.

onnx_diagnostic.ext_test_case.has_cuda() → bool

Returns torch.cuda.device_count() > 0.

onnx_diagnostic.ext_test_case.has_onnxruntime_training(push_back_batch: bool = False)

Tells if onnxruntime_training is installed.

onnx_diagnostic.ext_test_case.has_onnxscript(version: str, msg: str = '') → Callable

Skips a unit test if onnxscript is not recent enough.

onnx_diagnostic.ext_test_case.has_torch(version: str) → bool

Returns True if torch transformers is higher.

onnx_diagnostic.ext_test_case.has_transformers(version: str) → bool

Returns True if transformers version is higher.

onnx_diagnostic.ext_test_case.hide_stdout(f: Callable | None = None) → Callable

Catches warnings, hides standard output. The function may be disabled by setting UNHIDE=1 before running the unit test.

Parameters:

f – the function is called with the stdout as an argument

onnx_diagnostic.ext_test_case.ignore_errors(errors: Exception | Tuple[Exception]) → Callable

Catches exception, skip the test if the error is expected sometimes.

Parameters:

errors – errors to ignore

onnx_diagnostic.ext_test_case.ignore_warnings(warns: List[Warning]) → Callable

Catches warnings.

Parameters:

warns – warnings to ignore

onnx_diagnostic.ext_test_case.is_azure() → bool

Tells if the job is running on Azure DevOps.

onnx_diagnostic.ext_test_case.long_test(msg: str = '') → Callable

Skips a unit test if it runs on azure pipeline on Windows.

onnx_diagnostic.ext_test_case.measure_time(stmt: str | Callable, context: Dict[str, Any] | None = None, repeat: int = 10, number: int = 50, warmup: int = 1, div_by_number: bool = True, max_time: float | None = None) → Dict[str, str | int | float]

Measures a statement and returns the results as a dictionary.

Parameters:

• stmt – string or callable

• context – variable to know in a dictionary

• repeat – average over repeat experiment

• number – number of executions in one row

• warmup – number of iteration to do before starting the real measurement

• div_by_number – divide by the number of executions

• max_time – execute the statement until the total goes beyond this time (approximately), repeat is ignored, div_by_number must be set to True

Returns:

dictionary

<<<

from pprint import pprint
from math import cos
from onnx_diagnostic.ext_test_case import measure_time

res = measure_time(lambda: cos(0.5))
pprint(res)

>>>

    {'average': np.float64(2.208859950769693e-07),
     'context_size': 64,
     'deviation': np.float64(3.8651206822675487e-07),
     'max_exec': np.float64(1.3799799489788711e-06),
     'min_exec': np.float64(8.397997589781881e-08),
     'number': 50,
     'repeat': 10,
     'ttime': np.float64(2.208859950769693e-06),
     'warmup_time': 3.54530056938529e-05}

See Timer.repeat for a better understanding of parameter repeat and number. The function returns a duration corresponding to number times the execution of the main statement.

onnx_diagnostic.ext_test_case.never_test(msg: str = '') → Callable

Skips a unit test.

onnx_diagnostic.ext_test_case.requires_cuda(msg: str = '', version: str = '', memory: int = 0)

Skips a test if cuda is not available.

Parameters:

• msg – to overwrite the message

• version – minimum version

• memory – minimum number of Gb to run the test

onnx_diagnostic.ext_test_case.requires_diffusers(version: str, msg: str = '', or_older_than: str | None = None) → Callable

Skips a unit test if transformers is not recent enough.

onnx_diagnostic.ext_test_case.requires_experimental(version: str = '0.0.0', msg: str = '') → Callable

Skips a unit test if experimental-experiment is not recent enough.

onnx_diagnostic.ext_test_case.requires_numpy(version: str, msg: str = '') → Callable

Skips a unit test if numpy is not recent enough.

onnx_diagnostic.ext_test_case.requires_onnx(version: str, msg: str = '') → Callable

Skips a unit test if onnx is not recent enough.

onnx_diagnostic.ext_test_case.requires_onnx_array_api(version: str, msg: str = '') → Callable

Skips a unit test if onnx-array-api is not recent enough.

onnx_diagnostic.ext_test_case.requires_onnxruntime(version: str, msg: str = '') → Callable

Skips a unit test if onnxruntime is not recent enough.

onnx_diagnostic.ext_test_case.requires_onnxruntime_training(push_back_batch: bool = False, ortmodule: bool = False, msg: str = '') → Callable

Skips a unit test if onnxruntime is not onnxruntime_training.

onnx_diagnostic.ext_test_case.requires_onnxscript(version: str, msg: str = '') → Callable

Skips a unit test if onnxscript is not recent enough.

onnx_diagnostic.ext_test_case.requires_python(version: Tuple[int, ...], msg: str = '')

Skips a test if python is too old.

Parameters:

• msg – to overwrite the message

• version – minimum version

onnx_diagnostic.ext_test_case.requires_sklearn(version: str, msg: str = '') → Callable

Skips a unit test if scikit-learn is not recent enough.

onnx_diagnostic.ext_test_case.requires_torch(version: str, msg: str = '') → Callable

Skips a unit test if pytorch is not recent enough.

onnx_diagnostic.ext_test_case.requires_transformers(version: str, msg: str = '', or_older_than: str | None = None) → Callable

Skips a unit test if transformers is not recent enough.

onnx_diagnostic.ext_test_case.requires_zoo(msg: str = '') → Callable

Skips a unit test if environment variable ZOO is not equal to 1.

onnx_diagnostic.ext_test_case.skipif_ci_apple(msg) → Callable

Skips a unit test if it runs on azure pipeline on Windows.

onnx_diagnostic.ext_test_case.skipif_ci_linux(msg) → Callable

Skips a unit test if it runs on azure pipeline on Linux.

onnx_diagnostic.ext_test_case.skipif_ci_windows(msg) → Callable

Skips a unit test if it runs on azure pipeline on Windows.

onnx_diagnostic.ext_test_case.statistics_on_file(filename: str) → Dict[str, str | int | float]

Computes statistics on a file.

<<<

import pprint
from onnx_diagnostic.ext_test_case import statistics_on_file, __file__

pprint.pprint(statistics_on_file(__file__))

>>>

    {'chars': 30150, 'ext': '.py', 'lines': 1016}

onnx_diagnostic.ext_test_case.statistics_on_folder(folder: str | List[str], pattern: str = '.*[.]((py|rst))$', aggregation: int = 0) → List[Dict[str, str | int | float]]

Computes statistics on files in a folder.

Parameters:

• folder – folder or folders to investigate

• pattern – file pattern

• aggregation – show the first subfolders

Returns:

list of dictionaries

<<<

import os
import pprint
from onnx_diagnostic.ext_test_case import statistics_on_folder, __file__

pprint.pprint(statistics_on_folder(os.path.dirname(__file__)))

>>>

    [{'chars': 2208, 'ext': '.py', 'lines': 76, 'name': 'doc.py'},
     {'chars': 22567,
      'ext': '.py',
      'lines': 823,
      'name': '_command_lines_parser.py'},
     {'chars': 30150, 'ext': '.py', 'lines': 1016, 'name': 'ext_test_case.py'},
     {'chars': 254, 'ext': '.py', 'lines': 11, 'name': 'api.py'},
     {'chars': 144, 'ext': '.py', 'lines': 4, 'name': '__init__.py'},
     {'chars': 65, 'ext': '.py', 'lines': 3, 'name': '__main__.py'},
     {'chars': 2031,
      'ext': '.py',
      'lines': 67,
      'name': 'tasks/sentence_similarity.py'},
     {'chars': 2246,
      'ext': '.py',
      'lines': 61,
      'name': 'tasks/mixture_of_expert.py'},
     {'chars': 2082,
      'ext': '.py',
      'lines': 67,
      'name': 'tasks/text_classification.py'},
     {'chars': 3561,
      'ext': '.py',
      'lines': 104,
      'name': 'tasks/zero_shot_image_classification.py'},
     {'chars': 8084,
      'ext': '.py',
      'lines': 253,
      'name': 'tasks/text_generation.py'},
     {'chars': 4913,
      'ext': '.py',
      'lines': 144,
      'name': 'tasks/automatic_speech_recognition.py'},
     {'chars': 5421, 'ext': '.py', 'lines': 176, 'name': 'tasks/summarization.py'},
     {'chars': 2016, 'ext': '.py', 'lines': 67, 'name': 'tasks/fill_mask.py'},
     {'chars': 11260,
      'ext': '.py',
      'lines': 361,
      'name': 'tasks/image_text_to_text.py'},
     {'chars': 3035,
      'ext': '.py',
      'lines': 106,
      'name': 'tasks/object_detection.py'},
     {'chars': 2224, 'ext': '.py', 'lines': 74, 'name': 'tasks/text_to_image.py'},
     {'chars': 3337,
      'ext': '.py',
      'lines': 115,
      'name': 'tasks/image_classification.py'},
     {'chars': 5698,
      'ext': '.py',
      'lines': 181,
      'name': 'tasks/text2text_generation.py'},
     {'chars': 3544,
      'ext': '.py',
      'lines': 123,
      'name': 'tasks/feature_extraction.py'},
     {'chars': 1887, 'ext': '.py', 'lines': 66, 'name': 'tasks/__init__.py'},
     {'chars': 3615,
      'ext': '.py',
      'lines': 117,
      'name': 'tasks/mask_generation.py'},
     {'chars': 20111, 'ext': '.py', 'lines': 609, 'name': 'helpers/ort_session.py'},
     {'chars': 40332, 'ext': '.py', 'lines': 1428, 'name': 'helpers/helper.py'},
     {'chars': 17795,
      'ext': '.py',
      'lines': 474,
      'name': 'helpers/cache_helper.py'},
     {'chars': 2905, 'ext': '.py', 'lines': 115, 'name': 'helpers/args_helper.py'},
     {'chars': 9276,
      'ext': '.py',
      'lines': 265,
      'name': 'helpers/model_builder_helper.py'},
     {'chars': 3303, 'ext': '.py', 'lines': 105, 'name': 'helpers/rt_helper.py'},
     {'chars': 8876, 'ext': '.py', 'lines': 272, 'name': 'helpers/graph_helper.py'},
     {'chars': 3043,
      'ext': '.py',
      'lines': 103,
      'name': 'helpers/config_helper.py'},
     {'chars': 3962, 'ext': '.py', 'lines': 128, 'name': 'helpers/doc_helper.py'},
     {'chars': 11256, 'ext': '.py', 'lines': 393, 'name': 'helpers/bench_run.py'},
     {'chars': 51179, 'ext': '.py', 'lines': 1727, 'name': 'helpers/log_helper.py'},
     {'chars': 26457, 'ext': '.py', 'lines': 971, 'name': 'helpers/onnx_helper.py'},
     {'chars': 13746,
      'ext': '.py',
      'lines': 487,
      'name': 'helpers/mini_onnx_builder.py'},
     {'chars': 23695,
      'ext': '.py',
      'lines': 755,
      'name': 'helpers/torch_helper.py'},
     {'chars': 75, 'ext': '.py', 'lines': 1, 'name': 'helpers/__init__.py'},
     {'chars': 10935, 'ext': '.py', 'lines': 380, 'name': 'helpers/_log_helper.py'},
     {'chars': 4328, 'ext': '.py', 'lines': 187, 'name': 'helpers/memory_peak.py'},
     {'chars': 751,
      'ext': '.py',
      'lines': 34,
      'name': 'reference/quantized_tensor.py'},
     {'chars': 2529,
      'ext': '.py',
      'lines': 78,
      'name': 'reference/report_results_comparison.py'},
     {'chars': 17732,
      'ext': '.py',
      'lines': 560,
      'name': 'reference/ort_evaluator.py'},
     {'chars': 6230, 'ext': '.py', 'lines': 221, 'name': 'reference/evaluator.py'},
     {'chars': 17365,
      'ext': '.py',
      'lines': 530,
      'name': 'reference/torch_evaluator.py'},
     {'chars': 192, 'ext': '.py', 'lines': 4, 'name': 'reference/__init__.py'},
     {'chars': 439,
      'ext': '.py',
      'lines': 20,
      'name': 'reference/ops/op_complex.py'},
     {'chars': 647,
      'ext': '.py',
      'lines': 27,
      'name': 'reference/ops/op_fused_matmul.py'},
     {'chars': 380,
      'ext': '.py',
      'lines': 14,
      'name': 'reference/ops/op_tri_matrix.py'},
     {'chars': 458,
      'ext': '.py',
      'lines': 17,
      'name': 'reference/ops/op_quick_gelu.py'},
     {'chars': 531, 'ext': '.py', 'lines': 15, 'name': 'reference/ops/op_slice.py'},
     {'chars': 2582,
      'ext': '.py',
      'lines': 88,
      'name': 'reference/ops/op_qlinear_conv.py'},
     {'chars': 1161,
      'ext': '.py',
      'lines': 59,
      'name': 'reference/ops/op_constant_of_shape.py'},
     {'chars': 667,
      'ext': '.py',
      'lines': 16,
      'name': 'reference/ops/op_scatternd_of_shape.py'},
     {'chars': 147,
      'ext': '.py',
      'lines': 5,
      'name': 'reference/ops/op_negxplus1.py'},
     {'chars': 220,
      'ext': '.py',
      'lines': 6,
      'name': 'reference/ops/op_simplified_layer_normalization.py'},
     {'chars': 295,
      'ext': '.py',
      'lines': 10,
      'name': 'reference/ops/op_transpose_cast.py'},
     {'chars': 140,
      'ext': '.py',
      'lines': 7,
      'name': 'reference/ops/op_memcpy_host.py'},
     {'chars': 1405,
      'ext': '.py',
      'lines': 51,
      'name': 'reference/ops/op_average_pool_grad.py'},
     {'chars': 317,
      'ext': '.py',
      'lines': 9,
      'name': 'reference/ops/op_gather_grad.py'},
     {'chars': 853,
      'ext': '.py',
      'lines': 35,
      'name': 'reference/ops/op_qlinear_average_pool.py'},
     {'chars': 684,
      'ext': '.py',
      'lines': 24,
      'name': 'reference/ops/op_gather.py'},
     {'chars': 1419,
      'ext': '.py',
      'lines': 52,
      'name': 'reference/ops/op_attention.py'},
     {'chars': 384,
      'ext': '.py',
      'lines': 13,
      'name': 'reference/ops/op_bias_softmax.py'},
     {'chars': 224,
      'ext': '.py',
      'lines': 10,
      'name': 'reference/ops/op_replace_zero.py'},
     {'chars': 350,
      'ext': '.py',
      'lines': 10,
      'name': 'reference/ops/op_skip_layer_normalization.py'},
     {'chars': 455,
      'ext': '.py',
      'lines': 17,
      'name': 'reference/ops/op_mul_sigmoid.py'},
     {'chars': 0, 'ext': '.py', 'lines': 0, 'name': 'reference/ops/__init__.py'},
     {'chars': 1139,
      'ext': '.py',
      'lines': 33,
      'name': 'reference/ops/op_gather_elements.py'},
     {'chars': 1354, 'ext': '.py', 'lines': 53, 'name': 'reference/ops/op_scan.py'},
     {'chars': 434,
      'ext': '.py',
      'lines': 16,
      'name': 'reference/ops/op_rotary.py'},
     {'chars': 383,
      'ext': '.py',
      'lines': 11,
      'name': 'reference/ops/op_concat.py'},
     {'chars': 1091,
      'ext': '.py',
      'lines': 44,
      'name': 'reference/ops/op_add_add_mul_mul.py'},
     {'chars': 2202,
      'ext': '.py',
      'lines': 82,
      'name': 'reference/ops/op_scatter_elements.py'},
     {'chars': 1152,
      'ext': '.py',
      'lines': 38,
      'name': 'reference/ops/op_cast_like.py'},
     {'chars': 3001,
      'ext': '.py',
      'lines': 101,
      'name': 'reference/torch_ops/controlflow_ops.py'},
     {'chars': 608,
      'ext': '.py',
      'lines': 26,
      'name': 'reference/torch_ops/generator_ops.py'},
     {'chars': 3741,
      'ext': '.py',
      'lines': 109,
      'name': 'reference/torch_ops/reduce_ops.py'},
     {'chars': 3141,
      'ext': '.py',
      'lines': 76,
      'name': 'reference/torch_ops/other_ops.py'},
     {'chars': 1732,
      'ext': '.py',
      'lines': 50,
      'name': 'reference/torch_ops/sequence_ops.py'},
     {'chars': 2065,
      'ext': '.py',
      'lines': 65,
      'name': 'reference/torch_ops/binary_ops.py'},
     {'chars': 7418,
      'ext': '.py',
      'lines': 256,
      'name': 'reference/torch_ops/_op_run.py'},
     {'chars': 893,
      'ext': '.py',
      'lines': 52,
      'name': 'reference/torch_ops/__init__.py'},
     {'chars': 1451,
      'ext': '.py',
      'lines': 54,
      'name': 'reference/torch_ops/unary_ops.py'},
     {'chars': 2201,
      'ext': '.py',
      'lines': 78,
      'name': 'reference/torch_ops/access_ops.py'},
     {'chars': 3152,
      'ext': '.py',
      'lines': 91,
      'name': 'reference/torch_ops/shape_ops.py'},
     {'chars': 4988,
      'ext': '.py',
      'lines': 157,
      'name': 'reference/torch_ops/nn_ops.py'},
     {'chars': 6201,
      'ext': '.py',
      'lines': 242,
      'name': 'torch_onnx/runtime_info.py'},
     {'chars': 0, 'ext': '.py', 'lines': 0, 'name': 'torch_onnx/__init__.py'},
     {'chars': 11213, 'ext': '.py', 'lines': 349, 'name': 'torch_onnx/sbs.py'},
     {'chars': 5209, 'ext': '.py', 'lines': 137, 'name': 'export/shape_helper.py'},
     {'chars': 4417, 'ext': '.py', 'lines': 143, 'name': 'export/validate.py'},
     {'chars': 26330,
      'ext': '.py',
      'lines': 846,
      'name': 'export/dynamic_shapes.py'},
     {'chars': 92, 'ext': '.py', 'lines': 2, 'name': 'export/__init__.py'},
     {'chars': 25046,
      'ext': '.py',
      'lines': 857,
      'name': 'torch_export_patches/patch_module.py'},
     {'chars': 8338,
      'ext': '.py',
      'lines': 239,
      'name': 'torch_export_patches/onnx_export_serialization.py'},
     {'chars': 2434,
      'ext': '.py',
      'lines': 75,
      'name': 'torch_export_patches/patch_expressions.py'},
     {'chars': 5125,
      'ext': '.py',
      'lines': 170,
      'name': 'torch_export_patches/patch_inputs.py'},
     {'chars': 607,
      'ext': '.py',
      'lines': 13,
      'name': 'torch_export_patches/__init__.py'},
     {'chars': 16008,
      'ext': '.py',
      'lines': 468,
      'name': 'torch_export_patches/onnx_export_errors.py'},
     {'chars': 5483,
      'ext': '.py',
      'lines': 131,
      'name': 'torch_export_patches/patch_module_helper.py'},
     {'chars': 46212,
      'ext': '.py',
      'lines': 1244,
      'name': 'torch_export_patches/patches/patch_transformers.py'},
     {'chars': 12235,
      'ext': '.py',
      'lines': 376,
      'name': 'torch_export_patches/patches/patch_torch.py'},
     {'chars': 0,
      'ext': '.py',
      'lines': 0,
      'name': 'torch_export_patches/patches/__init__.py'},
     {'chars': 19029,
      'ext': '.py',
      'lines': 594,
      'name': 'torch_export_patches/eval/model_cases.py'},
     {'chars': 14953,
      'ext': '.py',
      'lines': 582,
      'name': 'torch_export_patches/eval/__init__.py'},
     {'chars': 747,
      'ext': '.py',
      'lines': 25,
      'name': 'torch_export_patches/serialization/diffusers_impl.py'},
     {'chars': 7815,
      'ext': '.py',
      'lines': 184,
      'name': 'torch_export_patches/serialization/transformers_impl.py'},
     {'chars': 1532,
      'ext': '.py',
      'lines': 36,
      'name': 'torch_export_patches/serialization/__init__.py'},
     {'chars': 46791,
      'ext': '.py',
      'lines': 1470,
      'name': 'torch_models/validate.py'},
     {'chars': 0, 'ext': '.py', 'lines': 0, 'name': 'torch_models/__init__.py'},
     {'chars': 82, 'ext': '.py', 'lines': 2, 'name': 'torch_models/llms.py'},
     {'chars': 9037,
      'ext': '.py',
      'lines': 306,
      'name': 'torch_models/hghub/hub_api.py'},
     {'chars': 7845,
      'ext': '.py',
      'lines': 231,
      'name': 'torch_models/hghub/model_inputs.py'},
     {'chars': 235593,
      'ext': '.py',
      'lines': 4378,
      'name': 'torch_models/hghub/hub_data_cached_configs.py'},
     {'chars': 7518,
      'ext': '.py',
      'lines': 213,
      'name': 'torch_models/hghub/hub_data.py'},
     {'chars': 54,
      'ext': '.py',
      'lines': 1,
      'name': 'torch_models/hghub/__init__.py'},
     {'chars': 2642,
      'ext': '.py',
      'lines': 84,
      'name': 'torch_models/untrained/llm_phi2.py'},
     {'chars': 1911,
      'ext': '.py',
      'lines': 63,
      'name': 'torch_models/untrained/llm_tiny_llm.py'},
     {'chars': 0,
      'ext': '.py',
      'lines': 0,
      'name': 'torch_models/untrained/__init__.py'}]

Aggregated:

<<<

import os
import pprint
from onnx_diagnostic.ext_test_case import statistics_on_folder, __file__

pprint.pprint(statistics_on_folder(os.path.dirname(__file__), aggregation=1))

>>>

    [{'chars': 2208, 'dir': '', 'ext': '.py', 'lines': 76, 'name': 'doc.py'},
     {'chars': 22567,
      'dir': '',
      'ext': '.py',
      'lines': 823,
      'name': '_command_lines_parser.py'},
     {'chars': 30150,
      'dir': '',
      'ext': '.py',
      'lines': 1016,
      'name': 'ext_test_case.py'},
     {'chars': 254, 'dir': '', 'ext': '.py', 'lines': 11, 'name': 'api.py'},
     {'chars': 144, 'dir': '', 'ext': '.py', 'lines': 4, 'name': '__init__.py'},
     {'chars': 65, 'dir': '', 'ext': '.py', 'lines': 3, 'name': '__main__.py'},
     {'chars': 2031,
      'dir': 'tasks',
      'ext': '.py',
      'lines': 67,
      'name': 'tasks/sentence_similarity.py'},
     {'chars': 2246,
      'dir': 'tasks',
      'ext': '.py',
      'lines': 61,
      'name': 'tasks/mixture_of_expert.py'},
     {'chars': 2082,
      'dir': 'tasks',
      'ext': '.py',
      'lines': 67,
      'name': 'tasks/text_classification.py'},
     {'chars': 3561,
      'dir': 'tasks',
      'ext': '.py',
      'lines': 104,
      'name': 'tasks/zero_shot_image_classification.py'},
     {'chars': 8084,
      'dir': 'tasks',
      'ext': '.py',
      'lines': 253,
      'name': 'tasks/text_generation.py'},
     {'chars': 4913,
      'dir': 'tasks',
      'ext': '.py',
      'lines': 144,
      'name': 'tasks/automatic_speech_recognition.py'},
     {'chars': 5421,
      'dir': 'tasks',
      'ext': '.py',
      'lines': 176,
      'name': 'tasks/summarization.py'},
     {'chars': 2016,
      'dir': 'tasks',
      'ext': '.py',
      'lines': 67,
      'name': 'tasks/fill_mask.py'},
     {'chars': 11260,
      'dir': 'tasks',
      'ext': '.py',
      'lines': 361,
      'name': 'tasks/image_text_to_text.py'},
     {'chars': 3035,
      'dir': 'tasks',
      'ext': '.py',
      'lines': 106,
      'name': 'tasks/object_detection.py'},
     {'chars': 2224,
      'dir': 'tasks',
      'ext': '.py',
      'lines': 74,
      'name': 'tasks/text_to_image.py'},
     {'chars': 3337,
      'dir': 'tasks',
      'ext': '.py',
      'lines': 115,
      'name': 'tasks/image_classification.py'},
     {'chars': 5698,
      'dir': 'tasks',
      'ext': '.py',
      'lines': 181,
      'name': 'tasks/text2text_generation.py'},
     {'chars': 3544,
      'dir': 'tasks',
      'ext': '.py',
      'lines': 123,
      'name': 'tasks/feature_extraction.py'},
     {'chars': 1887,
      'dir': 'tasks',
      'ext': '.py',
      'lines': 66,
      'name': 'tasks/__init__.py'},
     {'chars': 3615,
      'dir': 'tasks',
      'ext': '.py',
      'lines': 117,
      'name': 'tasks/mask_generation.py'},
     {'chars': 20111,
      'dir': 'helpers',
      'ext': '.py',
      'lines': 609,
      'name': 'helpers/ort_session.py'},
     {'chars': 40332,
      'dir': 'helpers',
      'ext': '.py',
      'lines': 1428,
      'name': 'helpers/helper.py'},
     {'chars': 17795,
      'dir': 'helpers',
      'ext': '.py',
      'lines': 474,
      'name': 'helpers/cache_helper.py'},
     {'chars': 2905,
      'dir': 'helpers',
      'ext': '.py',
      'lines': 115,
      'name': 'helpers/args_helper.py'},
     {'chars': 9276,
      'dir': 'helpers',
      'ext': '.py',
      'lines': 265,
      'name': 'helpers/model_builder_helper.py'},
     {'chars': 3303,
      'dir': 'helpers',
      'ext': '.py',
      'lines': 105,
      'name': 'helpers/rt_helper.py'},
     {'chars': 8876,
      'dir': 'helpers',
      'ext': '.py',
      'lines': 272,
      'name': 'helpers/graph_helper.py'},
     {'chars': 3043,
      'dir': 'helpers',
      'ext': '.py',
      'lines': 103,
      'name': 'helpers/config_helper.py'},
     {'chars': 3962,
      'dir': 'helpers',
      'ext': '.py',
      'lines': 128,
      'name': 'helpers/doc_helper.py'},
     {'chars': 11256,
      'dir': 'helpers',
      'ext': '.py',
      'lines': 393,
      'name': 'helpers/bench_run.py'},
     {'chars': 51179,
      'dir': 'helpers',
      'ext': '.py',
      'lines': 1727,
      'name': 'helpers/log_helper.py'},
     {'chars': 26457,
      'dir': 'helpers',
      'ext': '.py',
      'lines': 971,
      'name': 'helpers/onnx_helper.py'},
     {'chars': 13746,
      'dir': 'helpers',
      'ext': '.py',
      'lines': 487,
      'name': 'helpers/mini_onnx_builder.py'},
     {'chars': 23695,
      'dir': 'helpers',
      'ext': '.py',
      'lines': 755,
      'name': 'helpers/torch_helper.py'},
     {'chars': 75,
      'dir': 'helpers',
      'ext': '.py',
      'lines': 1,
      'name': 'helpers/__init__.py'},
     {'chars': 10935,
      'dir': 'helpers',
      'ext': '.py',
      'lines': 380,
      'name': 'helpers/_log_helper.py'},
     {'chars': 4328,
      'dir': 'helpers',
      'ext': '.py',
      'lines': 187,
      'name': 'helpers/memory_peak.py'},
     {'chars': 751,
      'dir': 'reference',
      'ext': '.py',
      'lines': 34,
      'name': 'reference/quantized_tensor.py'},
     {'chars': 2529,
      'dir': 'reference',
      'ext': '.py',
      'lines': 78,
      'name': 'reference/report_results_comparison.py'},
     {'chars': 17732,
      'dir': 'reference',
      'ext': '.py',
      'lines': 560,
      'name': 'reference/ort_evaluator.py'},
     {'chars': 6230,
      'dir': 'reference',
      'ext': '.py',
      'lines': 221,
      'name': 'reference/evaluator.py'},
     {'chars': 17365,
      'dir': 'reference',
      'ext': '.py',
      'lines': 530,
      'name': 'reference/torch_evaluator.py'},
     {'chars': 192,
      'dir': 'reference',
      'ext': '.py',
      'lines': 4,
      'name': 'reference/__init__.py'},
     {'chars': 439,
      'dir': 'reference',
      'ext': '.py',
      'lines': 20,
      'name': 'reference/ops/op_complex.py'},
     {'chars': 647,
      'dir': 'reference',
      'ext': '.py',
      'lines': 27,
      'name': 'reference/ops/op_fused_matmul.py'},
     {'chars': 380,
      'dir': 'reference',
      'ext': '.py',
      'lines': 14,
      'name': 'reference/ops/op_tri_matrix.py'},
     {'chars': 458,
      'dir': 'reference',
      'ext': '.py',
      'lines': 17,
      'name': 'reference/ops/op_quick_gelu.py'},
     {'chars': 531,
      'dir': 'reference',
      'ext': '.py',
      'lines': 15,
      'name': 'reference/ops/op_slice.py'},
     {'chars': 2582,
      'dir': 'reference',
      'ext': '.py',
      'lines': 88,
      'name': 'reference/ops/op_qlinear_conv.py'},
     {'chars': 1161,
      'dir': 'reference',
      'ext': '.py',
      'lines': 59,
      'name': 'reference/ops/op_constant_of_shape.py'},
     {'chars': 667,
      'dir': 'reference',
      'ext': '.py',
      'lines': 16,
      'name': 'reference/ops/op_scatternd_of_shape.py'},
     {'chars': 147,
      'dir': 'reference',
      'ext': '.py',
      'lines': 5,
      'name': 'reference/ops/op_negxplus1.py'},
     {'chars': 220,
      'dir': 'reference',
      'ext': '.py',
      'lines': 6,
      'name': 'reference/ops/op_simplified_layer_normalization.py'},
     {'chars': 295,
      'dir': 'reference',
      'ext': '.py',
      'lines': 10,
      'name': 'reference/ops/op_transpose_cast.py'},
     {'chars': 140,
      'dir': 'reference',
      'ext': '.py',
      'lines': 7,
      'name': 'reference/ops/op_memcpy_host.py'},
     {'chars': 1405,
      'dir': 'reference',
      'ext': '.py',
      'lines': 51,
      'name': 'reference/ops/op_average_pool_grad.py'},
     {'chars': 317,
      'dir': 'reference',
      'ext': '.py',
      'lines': 9,
      'name': 'reference/ops/op_gather_grad.py'},
     {'chars': 853,
      'dir': 'reference',
      'ext': '.py',
      'lines': 35,
      'name': 'reference/ops/op_qlinear_average_pool.py'},
     {'chars': 684,
      'dir': 'reference',
      'ext': '.py',
      'lines': 24,
      'name': 'reference/ops/op_gather.py'},
     {'chars': 1419,
      'dir': 'reference',
      'ext': '.py',
      'lines': 52,
      'name': 'reference/ops/op_attention.py'},
     {'chars': 384,
      'dir': 'reference',
      'ext': '.py',
      'lines': 13,
      'name': 'reference/ops/op_bias_softmax.py'},
     {'chars': 224,
      'dir': 'reference',
      'ext': '.py',
      'lines': 10,
      'name': 'reference/ops/op_replace_zero.py'},
     {'chars': 350,
      'dir': 'reference',
      'ext': '.py',
      'lines': 10,
      'name': 'reference/ops/op_skip_layer_normalization.py'},
     {'chars': 455,
      'dir': 'reference',
      'ext': '.py',
      'lines': 17,
      'name': 'reference/ops/op_mul_sigmoid.py'},
     {'chars': 0,
      'dir': 'reference',
      'ext': '.py',
      'lines': 0,
      'name': 'reference/ops/__init__.py'},
     {'chars': 1139,
      'dir': 'reference',
      'ext': '.py',
      'lines': 33,
      'name': 'reference/ops/op_gather_elements.py'},
     {'chars': 1354,
      'dir': 'reference',
      'ext': '.py',
      'lines': 53,
      'name': 'reference/ops/op_scan.py'},
     {'chars': 434,
      'dir': 'reference',
      'ext': '.py',
      'lines': 16,
      'name': 'reference/ops/op_rotary.py'},
     {'chars': 383,
      'dir': 'reference',
      'ext': '.py',
      'lines': 11,
      'name': 'reference/ops/op_concat.py'},
     {'chars': 1091,
      'dir': 'reference',
      'ext': '.py',
      'lines': 44,
      'name': 'reference/ops/op_add_add_mul_mul.py'},
     {'chars': 2202,
      'dir': 'reference',
      'ext': '.py',
      'lines': 82,
      'name': 'reference/ops/op_scatter_elements.py'},
     {'chars': 1152,
      'dir': 'reference',
      'ext': '.py',
      'lines': 38,
      'name': 'reference/ops/op_cast_like.py'},
     {'chars': 3001,
      'dir': 'reference',
      'ext': '.py',
      'lines': 101,
      'name': 'reference/torch_ops/controlflow_ops.py'},
     {'chars': 608,
      'dir': 'reference',
      'ext': '.py',
      'lines': 26,
      'name': 'reference/torch_ops/generator_ops.py'},
     {'chars': 3741,
      'dir': 'reference',
      'ext': '.py',
      'lines': 109,
      'name': 'reference/torch_ops/reduce_ops.py'},
     {'chars': 3141,
      'dir': 'reference',
      'ext': '.py',
      'lines': 76,
      'name': 'reference/torch_ops/other_ops.py'},
     {'chars': 1732,
      'dir': 'reference',
      'ext': '.py',
      'lines': 50,
      'name': 'reference/torch_ops/sequence_ops.py'},
     {'chars': 2065,
      'dir': 'reference',
      'ext': '.py',
      'lines': 65,
      'name': 'reference/torch_ops/binary_ops.py'},
     {'chars': 7418,
      'dir': 'reference',
      'ext': '.py',
      'lines': 256,
      'name': 'reference/torch_ops/_op_run.py'},
     {'chars': 893,
      'dir': 'reference',
      'ext': '.py',
      'lines': 52,
      'name': 'reference/torch_ops/__init__.py'},
     {'chars': 1451,
      'dir': 'reference',
      'ext': '.py',
      'lines': 54,
      'name': 'reference/torch_ops/unary_ops.py'},
     {'chars': 2201,
      'dir': 'reference',
      'ext': '.py',
      'lines': 78,
      'name': 'reference/torch_ops/access_ops.py'},
     {'chars': 3152,
      'dir': 'reference',
      'ext': '.py',
      'lines': 91,
      'name': 'reference/torch_ops/shape_ops.py'},
     {'chars': 4988,
      'dir': 'reference',
      'ext': '.py',
      'lines': 157,
      'name': 'reference/torch_ops/nn_ops.py'},
     {'chars': 6201,
      'dir': 'torch_onnx',
      'ext': '.py',
      'lines': 242,
      'name': 'torch_onnx/runtime_info.py'},
     {'chars': 0,
      'dir': 'torch_onnx',
      'ext': '.py',
      'lines': 0,
      'name': 'torch_onnx/__init__.py'},
     {'chars': 11213,
      'dir': 'torch_onnx',
      'ext': '.py',
      'lines': 349,
      'name': 'torch_onnx/sbs.py'},
     {'chars': 5209,
      'dir': 'export',
      'ext': '.py',
      'lines': 137,
      'name': 'export/shape_helper.py'},
     {'chars': 4417,
      'dir': 'export',
      'ext': '.py',
      'lines': 143,
      'name': 'export/validate.py'},
     {'chars': 26330,
      'dir': 'export',
      'ext': '.py',
      'lines': 846,
      'name': 'export/dynamic_shapes.py'},
     {'chars': 92,
      'dir': 'export',
      'ext': '.py',
      'lines': 2,
      'name': 'export/__init__.py'},
     {'chars': 25046,
      'dir': 'torch_export_patches',
      'ext': '.py',
      'lines': 857,
      'name': 'torch_export_patches/patch_module.py'},
     {'chars': 8338,
      'dir': 'torch_export_patches',
      'ext': '.py',
      'lines': 239,
      'name': 'torch_export_patches/onnx_export_serialization.py'},
     {'chars': 2434,
      'dir': 'torch_export_patches',
      'ext': '.py',
      'lines': 75,
      'name': 'torch_export_patches/patch_expressions.py'},
     {'chars': 5125,
      'dir': 'torch_export_patches',
      'ext': '.py',
      'lines': 170,
      'name': 'torch_export_patches/patch_inputs.py'},
     {'chars': 607,
      'dir': 'torch_export_patches',
      'ext': '.py',
      'lines': 13,
      'name': 'torch_export_patches/__init__.py'},
     {'chars': 16008,
      'dir': 'torch_export_patches',
      'ext': '.py',
      'lines': 468,
      'name': 'torch_export_patches/onnx_export_errors.py'},
     {'chars': 5483,
      'dir': 'torch_export_patches',
      'ext': '.py',
      'lines': 131,
      'name': 'torch_export_patches/patch_module_helper.py'},
     {'chars': 46212,
      'dir': 'torch_export_patches',
      'ext': '.py',
      'lines': 1244,
      'name': 'torch_export_patches/patches/patch_transformers.py'},
     {'chars': 12235,
      'dir': 'torch_export_patches',
      'ext': '.py',
      'lines': 376,
      'name': 'torch_export_patches/patches/patch_torch.py'},
     {'chars': 0,
      'dir': 'torch_export_patches',
      'ext': '.py',
      'lines': 0,
      'name': 'torch_export_patches/patches/__init__.py'},
     {'chars': 19029,
      'dir': 'torch_export_patches',
      'ext': '.py',
      'lines': 594,
      'name': 'torch_export_patches/eval/model_cases.py'},
     {'chars': 14953,
      'dir': 'torch_export_patches',
      'ext': '.py',
      'lines': 582,
      'name': 'torch_export_patches/eval/__init__.py'},
     {'chars': 747,
      'dir': 'torch_export_patches',
      'ext': '.py',
      'lines': 25,
      'name': 'torch_export_patches/serialization/diffusers_impl.py'},
     {'chars': 7815,
      'dir': 'torch_export_patches',
      'ext': '.py',
      'lines': 184,
      'name': 'torch_export_patches/serialization/transformers_impl.py'},
     {'chars': 1532,
      'dir': 'torch_export_patches',
      'ext': '.py',
      'lines': 36,
      'name': 'torch_export_patches/serialization/__init__.py'},
     {'chars': 46791,
      'dir': 'torch_models',
      'ext': '.py',
      'lines': 1470,
      'name': 'torch_models/validate.py'},
     {'chars': 0,
      'dir': 'torch_models',
      'ext': '.py',
      'lines': 0,
      'name': 'torch_models/__init__.py'},
     {'chars': 82,
      'dir': 'torch_models',
      'ext': '.py',
      'lines': 2,
      'name': 'torch_models/llms.py'},
     {'chars': 9037,
      'dir': 'torch_models',
      'ext': '.py',
      'lines': 306,
      'name': 'torch_models/hghub/hub_api.py'},
     {'chars': 7845,
      'dir': 'torch_models',
      'ext': '.py',
      'lines': 231,
      'name': 'torch_models/hghub/model_inputs.py'},
     {'chars': 235593,
      'dir': 'torch_models',
      'ext': '.py',
      'lines': 4378,
      'name': 'torch_models/hghub/hub_data_cached_configs.py'},
     {'chars': 7518,
      'dir': 'torch_models',
      'ext': '.py',
      'lines': 213,
      'name': 'torch_models/hghub/hub_data.py'},
     {'chars': 54,
      'dir': 'torch_models',
      'ext': '.py',
      'lines': 1,
      'name': 'torch_models/hghub/__init__.py'},
     {'chars': 2642,
      'dir': 'torch_models',
      'ext': '.py',
      'lines': 84,
      'name': 'torch_models/untrained/llm_phi2.py'},
     {'chars': 1911,
      'dir': 'torch_models',
      'ext': '.py',
      'lines': 63,
      'name': 'torch_models/untrained/llm_tiny_llm.py'},
     {'chars': 0,
      'dir': 'torch_models',
      'ext': '.py',
      'lines': 0,
      'name': 'torch_models/untrained/__init__.py'}]

onnx_diagnostic.ext_test_case.unit_test_going()

Enables a flag telling the script is running while testing it. Avois unit tests to be very long.