Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode Skip to content
experimental-experiment 0.1.1 documentation
Logo
experimental-experiment 0.1.1 documentation

Contents

  • Design
    • Custom Exporter
    • Pattern Optimizer
    • Dynamo Backends
  • Tutorial
    • Unexpected Errors
    • Start from a docker
  • API
    • .gradient
      • .gradient.ops
        • .gradient.ops.op_broadcast_gradient_args
      • .gradient.grad_helper
      • .gradient.loss_helper
    • .reference
      • .reference.ops
        • .reference.ops.op_add_add_mul_mul
        • .reference.ops.op_average_pool_grad
        • .reference.ops.op_cast_like
        • .reference.ops.op_complex
        • .reference.ops.op_concat
        • .reference.ops.op_constant_of_shape
        • .reference.ops.op_fused_matmul
        • .reference.ops.op_gather_grad
        • .reference.ops.op_memcpy_host
        • .reference.ops.op_mul_sigmoid
        • .reference.ops.op_negxplus1
        • .reference.ops.op_quick_gelu
        • .reference.ops.op_replace_zero
        • .reference.ops.op_rotary
        • .reference.ops.op_qlinear_average_pool
        • .reference.ops.op_qlinear_conv
        • .reference.ops.op_scatter_elements
        • .reference.ops.op_scatternd_of_shape
        • .reference.ops.op_simplified_layer_normalization
        • .reference.ops.op_skip_layer_normalization
        • .reference.ops.op_slice
        • .reference.ops.op_transpose_cast
        • .reference.ops.op_tri_matrix
      • .reference.evaluator
      • .reference.ort_evaluator
      • .reference.quantized_tensor
    • .convert
      • .convert.convert_helper
      • .convert.ort_helper
    • .plotting
      • .plotting.data
      • .plotting.memory
    • .skl
      • .skl.convert
      • .skl.helpers
    • .torch_interpreter
      • .torch_interpreter._aten_functions
      • .torch_interpreter._aten_functions_attention
      • .torch_interpreter._non_aten_functions
      • .torch_interpreter._aten_methods
      • .torch_interpreter._doc_
      • .torch_interpreter._exceptions
      • .torch_interpreter._prims_functions
      • .torch_interpreter._torch_helper
      • .torch_interpreter.aten_functions
      • .torch_interpreter.aten_methods
      • .torch_interpreter.dispatcher
      • .torch_interpreter.export_options
      • .torch_interpreter.interpreter
      • .torch_interpreter.investigate_helper
      • .torch_interpreter.onnx_export
      • .torch_interpreter.oxs_dispatcher
      • .torch_interpreter.oxs_opset
      • .torch_interpreter.piece_by_piece
      • .torch_interpreter.piece_by_piece_serialize
      • .torch_interpreter.tracing
    • .torch_models
      • .torch_models.diffusion_model_helper
      • .torch_models.dump_helper
      • .torch_models.llama_helper
      • .torch_models.llm_model_helper
      • .torch_models.llm_model_setup
      • .torch_models.mistral_helper
      • .torch_models.phi3_helper
      • .torch_models.phi_helper
      • .torch_models.training_helper
    • .xbuilder
      • .xbuilder._graph_builder_runtime
      • .xbuilder._onnx_helper
      • .xbuilder._shape_helper
      • .xbuilder.expression_dimension
      • .xbuilder.graph_builder
      • .xbuilder.graph_builder_opset
      • .xbuilder.model_container
      • .xbuilder.optimization_options
      • .xbuilder.reverse_graph_builder
      • .xbuilder.shape_type_compute
      • .xbuilder.type_inference
    • .xoptim
      • .xoptim.patterns_investigation
        • .xoptim.patterns_investigation.element_wise
        • .xoptim.patterns_investigation.llm_patterns
      • .xoptim.patterns_ml
        • .xoptim.patterns_ml.tree_ensemble
      • .xoptim.patterns_exp
        • .xoptim.patterns_exp.binary_operators
        • .xoptim.patterns_exp.constant_of_shape_scatter_nd
        • .xoptim.patterns_exp.constants
        • .xoptim.patterns_exp.simple_rotary
        • .xoptim.patterns_exp.unary_operators
        • .xoptim.patterns_exp.where_replace
      • .xoptim.patterns
        • .xoptim.patterns.onnx_any
        • .xoptim.patterns.onnx_cast
        • .xoptim.patterns.onnx_clip
        • .xoptim.patterns.onnx_constants
        • .xoptim.patterns.onnx_conv
        • .xoptim.patterns.onnx_dropout
        • .xoptim.patterns.onnx_equal
        • .xoptim.patterns.onnx_expand
        • .xoptim.patterns.onnx_functions
        • .xoptim.patterns.onnx_layer_normalization
        • .xoptim.patterns.onnx_matmul
        • .xoptim.patterns.onnx_mul
        • .xoptim.patterns.onnx_reduce
        • .xoptim.patterns.onnx_reshape
        • .xoptim.patterns.onnx_rotary
        • .xoptim.patterns.onnx_shape
        • .xoptim.patterns.onnx_slice
        • .xoptim.patterns.onnx_split
        • .xoptim.patterns.onnx_sub
        • .xoptim.patterns.onnx_sequence
        • .xoptim.patterns.onnx_transpose
        • .xoptim.patterns.onnx_unsqueeze
      • .xoptim.patterns_ort
        • .xoptim.patterns_ort.activation
        • .xoptim.patterns_ort.activation_grad
        • .xoptim.patterns_ort.attention_patterns
        • .xoptim.patterns_ort.batch_normalization
        • .xoptim.patterns_ort.fused_conv
        • .xoptim.patterns_ort.fused_matmul
        • .xoptim.patterns_ort.gather_grad
        • .xoptim.patterns_ort.llm_optim
        • .xoptim.patterns_ort.simplified_layer_normalization
      • .xoptim.patterns_fix
        • .xoptim.patterns_fix.add_reduction_scatter_nd
      • .xoptim.graph_builder_optim
      • .xoptim.order_optim
      • .xoptim.patterns_api
      • .xoptim.repeated_optim
      • .xoptim.unfused
    • .torch_dynamo
      • .torch_dynamo._dynamo_exporter
      • .torch_dynamo.backend_helper
      • .torch_dynamo.debug_backend
      • .torch_dynamo.fast_backend
      • experimental_experiment.torch_dynamo.partition
    • .torch_bench
      • experimental_experiment.torch_bench._bash_bench_benchmark_runner
      • experimental_experiment.torch_bench._bash_bench_benchmark_runner_agg
      • experimental_experiment.torch_bench._bash_bench_benchmark_runner_agg_helper
      • experimental_experiment.torch_bench._bash_bench_cmd
      • experimental_experiment.torch_bench._bash_bench_model_runner
      • experimental_experiment.torch_bench._bash_bench_models_helper
      • experimental_experiment.torch_bench._bash_bench_set_dummies
      • experimental_experiment.torch_bench._bash_bench_set_explicit
      • experimental_experiment.torch_bench._bash_bench_set_huggingface
      • experimental_experiment.torch_bench._bash_bench_set_timm
      • experimental_experiment.torch_bench._bash_bench_set_torchbench
      • experimental_experiment.torch_bench._bash_bench_set_torchbench_ado
      • experimental_experiment.torch_bench._bash_bench_untrained
      • experimental_experiment.torch_bench._dort_cmd_common
      • experimental_experiment.torch_bench._dort_cmd_common_models
      • .torch_bench.bash_bench_agg
      • .torch_bench.bash_bench_explicit
      • .torch_bench.bash_bench_huggingface
      • .torch_bench.bash_bench_timm
      • .torch_bench.bash_bench_torchbench
      • .torch_bench.bash_bench_torchbench_ado
      • .torch_bench.bash_bench_untrained
      • .torch_bench.check_model
      • .torch_bench.dort_bench
      • .torch_bench.dort_bench_profile
      • .torch_bench.dort_profile
      • .torch_bench.export_model
      • .torch_bench.export_model_helper
    • ._bench_test
    • ._command_lines_parser
    • .args
    • .bench_run
    • .checks
    • .ext_test_case
    • .helpers
    • .memory_peak
    • .model_run
    • .onnx_tools
    • .ort_session
    • .torch_test_helper
  • Galleries of Examples and Recipes
    • Examples Gallery
      • 101: A custom backend for torch
      • 101: Linear Regression and export to ONNX
      • 101: Onnx Model Optimization based on Pattern Rewriting
      • 101: Onnx Model Rewriting
      • 101: Profile an existing model with onnxruntime
      • 101: Some dummy examples with torch.export.export
      • 102: Examples with onnxscript
      • 102: First test with ExecuTorch
      • 102: Tweak onnx export
      • 201: Evaluate different ways to export a torch model to ONNX
      • 201: Use torch to export a scikit-learn model into ONNX
      • Playground for big optimization pattern
    • Exporter Recipes Gallery
      • Check the exporter on a dummy from HuggingFace
      • Export Phi-3.5-mini-instruct piece by piece
      • Export Phi-3.5-mini-instruct with draft_export
      • Export Phi-3.5-mini-instruct with report_exportability
      • Export a model with a loop (scan)
      • to_onnx and a custom operator inplace
      • to_onnx and a custom operator registered with a function
      • to_onnx and a model with a test
      • to_onnx and padding one dimension to a mulitple of a constant
      • to_onnx and submodules from LLMs
  • Command Lines
    • Benchmarks from the command line
      • experimental_experiment.torch_bench.dort_bench
      • experimental_experiment.torch_bench.dort_profile
      • Interesting scripts or command lines
      • Measuring the exporters on a short list of sets of models
    • Tools from the command line
      • python -m experimental_experiment lighten and unlighten
      • python -m experimental_experiment optimize
      • python -m experimental_experiment print
      • python -m experimental_experiment run
  • Miscellaneous
    • Export Times
    • Long Outputs uneasy to read
    • Supported Models By the Custom Backend
      • Phi

More

  • Change Logs
  • License
Back to top
View this page

Exporter Recipes Gallery¶

A model can be converted to ONNX if torch.export.export() is able to convert that model into a graph. This is not always possible but usually possible with some code changes. These changes may not be desired as they may hurt the performance and make the code more complex than it should be. The conversion is a necessary step to be able to use ONNX. Next examples shows some recurrent code patterns and ways to rewrite them so that the exporter works.

See torch.onnx.export: export to ONNX for an organized version of this gallery.

A couple of examples to illustrate different implementation of dot product (see also sphinx-gallery).

Getting started¶

pytorch nightly build should be installed, see Start Locally.

git clone https://github.com/sdpython/experimental-experiment.git
pip install onnxruntime-gpu nvidia-ml-py
pip install -r requirements-dev.txt
export PYTHONPATH=$PYTHONPATH:<this folder>

Common Errors¶

Some of them are exposed in the examples. Others may be found at Frequent Exceptions or Errors with the Exporter.

Check the exporter on a dummy from HuggingFace

Check the exporter on a dummy from HuggingFace

Export Phi-3.5-mini-instruct piece by piece

Export Phi-3.5-mini-instruct piece by piece

Export Phi-3.5-mini-instruct with draft_export

Export Phi-3.5-mini-instruct with draft_export

Export Phi-3.5-mini-instruct with report_exportability

Export Phi-3.5-mini-instruct with report_exportability

Export a model with a loop (scan)

Export a model with a loop (scan)

to_onnx and a custom operator inplace

to_onnx and a custom operator inplace

to_onnx and a custom operator registered with a function

to_onnx and a custom operator registered with a function

to_onnx and a model with a test

to_onnx and a model with a test

to_onnx and padding one dimension to a mulitple of a constant

to_onnx and padding one dimension to a mulitple of a constant

to_onnx and submodules from LLMs

to_onnx and submodules from LLMs

Download all examples in Python source code: auto_recipes_python.zip

Download all examples in Jupyter notebooks: auto_recipes_jupyter.zip

Gallery generated by Sphinx-Gallery

Next
Check the exporter on a dummy from HuggingFace
Previous
Playground for big optimization pattern
Copyright © 2023-2024
Made with Sphinx and @pradyunsg's Furo
On this page
  • Exporter Recipes Gallery
    • Getting started
    • Common Errors