-m onnx_diagnostic validate … validate a model id

The command line is a wrapper around function onnx_diagnostic.torch_models.validate.validate_model().

Description

The command lines validate a model id available on HuggingFace but not only. It creates dummy inputs, runs the models on them, exports the model, measures the discrepancies…

    usage: validate [-h] [-m MID] [-t TASK] [-e EXPORT] [--opt OPT] [-r | --run | --no-run] [-q | --quiet | --no-quiet] [--patch [PATCH ...]] [--rewrite | --no-rewrite]
                    [--stop-if-static STOP_IF_STATIC] [--same-as-trained | --no-same-as-trained] [--trained | --no-trained] [--inputs2 INPUTS2]
                    [--runtime {onnxruntime,torch,ref,orteval,orteval10}] [-o DUMP_FOLDER] [--drop DROP] [--opset OPSET] [--subfolder SUBFOLDER] [--ortfusiontype ORTFUSIONTYPE]
                    [-v VERBOSE] [--dtype DTYPE] [--device DEVICE] [--iop [KEY=VALUE ...]] [--mop [KEY=VALUE ...]] [--repeat REPEAT] [--warmup WARMUP] [--outnames OUTNAMES]
                    [--ort-logs | --no-ort-logs] [--quiet-input-sets QUIET_INPUT_SETS]
    
    Validates a model for a particular task given the model id.
    It exports the model and then validates it by computing the discrepancies
    on different input sets.
    
    options:
      -h, --help            show this help message and exit
      -m MID, --mid MID     model id, usually <author>/<name>
      -t TASK, --task TASK  force the task to use
      -e EXPORT, --export EXPORT
                            export the model with this exporter
      --opt OPT             optimization to apply after the export
      -r, --run, --no-run   Runs the model to check it runs.
      -q, --quiet, --no-quiet
                            Catches exception, reports them in the summary.
      --patch [PATCH ...]   Applies patches before exporting, it can be a boolean to enable to disable the patches or be more finetuned. It is possible to disable patch for torch by adding --patch "patch_sympy=False" --patch "patch_torch=False", default is True.
      --rewrite, --no-rewrite
                            Applies rewrite before exporting.
      --stop-if-static STOP_IF_STATIC
                            Raises an exception if a dynamic dimension becomes static.
      --same-as-trained, --no-same-as-trained
                            Validates or exports a model identical to the trained model but not trained.
      --trained, --no-trained
                            Validates or exports the trained model (requires downloading).
      --inputs2 INPUTS2     Validates or exports the model on a second set of inputs
                            to check the exported model supports dynamism. The values is used as an increment to the first set of inputs. A high value may trick a different behavior in the model and missed by the exporter.
      --runtime {onnxruntime,torch,ref,orteval,orteval10}
                            onnx runtime to use, `onnxruntime` by default
      -o DUMP_FOLDER, --dump-folder DUMP_FOLDER
                            A folder is created to dumps statistics,
                            exported program, onnx...
      --drop DROP           Drops the following inputs names, it should be a list
                            with comma separated values, example:
                            --drop position_ids
      --opset OPSET         onnx opset to use, 18 by default
      --subfolder SUBFOLDER
                            Subfolder where to find the model and the configuration.
      --ortfusiontype ORTFUSIONTYPE
                            Applies onnxruntime fusion, this parameter should contain the
                            model type or multiple values separated by `|`. `ALL` can be used
                            to run them all.
      -v VERBOSE, --verbose VERBOSE
                            verbosity
      --dtype DTYPE         Changes dtype if necessary.
      --device DEVICE       Changes the device if necessary.
      --iop [KEY=VALUE ...]
                            Additional input options, use to change the defaultinputs use to export, example:
                              --iop cls_cache=SlidingWindowCache
                              --iop cls_cache=StaticCache
      --mop [KEY=VALUE ...]
                            Additional model options, use to change some parameters of the model, example:
                              --mop attn_implementation=sdpa --mop attn_implementation=eager
                              --mop "rope_scaling={'rope_type': 'dynamic', 'factor': 10.0}"
      --repeat REPEAT       number of times to run the model to measures inference time
      --warmup WARMUP       number of times to run the model to do warmup
      --outnames OUTNAMES   This comma separated list defines the output names the onnx exporter should use.
      --ort-logs, --no-ort-logs
                            Enables onnxruntime logging when the session is created
      --quiet-input-sets QUIET_INPUT_SETS
                            Avoids raising an exception when an input sets does not work with the exported model.
                            Example: --quiet-input-sets=inputs,inputs22
    
    If the model id is specified, one untrained version of it is instantiated.
    Examples:
    
    python -m onnx_diagnostic validate -m microsoft/Phi-4-mini-reasoning \
        --run -v 1 -o dump_test --no-quiet --repeat 2 --warmup 2 \
        --dtype float16 --device cuda --patch --export onnx-dynamo --opt ir
    
    python -m onnx_diagnostic validate -m microsoft/Phi-4-mini-reasoning \
        --run -v 1 -o dump_test --no-quiet --repeat 2 --warmup 2 \
        --dtype float16 --device cuda --patch --export custom --opt default
    
    python -m onnx_diagnostic validate -m microsoft/Phi-4-mini-reasoning \
        --run -v 1 -o dump_test --no-quiet --repeat 2 --warmup 2 \
        --dtype float16 --device cuda --export modelbuilder
    
    position_ids is usually not needed, they can be removed by adding:
    
        --drop position_ids
    
    The behaviour may be modified compare the original configuration,
    the following argument can be rope_scaling to dynamic:
    
        --mop "rope_scaling={'rope_type': 'dynamic', 'factor': 10.0}""
    
    You can profile the command line by running:
    
        pyinstrument -m onnx_diagnostic validate ...
        pyinstrument -r html -o profile.html -m onnx_diagnostic validate ...

Get the list of supported tasks

The task are the same defined by HuggingFace. The tool only supports a subset of them.

python -m onnx_diagnostic validate

    -- list of supported tasks:
    MoE
    automatic-speech-recognition
    feature-extraction
    fill-mask
    image-classification
    image-text-to-text
    image-to-video
    mask-generation
    object-detection
    sentence-similarity
    summarization
    text-classification
    text-generation
    text-to-image
    text2text-generation
    zero-shot-image-classification

Get the default inputs for a specific task

This returns the dummy inputs for a specific task. There may be too many inputs. Only those the forward method defines are kept.

python -m onnx_diagnostic validate -t text-generation

    -- inputs
      + input_ids       : T7s2x3
      + attention_mask  : T7s2x33
      + position_ids    : T7s2x3
      + past_key_values : DynamicCache(key_cache=#4[T1s2x24x30x16,T1s2x24x30x16,T1s2x24x30x16,T1s2x24x30x16], value_cache=#4[T1s2x24x30x16,T1s2x24x30x16,T1s2x24x30x16,T1s2x24x30x16])
    -- dynamic_shapes
      + input_ids       : {0:DYN(batch),1:DYN(seq_length)}
      + attention_mask  : {0:DYN(batch),1:DYN(cache+seq)}
      + position_ids    : {0:DYN(batch),1:DYN(seq_length)}
      + past_key_values : #2[#4[{0:DYN(batch),2:DYN(cache_length)},{0:DYN(batch),2:DYN(cache_length)},{0:DYN(batch),2:DYN(cache_length)},{0:DYN(batch),2:DYN(cache_length)}],#4[{0:DYN(batch),2:DYN(cache_length)},{0:DYN(batch),2:DYN(cache_length)},{0:DYN(batch),2:DYN(cache_length)},{0:DYN(batch),2:DYN(cache_length)}]]

Validate dummy inputs for a model

The dummy inputs may not work for this model and this task. The following command line checks that. It is no use to export if this fails.

python -m onnx_diagnostic validate -m arnir0/Tiny-LLM --run -v 1

    [validate_model] validate model id 'arnir0/Tiny-LLM'
    [validate_model] patch={'patch': True}
    [validate_model] get dummy inputs with input_options=None...
    [validate_model] rewrite=True, patch_kwargs={'patch': True, 'patch_transformers': True, 'patch_diffusers': True}, stop_if_static=0
    [validate_model] exporter=None, optimization=None
    [validate_model] dump_folder=None
    [validate_model] output_names=None
    [get_untrained_model_with_inputs] model_id='arnir0/Tiny-LLM', subfolder=None
    [get_untrained_model_with_inputs] use preinstalled 'arnir0/Tiny-LLM'
    [get_untrained_model_with_inputs] architecture='LlamaForCausalLM'
    [get_untrained_model_with_inputs] cls='LlamaConfig'
    [get_untrained_model_with_inputs] task='text-generation'
    [get_untrained_model_with_inputs] default config._attn_implementation=None
    [get_untrained_model_with_inputs] package_source=transformers from ~/github/transformers/src/transformers/__init__.py
    [get_untrained_model_with_inputs] instantiate model_id 'arnir0/Tiny-LLM', subfolder=None
    [get_untrained_model_with_inputs] -- done(2) in 4.659999831346795e-06s
    [get_untrained_model_with_inputs] instantiate_specific_model <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>
    [get_untrained_model_with_inputs] -- done(3) in 8.278000223072013e-06s (model is <class 'NoneType'>)
    [get_untrained_model_with_inputs] instantiate_specific_model(2) <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>
    [get_untrained_model_with_inputs] -- done(4) in 0.09629192600004899s (model is <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>)
    [get_untrained_model_with_inputs] use fct=<function get_inputs at 0x7ba139297240>
    [validate_model] --
    [validate_model] task=text-generation
    [validate_model] size=49.549072265625 Mb
    [validate_model] n_weights=12.988992 millions parameters
    [validate_model] +INPUT input_ids=T7s2x3
    [validate_model] +INPUT attention_mask=T7s2x33
    [validate_model] +INPUT position_ids=T7s2x3
    [validate_model] +INPUT past_key_values=DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96])
    [validate_model] +SHAPE input_ids={0:DYN(batch),1:DYN(seq_length)}
    [validate_model] +SHAPE attention_mask={0:DYN(batch),1:DYN(cache+seq)}
    [validate_model] +SHAPE position_ids={0:DYN(batch),1:DYN(seq_length)}
    [validate_model] +SHAPE past_key_values=#2[#1[{0:DYN(batch),2:DYN(cache_length)}],#1[{0:DYN(batch),2:DYN(cache_length)}]]
    [validate_model] second_input_keys=['inputs2', 'inputs_empty_cache', 'inputs_batch1']
    [validate_model] --
    [validate_model] -- run the model inputs='inputs'...
    [validate_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_model] done ([run])
    [validate_model] -- run the model inputs='inputs2'...
    [validate_model] inputs2=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_model] done ([run22])
    [validate_model] -- run the model inputs='inputs_empty_cache'...
    [validate_model] inputs_empty_cache=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_model] done ([run2_empty_cache])
    [validate_model] -- run the model inputs='inputs_batch1'...
    [validate_model] inputs_batch1=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_model] done ([run2_batch1])
    [validate_model] -- done (final)
    
    -- summary --
    :model_class,LlamaForCausalLM;
    :model_config,{'vocab_size':32000,'max_position_embeddings':1024,'hidden_size':192,'intermediate_size':1024,'num_hidden_layers':1,'num_attention_heads':2,'num_key_value_heads':1,'hidden_act':'silu','initializer_range':0.02,'rms_norm_eps':1e-05,'pretraining_tp':1,'use_cache':True,'attention_bias':False,'attention_dropout':0.0,'mlp_bias':False,'head_dim':96,'rope_parameters':{'rope_type':'default','rope_theta':10000.0},'return_dict':True,'output_hidden_states':False,'dtype':'float32','tie_word_embeddings':False,'chunk_size_feed_forward':0,'is_encoder_decoder':False,'is_decoder':False,'cross_attention_hidden_size':None,'add_cross_attention':False,'tie_encoder_decoder':False,'architectures':['LlamaForCausalLM'],'finetuning_task':None,'id2label':{0:'LABEL_0',1:'LABEL_1'},'label2id':{'LABEL_0':0,'LABEL_1':1},'task_specific_params':None,'problem_type':None,'tokenizer_class':None,'prefix':None,'bos_token_id':1,'pad_token_id':None,'eos_token_id':2,'sep_token_id':None,'decoder_start_token_id':None,'max_length':20,'min_length':0,'do_sample':False,'early_stopping':False,'num_beams':1,'temperature':1.0,'top_k':50,'top_p':1.0,'typical_p':1.0,'repetition_penalty':1.0,'length_penalty':1.0,'no_repeat_ngram_size':0,'encoder_no_repeat_ngram_size':0,'bad_words_ids':None,'num_return_sequences':1,'output_scores':False,'return_dict_in_generate':False,'forced_bos_token_id':None,'forced_eos_token_id':None,'remove_invalid_values':False,'exponential_decay_length_penalty':None,'suppress_tokens':None,'begin_suppress_tokens':None,'num_beam_groups':1,'diversity_penalty':0.0,'_name_or_path':'','transformers_version':'5.0.0.dev0','model_type':'llama','rope_theta':10000.0,'subfolder':None,'output_attentions':False};
    :model_config_class,LlamaConfig;
    :model_file,~/github/transformers/src/transformers/models/llama/modeling_llama.py;
    :model_id,arnir0/Tiny-LLM;
    :model_inputs,dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]));
    :model_inputs_options,;
    :model_module,transformers.models.llama.modeling_llama;
    :model_nweights,12988992;
    :model_shapes,dict(input_ids:{0:DYN(batch),1:DYN(seq_length)},attention_mask:{0:DYN(batch),1:DYN(cache+seq)},position_ids:{0:DYN(batch),1:DYN(seq_length)},past_key_values:#2[#1[{0:DYN(batch),2:DYN(cache_length)}],#1[{0:DYN(batch),2:DYN(cache_length)}]]);
    :model_size,51955968;
    :model_subfolder,;
    :model_task,text-generation;
    :run_expected,CausalLMOutputWithPast(logits:T1s2x3x32000,past_key_values:DynamicCache(key_cache=#1[T1s2x1x33x96], value_cache=#1[T1s2x1x33x96]));
    :run_expected22,CausalLMOutputWithPast(logits:T1s3x4x32000,past_key_values:DynamicCache(key_cache=#1[T1s3x1x35x96], value_cache=#1[T1s3x1x35x96]));
    :run_expected2_batch1,CausalLMOutputWithPast(logits:T1s1x3x32000,past_key_values:DynamicCache(key_cache=#1[T1s1x1x33x96], value_cache=#1[T1s1x1x33x96]));
    :run_expected2_empty_cache,CausalLMOutputWithPast(logits:T1s2x3x32000,past_key_values:DynamicCache(key_cache=#1[T1s2x1x3x96], value_cache=#1[T1s2x1x3x96]));
    :second_input_keys,inputs2,inputs_empty_cache,inputs_batch1;
    :time_create_torch_model,0.0999675890002436;
    :time_preprocess_model_id,4.300000000512227e-06;
    :time_run,0.011538773000211222;
    :time_run22,0.007581473999834998;
    :time_run2_batch1,0.005297539999901346;
    :time_run2_empty_cache,0.002596656999685365;
    :time_total_validation_torch,0.031205453999973543;
    :version_date,2025-10-24T18:03:23;
    :version_device,;
    :version_do_run,True;
    :version_drop_inputs,[];
    :version_dtype,;
    :version_dump_folder,;
    :version_exporter,;
    :version_inputs2,1;
    :version_model_id,arnir0/Tiny-LLM;
    :version_numpy,2.3.4;
    :version_onnx,1.20.0;
    :version_onnx_diagnostic,0.7.16;
    :version_onnx_ir,0.1.12;
    :version_onnxruntime,1.24.0;
    :version_onnxscript,?;
    :version_opset,18;
    :version_optimization,;
    :version_ortfusiontype,;
    :version_patch,{'patch': True};
    :version_patch_kwargs,{'patch':True,'patch_transformers':True,'patch_diffusers':True};
    :version_quiet,False;
    :version_rewrite,True;
    :version_runtime,onnxruntime;
    :version_same_as_pretrained,False;
    :version_scipy,1.16.2;
    :version_stop_if_static,0;
    :version_torch,2.10.0.dev20251022+cu130;
    :version_transformers,5.0.0.dev0;
    :version_use_pretrained,False;

Validate and export a model

Exports a model given the task. Checks for discrepancies as well. The latency given are just for one run. It tells how long the benchmark runs but it is far from the latency measure we can get by running multiple times the same model.

python -m onnx_diagnostic validate -m arnir0/Tiny-LLM --run -v 1 --export export-nostrict -o dump_models --patch

    [validate_model] dump into 'arnir0_Tiny-LLM/export-nostrict/op18'
    [validate_model] validate model id 'arnir0/Tiny-LLM'
    [validate_model] patch={'patch': True}
    [validate_model] get dummy inputs with input_options=None...
    [validate_model] rewrite=True, patch_kwargs={'patch': True, 'patch_transformers': True, 'patch_diffusers': True}, stop_if_static=0
    [validate_model] exporter='export-nostrict', optimization=None
    [validate_model] dump_folder='dump_models/arnir0_Tiny-LLM/export-nostrict/op18'
    [validate_model] output_names=None
    [get_untrained_model_with_inputs] model_id='arnir0/Tiny-LLM', subfolder=None
    [get_untrained_model_with_inputs] use preinstalled 'arnir0/Tiny-LLM'
    [get_untrained_model_with_inputs] architecture='LlamaForCausalLM'
    [get_untrained_model_with_inputs] cls='LlamaConfig'
    [get_untrained_model_with_inputs] task='text-generation'
    [get_untrained_model_with_inputs] default config._attn_implementation=None
    [get_untrained_model_with_inputs] package_source=transformers from ~/github/transformers/src/transformers/__init__.py
    [get_untrained_model_with_inputs] instantiate model_id 'arnir0/Tiny-LLM', subfolder=None
    [get_untrained_model_with_inputs] -- done(2) in 4.520999937085435e-06s
    [get_untrained_model_with_inputs] instantiate_specific_model <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>
    [get_untrained_model_with_inputs] -- done(3) in 6.292000307439594e-06s (model is <class 'NoneType'>)
    [get_untrained_model_with_inputs] instantiate_specific_model(2) <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>
    [get_untrained_model_with_inputs] -- done(4) in 0.08515124800032936s (model is <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>)
    [get_untrained_model_with_inputs] use fct=<function get_inputs at 0x7ba139297240>
    [validate_model] --
    [validate_model] task=text-generation
    [validate_model] size=49.549072265625 Mb
    [validate_model] n_weights=12.988992 millions parameters
    [validate_model] +INPUT input_ids=T7s2x3
    [validate_model] +INPUT attention_mask=T7s2x33
    [validate_model] +INPUT position_ids=T7s2x3
    [validate_model] +INPUT past_key_values=DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96])
    [validate_model] +SHAPE input_ids={0:DYN(batch),1:DYN(seq_length)}
    [validate_model] +SHAPE attention_mask={0:DYN(batch),1:DYN(cache+seq)}
    [validate_model] +SHAPE position_ids={0:DYN(batch),1:DYN(seq_length)}
    [validate_model] +SHAPE past_key_values=#2[#1[{0:DYN(batch),2:DYN(cache_length)}],#1[{0:DYN(batch),2:DYN(cache_length)}]]
    [validate_model] second_input_keys=['inputs2', 'inputs_empty_cache', 'inputs_batch1']
    [validate_model] --
    [validate_model] -- run the model inputs='inputs'...
    [validate_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_model] done ([run])
    [validate_model] -- run the model inputs='inputs2'...
    [validate_model] inputs2=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_model] done ([run22])
    [validate_model] -- run the model inputs='inputs_empty_cache'...
    [validate_model] inputs_empty_cache=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_model] done ([run2_empty_cache])
    [validate_model] -- run the model inputs='inputs_batch1'...
    [validate_model] inputs_batch1=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_model] done ([run2_batch1])
    [validate_model] -- export the model with 'export-nostrict', optimization=None
    [validate_model] applies patches before exporting stop_if_static=0
    [validate_model] run patched model...
    [validate_model] patched inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_model] done (patched run)
    [validate_model] patched discrepancies=abs=0, rel=0
    [call_torch_export_export] exporter='export-nostrict', strict=False, optimization=None
    [call_torch_export_export] args=()
    [call_torch_export_export] kwargs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [call_torch_export_export] dynamic_shapes=dict(input_ids:{0:DYN(batch),1:DYN(seq_length)},attention_mask:{0:DYN(batch),1:DYN(cache+seq)},position_ids:{0:DYN(batch),1:DYN(seq_length)},past_key_values:#2[#1[{0:DYN(batch),2:DYN(cache_length)}],#1[{0:DYN(batch),2:DYN(cache_length)}]])
    [call_torch_export_export] dynamic_shapes_export_export=dict(input_ids:{0:DYNAMIC,1:DYNAMIC},attention_mask:{0:DYNAMIC,1:DYNAMIC},position_ids:{0:DYNAMIC,1:DYNAMIC},past_key_values:#2[#1[{0:DYNAMIC,2:DYNAMIC}],#1[{0:DYNAMIC,2:DYNAMIC}]])
    [call_torch_export_export] export...
    [call_torch_export_export] done (export) with 152 nodes
    [validate_model] run exported model...
    [validate_model] patched inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_model] done (exported run)
    [validate_model] exported discrepancies=abs=0, rel=0
    [validate_model] -- dumps exported program in 'dump_models/arnir0_Tiny-LLM/export-nostrict/op18'...
    [validate_model] done (dump ep)
    [validate_model] dumps statistics in 'dump_models/arnir0_Tiny-LLM/export-nostrict/op18'...
    [validate_model] done (dump)
    [validate_model] -- done (final)
    
    -- summary --
    :disc_exported_abs,0;
    :disc_exported_dnan,0;
    :disc_exported_n,204672.0;
    :disc_exported_rel,0;
    :disc_exported_sum,0.0;
    :disc_patched_abs,0;
    :disc_patched_dnan,0;
    :disc_patched_n,204672.0;
    :disc_patched_rel,0;
    :disc_patched_sum,0.0;
    :dump_folder,dump_models/arnir0_Tiny-LLM/export-nostrict/op18;
    :dump_folder_name,arnir0_Tiny-LLM/export-nostrict/op18;
    :export_args,();
    :export_dynamic_shapes,dict(input_ids:{0:DYN(batch),1:DYN(seq_length)},attention_mask:{0:DYN(batch),1:DYN(cache+seq)},position_ids:{0:DYN(batch),1:DYN(seq_length)},past_key_values:#2[#1[{0:DYN(batch),2:DYN(cache_length)}],#1[{0:DYN(batch),2:DYN(cache_length)}]]);
    :export_dynamic_shapes_export_export,dict(input_ids:{0:DYNAMIC,1:DYNAMIC},attention_mask:{0:DYNAMIC,1:DYNAMIC},position_ids:{0:DYNAMIC,1:DYNAMIC},past_key_values:#2[#1[{0:DYNAMIC,2:DYNAMIC}],#1[{0:DYNAMIC,2:DYNAMIC}]]);
    :export_exporter,export-nostrict;
    :export_graph_nodes,152;
    :export_kwargs,dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]));
    :export_optimization,;
    :export_strict,False;
    :model_class,LlamaForCausalLM;
    :model_config,{'vocab_size':32000,'max_position_embeddings':1024,'hidden_size':192,'intermediate_size':1024,'num_hidden_layers':1,'num_attention_heads':2,'num_key_value_heads':1,'hidden_act':'silu','initializer_range':0.02,'rms_norm_eps':1e-05,'pretraining_tp':1,'use_cache':True,'attention_bias':False,'attention_dropout':0.0,'mlp_bias':False,'head_dim':96,'rope_parameters':{'rope_type':'default','rope_theta':10000.0},'return_dict':True,'output_hidden_states':False,'dtype':'float32','tie_word_embeddings':False,'chunk_size_feed_forward':0,'is_encoder_decoder':False,'is_decoder':False,'cross_attention_hidden_size':None,'add_cross_attention':False,'tie_encoder_decoder':False,'architectures':['LlamaForCausalLM'],'finetuning_task':None,'id2label':{0:'LABEL_0',1:'LABEL_1'},'label2id':{'LABEL_0':0,'LABEL_1':1},'task_specific_params':None,'problem_type':None,'tokenizer_class':None,'prefix':None,'bos_token_id':1,'pad_token_id':None,'eos_token_id':2,'sep_token_id':None,'decoder_start_token_id':None,'max_length':20,'min_length':0,'do_sample':False,'early_stopping':False,'num_beams':1,'temperature':1.0,'top_k':50,'top_p':1.0,'typical_p':1.0,'repetition_penalty':1.0,'length_penalty':1.0,'no_repeat_ngram_size':0,'encoder_no_repeat_ngram_size':0,'bad_words_ids':None,'num_return_sequences':1,'output_scores':False,'return_dict_in_generate':False,'forced_bos_token_id':None,'forced_eos_token_id':None,'remove_invalid_values':False,'exponential_decay_length_penalty':None,'suppress_tokens':None,'begin_suppress_tokens':None,'num_beam_groups':1,'diversity_penalty':0.0,'_name_or_path':'','transformers_version':'5.0.0.dev0','model_type':'llama','rope_theta':10000.0,'subfolder':None,'output_attentions':False};
    :model_config_class,LlamaConfig;
    :model_file,~/github/transformers/src/transformers/models/llama/modeling_llama.py;
    :model_id,arnir0/Tiny-LLM;
    :model_inputs,dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]));
    :model_inputs_options,;
    :model_module,transformers.models.llama.modeling_llama;
    :model_nweights,12988992;
    :model_shapes,dict(input_ids:{0:DYN(batch),1:DYN(seq_length)},attention_mask:{0:DYN(batch),1:DYN(cache+seq)},position_ids:{0:DYN(batch),1:DYN(seq_length)},past_key_values:#2[#1[{0:DYN(batch),2:DYN(cache_length)}],#1[{0:DYN(batch),2:DYN(cache_length)}]]);
    :model_size,51955968;
    :model_subfolder,;
    :model_task,text-generation;
    :run_expected,CausalLMOutputWithPast(logits:T1s2x3x32000,past_key_values:DynamicCache(key_cache=#1[T1s2x1x33x96], value_cache=#1[T1s2x1x33x96]));
    :run_expected22,CausalLMOutputWithPast(logits:T1s3x4x32000,past_key_values:DynamicCache(key_cache=#1[T1s3x1x35x96], value_cache=#1[T1s3x1x35x96]));
    :run_expected2_batch1,CausalLMOutputWithPast(logits:T1s1x3x32000,past_key_values:DynamicCache(key_cache=#1[T1s1x1x33x96], value_cache=#1[T1s1x1x33x96]));
    :run_expected2_empty_cache,CausalLMOutputWithPast(logits:T1s2x3x32000,past_key_values:DynamicCache(key_cache=#1[T1s2x1x3x96], value_cache=#1[T1s2x1x3x96]));
    :second_input_keys,inputs2,inputs_empty_cache,inputs_batch1;
    :time_create_torch_model,0.08846088299969779;
    :time_export_export,1.4313744240002961;
    :time_preprocess_model_id,3.949000074499054e-06;
    :time_run,0.005591874999936408;
    :time_run22,0.004505031000007875;
    :time_run2_batch1,0.006185855999774503;
    :time_run2_empty_cache,0.0019644130002234306;
    :time_run_exported,0.00648396499991577;
    :time_run_patched,0.0021443519999593263;
    :time_torch_export_export,1.4313670299998194;
    :time_torch_export_export_n,1;
    :time_total_exporter,1.4945404310001322;
    :time_total_validation_torch,0.020668378999744164;
    :version_date,2025-10-24T18:03:23;
    :version_device,;
    :version_do_run,True;
    :version_drop_inputs,[];
    :version_dtype,;
    :version_dump_folder,dump_models;
    :version_exporter,export-nostrict;
    :version_inputs2,1;
    :version_model_id,arnir0/Tiny-LLM;
    :version_numpy,2.3.4;
    :version_onnx,1.20.0;
    :version_onnx_diagnostic,0.7.16;
    :version_onnx_ir,0.1.12;
    :version_onnxruntime,1.24.0;
    :version_onnxscript,?;
    :version_opset,18;
    :version_optimization,;
    :version_ortfusiontype,;
    :version_patch,{'patch': True};
    :version_patch_kwargs,{'patch':True,'patch_transformers':True,'patch_diffusers':True};
    :version_quiet,False;
    :version_rewrite,True;
    :version_runtime,onnxruntime;
    :version_same_as_pretrained,False;
    :version_scipy,1.16.2;
    :version_stop_if_static,0;
    :version_torch,2.10.0.dev20251022+cu130;
    :version_transformers,5.0.0.dev0;
    :version_use_pretrained,False;

Validate ONNX discrepancies

Let’s export with ONNX this time and checks for discrepancies.

python -m onnx_diagnostic validate -m arnir0/Tiny-LLM --run -v 1 --export onnx-dynamo -o dump_models --patch --opt ir

    [validate_model] dump into 'arnir0_Tiny-LLM/onnx-dynamo/ir/op18'
    [validate_model] validate model id 'arnir0/Tiny-LLM'
    [validate_model] patch={'patch': True}
    [validate_model] get dummy inputs with input_options=None...
    [validate_model] rewrite=True, patch_kwargs={'patch': True, 'patch_transformers': True, 'patch_diffusers': True}, stop_if_static=0
    [validate_model] exporter='onnx-dynamo', optimization='ir'
    [validate_model] dump_folder='dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18'
    [validate_model] output_names=None
    [get_untrained_model_with_inputs] model_id='arnir0/Tiny-LLM', subfolder=None
    [get_untrained_model_with_inputs] use preinstalled 'arnir0/Tiny-LLM'
    [get_untrained_model_with_inputs] architecture='LlamaForCausalLM'
    [get_untrained_model_with_inputs] cls='LlamaConfig'
    [get_untrained_model_with_inputs] task='text-generation'
    [get_untrained_model_with_inputs] default config._attn_implementation=None
    [get_untrained_model_with_inputs] package_source=transformers from ~/github/transformers/src/transformers/__init__.py
    [get_untrained_model_with_inputs] instantiate model_id 'arnir0/Tiny-LLM', subfolder=None
    [get_untrained_model_with_inputs] -- done(2) in 1.8522000118537107e-05s
    [get_untrained_model_with_inputs] instantiate_specific_model <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>
    [get_untrained_model_with_inputs] -- done(3) in 7.603000085509848e-06s (model is <class 'NoneType'>)
    [get_untrained_model_with_inputs] instantiate_specific_model(2) <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>
    [get_untrained_model_with_inputs] -- done(4) in 0.11243973299997378s (model is <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>)
    [get_untrained_model_with_inputs] use fct=<function get_inputs at 0x75267a899d00>
    [validate_model] --
    [validate_model] task=text-generation
    [validate_model] size=49.549072265625 Mb
    [validate_model] n_weights=12.988992 millions parameters
    [validate_model] +INPUT input_ids=T7s2x3
    [validate_model] +INPUT attention_mask=T7s2x33
    [validate_model] +INPUT position_ids=T7s2x3
    [validate_model] +INPUT past_key_values=DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96])
    [validate_model] +SHAPE input_ids={0:DYN(batch),1:DYN(seq_length)}
    [validate_model] +SHAPE attention_mask={0:DYN(batch),1:DYN(cache+seq)}
    [validate_model] +SHAPE position_ids={0:DYN(batch),1:DYN(seq_length)}
    [validate_model] +SHAPE past_key_values=#2[#1[{0:DYN(batch),2:DYN(cache_length)}],#1[{0:DYN(batch),2:DYN(cache_length)}]]
    [validate_model] second_input_keys=['inputs2', 'inputs_empty_cache', 'inputs_batch1']
    [validate_model] --
    [validate_model] -- run the model inputs='inputs'...
    [validate_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_model] done ([run])
    [validate_model] -- run the model inputs='inputs2'...
    [validate_model] inputs2=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_model] done ([run22])
    [validate_model] -- run the model inputs='inputs_empty_cache'...
    [validate_model] inputs_empty_cache=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_model] done ([run2_empty_cache])
    [validate_model] -- run the model inputs='inputs_batch1'...
    [validate_model] inputs_batch1=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_model] done ([run2_batch1])
    [validate_model] -- export the model with 'onnx-dynamo', optimization='ir'
    [validate_model] applies patches before exporting stop_if_static=0
    [validate_model] run patched model...
    [validate_model] patched inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_model] done (patched run)
    [validate_model] patched discrepancies=abs=0, rel=0
    [call_torch_export_onnx] exporter='onnx-dynamo', optimization='ir'
    [call_torch_export_onnx] args=()
    [call_torch_export_onnx] kwargs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [call_torch_export_onnx] dynamic_shapes=dict(input_ids:{0:DYN(batch),1:DYN(seq_length)},attention_mask:{0:DYN(batch),1:DYN(cache+seq)},position_ids:{0:DYN(batch),1:DYN(seq_length)},past_key_values:#2[#1[{0:DYN(batch),2:DYN(cache_length)}],#1[{0:DYN(batch),2:DYN(cache_length)}]])
    [call_torch_export_onnx] export...
    [call_torch_export_onnx] export_export_kwargs=dict(dynamo:bool,dynamic_shapes:dict(input_ids:{0:DYN(batch),1:DYN(seq_length)},attention_mask:{0:DYN(batch),1:DYN(cache+seq)},position_ids:{0:DYN(batch),1:DYN(seq_length)},past_key_values:#2[#1[{0:DYN(batch),2:DYN(cache_length)}],#1[{0:DYN(batch),2:DYN(cache_length)}]]),opset_version:int)
    [torch.onnx] Obtain model graph for `LlamaForCausalLM([...]` with `torch.export.export(..., strict=False)`...
    [torch.onnx] Obtain model graph for `LlamaForCausalLM([...]` with `torch.export.export(..., strict=False)`... ✅
    [torch.onnx] Run decomposition...
    [torch.onnx] Run decomposition... ✅
    [torch.onnx] Translate the graph into ONNX...
    [torch.onnx] Translate the graph into ONNX... ✅
    Applied 37 of general pattern rewrite rules.
    [call_torch_export_onnx] done (export)
    [call_torch_export_onnx] starts optimization='ir'...
    [call_torch_export_onnx] done (optimization)
    [validate_model] dumps onnx program in 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18'...
    [validate_model] done (dump onnx) in 0.18941847900032371
    [validate_model] dumps statistics in 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18'...
    [validate_model] done (dump)
    [validation_model] -- delete the model
    [validation_model] -- done
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour=None
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour=None
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=7.748603820800781e-07, rel=0.00044309172106863606, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00032628376058705446, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=6.556510925292969e-07, rel=0.0002520801563113858, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=7.748603820800781e-07, rel=0.00037692802454639585, n=102336.0
    [validate_model] -- done (final)
    
    -- summary --
    :disc_onnx_ort_run22_abs,8.344650268554688e-07;
    :disc_onnx_ort_run22_dnan,0;
    :disc_onnx_ort_run22_n,404160.0;
    :disc_onnx_ort_run22_rel,0.00032628376058705446;
    :disc_onnx_ort_run22_sum,0.039355116007072866;
    :disc_onnx_ort_run2_batch1_abs,7.748603820800781e-07;
    :disc_onnx_ort_run2_batch1_dnan,0;
    :disc_onnx_ort_run2_batch1_n,102336.0;
    :disc_onnx_ort_run2_batch1_rel,0.00037692802454639585;
    :disc_onnx_ort_run2_batch1_sum,0.011718470417235949;
    :disc_onnx_ort_run2_empty_cache_abs,6.556510925292969e-07;
    :disc_onnx_ort_run2_empty_cache_dnan,0;
    :disc_onnx_ort_run2_empty_cache_n,193152.0;
    :disc_onnx_ort_run2_empty_cache_rel,0.0002520801563113858;
    :disc_onnx_ort_run2_empty_cache_sum,0.012811366097139398;
    :disc_onnx_ort_run_abs,7.748603820800781e-07;
    :disc_onnx_ort_run_dnan,0;
    :disc_onnx_ort_run_n,204672.0;
    :disc_onnx_ort_run_rel,0.00044309172106863606;
    :disc_onnx_ort_run_sum,0.02031988672524676;
    :disc_patched_abs,0;
    :disc_patched_dnan,0;
    :disc_patched_n,204672.0;
    :disc_patched_rel,0;
    :disc_patched_sum,0.0;
    :dump_folder,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18;
    :dump_folder_name,arnir0_Tiny-LLM/onnx-dynamo/ir/op18;
    :export_args,();
    :export_dynamo,True;
    :export_exporter,onnx-dynamo;
    :export_kwargs,dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]));
    :export_opset,18;
    :export_optimization,ir;
    :model_class,LlamaForCausalLM;
    :model_config,{'vocab_size':32000,'max_position_embeddings':1024,'hidden_size':192,'intermediate_size':1024,'num_hidden_layers':1,'num_attention_heads':2,'num_key_value_heads':1,'hidden_act':'silu','initializer_range':0.02,'rms_norm_eps':1e-05,'pretraining_tp':1,'use_cache':True,'attention_bias':False,'attention_dropout':0.0,'mlp_bias':False,'head_dim':96,'rope_parameters':{'rope_type':'default','rope_theta':10000.0},'return_dict':True,'output_hidden_states':False,'dtype':'float32','tie_word_embeddings':False,'chunk_size_feed_forward':0,'is_encoder_decoder':False,'is_decoder':False,'cross_attention_hidden_size':None,'add_cross_attention':False,'tie_encoder_decoder':False,'architectures':['LlamaForCausalLM'],'finetuning_task':None,'id2label':{0:'LABEL_0',1:'LABEL_1'},'label2id':{'LABEL_0':0,'LABEL_1':1},'task_specific_params':None,'problem_type':None,'tokenizer_class':None,'prefix':None,'bos_token_id':1,'pad_token_id':None,'eos_token_id':2,'sep_token_id':None,'decoder_start_token_id':None,'max_length':20,'min_length':0,'do_sample':False,'early_stopping':False,'num_beams':1,'temperature':1.0,'top_k':50,'top_p':1.0,'typical_p':1.0,'repetition_penalty':1.0,'length_penalty':1.0,'no_repeat_ngram_size':0,'encoder_no_repeat_ngram_size':0,'bad_words_ids':None,'num_return_sequences':1,'output_scores':False,'return_dict_in_generate':False,'forced_bos_token_id':None,'forced_eos_token_id':None,'remove_invalid_values':False,'exponential_decay_length_penalty':None,'suppress_tokens':None,'begin_suppress_tokens':None,'num_beam_groups':1,'diversity_penalty':0.0,'_name_or_path':'','transformers_version':'5.0.0.dev0','model_type':'llama','rope_theta':10000.0,'subfolder':None,'output_attentions':False};
    :model_config_class,LlamaConfig;
    :model_file,~/github/transformers/src/transformers/models/llama/modeling_llama.py;
    :model_id,arnir0/Tiny-LLM;
    :model_inputs,dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]));
    :model_inputs_options,;
    :model_module,transformers.models.llama.modeling_llama;
    :model_nweights,12988992;
    :model_shapes,dict(input_ids:{0:DYN(batch),1:DYN(seq_length)},attention_mask:{0:DYN(batch),1:DYN(cache+seq)},position_ids:{0:DYN(batch),1:DYN(seq_length)},past_key_values:#2[#1[{0:DYN(batch),2:DYN(cache_length)}],#1[{0:DYN(batch),2:DYN(cache_length)}]]);
    :model_size,51955968;
    :model_subfolder,;
    :model_task,text-generation;
    :n_node_Add,10;
    :n_node_And,2;
    :n_node_Cast,2;
    :n_node_Concat,16;
    :n_node_Cos,1;
    :n_node_Expand,6;
    :n_node_Gather,1;
    :n_node_GatherND,1;
    :n_node_IsNaN,1;
    :n_node_LessOrEqual,1;
    :n_node_MatMul,11;
    :n_node_Mul,14;
    :n_node_Neg,2;
    :n_node_Pow,3;
    :n_node_Range,3;
    :n_node_Reciprocal,3;
    :n_node_ReduceMean,3;
    :n_node_Reshape,13;
    :n_node_Shape,5;
    :n_node_Sigmoid,1;
    :n_node_Sin,1;
    :n_node_Slice,7;
    :n_node_Softmax,1;
    :n_node_Sqrt,3;
    :n_node_Squeeze,4;
    :n_node_Transpose,6;
    :n_node_Unsqueeze,7;
    :n_node_Where,2;
    :n_node_functions,0;
    :n_node_initializer_1,16;
    :n_node_initializer_7,15;
    :n_node_initializer_9,1;
    :n_node_nodes,130;
    :n_node_nodes_nocst,130;
    :onnx_filename,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.onnx;
    :onnx_ort_inputs,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs22,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs2_batch1,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_empty_cache,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_size,200677;
    :run_expected,CausalLMOutputWithPast(logits:T1s2x3x32000,past_key_values:DynamicCache(key_cache=#1[T1s2x1x33x96], value_cache=#1[T1s2x1x33x96]));
    :run_expected22,CausalLMOutputWithPast(logits:T1s3x4x32000,past_key_values:DynamicCache(key_cache=#1[T1s3x1x35x96], value_cache=#1[T1s3x1x35x96]));
    :run_expected2_batch1,CausalLMOutputWithPast(logits:T1s1x3x32000,past_key_values:DynamicCache(key_cache=#1[T1s1x1x33x96], value_cache=#1[T1s1x1x33x96]));
    :run_expected2_empty_cache,CausalLMOutputWithPast(logits:T1s2x3x32000,past_key_values:DynamicCache(key_cache=#1[T1s2x1x3x96], value_cache=#1[T1s2x1x3x96]));
    :run_feeds_inputs,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :run_feeds_inputs2,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :run_feeds_inputs_batch1,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :run_feeds_inputs_empty_cache,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :run_output_inputs,#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96];
    :run_output_inputs2,#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96];
    :run_output_inputs_batch1,#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96];
    :run_output_inputs_empty_cache,#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96];
    :second_input_keys,inputs2,inputs_empty_cache,inputs_batch1;
    :time_create_onnx_ort,0.0420845339999687;
    :time_create_torch_model,0.162652282000181;
    :time_export_onnx,4.660656690999986;
    :time_export_onnx_opt_ir,0.04051933800019469;
    :time_onnx_save,0.18941847900032371;
    :time_preprocess_model_id,1.3029998626734596e-06;
    :time_run,0.017221509000137303;
    :time_run22,0.008912980000332027;
    :time_run2_batch1,0.00657328499983123;
    :time_run2_empty_cache,0.004719408999790176;
    :time_run_onnx_ort,0.012560924999888812;
    :time_run_onnx_ort22,0.002003637000143499;
    :time_run_onnx_ort2_batch1,0.001152597999862337;
    :time_run_onnx_ort2_empty_cache,0.0012609820000761829;
    :time_run_patched,0.003948347000005015;
    :time_torch_export_export,1.6286030530000062;
    :time_torch_export_export_n,1;
    :time_total,6.877492070999779;
    :time_total_exporter,6.037616209999669;
    :time_total_validation_onnx,0.10096310500011896;
    :time_total_validation_torch,0.042629391999980726;
    :version_date,2025-10-24T18:03:33;
    :version_device,;
    :version_do_run,True;
    :version_drop_inputs,[];
    :version_dtype,;
    :version_dump_folder,dump_models;
    :version_exporter,onnx-dynamo;
    :version_inputs2,1;
    :version_model_id,arnir0/Tiny-LLM;
    :version_numpy,2.3.4;
    :version_onnx,1.20.0;
    :version_onnx_diagnostic,0.7.16;
    :version_onnx_ir,0.1.12;
    :version_onnxruntime,1.24.0;
    :version_onnxscript,?;
    :version_opset,18;
    :version_optimization,ir;
    :version_ortfusiontype,;
    :version_patch,{'patch': True};
    :version_patch_kwargs,{'patch':True,'patch_transformers':True,'patch_diffusers':True};
    :version_quiet,False;
    :version_rewrite,True;
    :version_runtime,onnxruntime;
    :version_same_as_pretrained,False;
    :version_scipy,1.16.2;
    :version_stop_if_static,0;
    :version_torch,2.10.0.dev20251022+cu130;
    :version_transformers,5.0.0.dev0;
    :version_use_pretrained,False;
    [runpythonerror]
    W1024 18:03:36.003000 10717 torch/fx/experimental/symbolic_shapes.py:6918] _maybe_guard_rel() was called on non-relation expression Eq(s72, 1) | Eq(Max(s44, s72), s72)
    W1024 18:03:36.008000 10717 torch/fx/experimental/symbolic_shapes.py:6918] _maybe_guard_rel() was called on non-relation expression Eq(s70, 1) | Eq(Max(s70, s9), s70)
    W1024 18:03:36.034000 10717 torch/fx/experimental/symbolic_shapes.py:6918] _maybe_guard_rel() was called on non-relation expression Eq(s44, 1) | Eq(Max(s44, s72), s44)
    W1024 18:03:36.038000 10717 torch/fx/experimental/symbolic_shapes.py:6918] _maybe_guard_rel() was called on non-relation expression Eq(s9, 1) | Eq(Max(s70, s9), s9)
    ~/vv/this312/lib/python3.12/site-packages/torch/onnx/_internal/exporter/_dynamic_shapes.py:272: UserWarning: # The axis name: batch will not be used, since it shares the same shape constraints with another axis: batch.
      warnings.warn(
    ~/vv/this312/lib/python3.12/site-packages/torch/onnx/_internal/exporter/_dynamic_shapes.py:272: UserWarning: # The axis name: seq_length will not be used, since it shares the same shape constraints with another axis: seq_length.
      warnings.warn(

Run onnxruntime fusions

This option runs transformers optimizations implemented in onnxruntime. The list of supported model_type can be found in the documentation of function onnx_diagnostic.torch_models.validate.run_ort_fusion().

python -m onnx_diagnostic validate -m arnir0/Tiny-LLM --run -v 1 --export onnx-dynamo -o dump_models --patch --opt ir --ortfusiontype ALL

    [validate_model] dump into 'arnir0_Tiny-LLM/onnx-dynamo/ir/op18'
    [validate_model] validate model id 'arnir0/Tiny-LLM'
    [validate_model] patch={'patch': True}
    [validate_model] get dummy inputs with input_options=None...
    [validate_model] rewrite=True, patch_kwargs={'patch': True, 'patch_transformers': True, 'patch_diffusers': True}, stop_if_static=0
    [validate_model] exporter='onnx-dynamo', optimization='ir'
    [validate_model] dump_folder='dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18'
    [validate_model] output_names=None
    [get_untrained_model_with_inputs] model_id='arnir0/Tiny-LLM', subfolder=None
    [get_untrained_model_with_inputs] use preinstalled 'arnir0/Tiny-LLM'
    [get_untrained_model_with_inputs] architecture='LlamaForCausalLM'
    [get_untrained_model_with_inputs] cls='LlamaConfig'
    [get_untrained_model_with_inputs] task='text-generation'
    [get_untrained_model_with_inputs] default config._attn_implementation=None
    [get_untrained_model_with_inputs] package_source=transformers from ~/github/transformers/src/transformers/__init__.py
    [get_untrained_model_with_inputs] instantiate model_id 'arnir0/Tiny-LLM', subfolder=None
    [get_untrained_model_with_inputs] -- done(2) in 1.8183000065619126e-05s
    [get_untrained_model_with_inputs] instantiate_specific_model <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>
    [get_untrained_model_with_inputs] -- done(3) in 6.1679997997998726e-06s (model is <class 'NoneType'>)
    [get_untrained_model_with_inputs] instantiate_specific_model(2) <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>
    [get_untrained_model_with_inputs] -- done(4) in 0.1184320669999579s (model is <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>)
    [get_untrained_model_with_inputs] use fct=<function get_inputs at 0x73511829de40>
    [validate_model] --
    [validate_model] task=text-generation
    [validate_model] size=49.549072265625 Mb
    [validate_model] n_weights=12.988992 millions parameters
    [validate_model] +INPUT input_ids=T7s2x3
    [validate_model] +INPUT attention_mask=T7s2x33
    [validate_model] +INPUT position_ids=T7s2x3
    [validate_model] +INPUT past_key_values=DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96])
    [validate_model] +SHAPE input_ids={0:DYN(batch),1:DYN(seq_length)}
    [validate_model] +SHAPE attention_mask={0:DYN(batch),1:DYN(cache+seq)}
    [validate_model] +SHAPE position_ids={0:DYN(batch),1:DYN(seq_length)}
    [validate_model] +SHAPE past_key_values=#2[#1[{0:DYN(batch),2:DYN(cache_length)}],#1[{0:DYN(batch),2:DYN(cache_length)}]]
    [validate_model] second_input_keys=['inputs2', 'inputs_empty_cache', 'inputs_batch1']
    [validate_model] --
    [validate_model] -- run the model inputs='inputs'...
    [validate_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_model] done ([run])
    [validate_model] -- run the model inputs='inputs2'...
    [validate_model] inputs2=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_model] done ([run22])
    [validate_model] -- run the model inputs='inputs_empty_cache'...
    [validate_model] inputs_empty_cache=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_model] done ([run2_empty_cache])
    [validate_model] -- run the model inputs='inputs_batch1'...
    [validate_model] inputs_batch1=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_model] done ([run2_batch1])
    [validate_model] -- export the model with 'onnx-dynamo', optimization='ir'
    [validate_model] applies patches before exporting stop_if_static=0
    [validate_model] run patched model...
    [validate_model] patched inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_model] done (patched run)
    [validate_model] patched discrepancies=abs=0, rel=0
    [call_torch_export_onnx] exporter='onnx-dynamo', optimization='ir'
    [call_torch_export_onnx] args=()
    [call_torch_export_onnx] kwargs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [call_torch_export_onnx] dynamic_shapes=dict(input_ids:{0:DYN(batch),1:DYN(seq_length)},attention_mask:{0:DYN(batch),1:DYN(cache+seq)},position_ids:{0:DYN(batch),1:DYN(seq_length)},past_key_values:#2[#1[{0:DYN(batch),2:DYN(cache_length)}],#1[{0:DYN(batch),2:DYN(cache_length)}]])
    [call_torch_export_onnx] export...
    [call_torch_export_onnx] export_export_kwargs=dict(dynamo:bool,dynamic_shapes:dict(input_ids:{0:DYN(batch),1:DYN(seq_length)},attention_mask:{0:DYN(batch),1:DYN(cache+seq)},position_ids:{0:DYN(batch),1:DYN(seq_length)},past_key_values:#2[#1[{0:DYN(batch),2:DYN(cache_length)}],#1[{0:DYN(batch),2:DYN(cache_length)}]]),opset_version:int)
    [torch.onnx] Obtain model graph for `LlamaForCausalLM([...]` with `torch.export.export(..., strict=False)`...
    [torch.onnx] Obtain model graph for `LlamaForCausalLM([...]` with `torch.export.export(..., strict=False)`... ✅
    [torch.onnx] Run decomposition...
    [torch.onnx] Run decomposition... ✅
    [torch.onnx] Translate the graph into ONNX...
    [torch.onnx] Translate the graph into ONNX... ✅
    Applied 37 of general pattern rewrite rules.
    [call_torch_export_onnx] done (export)
    [call_torch_export_onnx] starts optimization='ir'...
    [call_torch_export_onnx] done (optimization)
    [validate_model] dumps onnx program in 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18'...
    [validate_model] done (dump onnx) in 0.19449118099964835
    [validate_model] dumps statistics in 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18'...
    [validate_model] done (dump)
    [validation_model] -- delete the model
    [validation_model] -- done
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour=None
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour=None
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=7.748603820800781e-07, rel=0.00044309172106863606, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00032628376058705446, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=6.556510925292969e-07, rel=0.0002520801563113858, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=7.748603820800781e-07, rel=0.00037692802454639585, n=102336.0
    [validate_model] run onnxruntime fusion for 'bart'
    failed in shape inference <class 'AssertionError'>
    [validate_model] done 'bart' in 0.1720465699995657, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.bart.onnx'
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour='ortbart'
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour='ortbart'
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00038373230338287646, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=1.0132789611816406e-06, rel=0.00026781923006472165, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=1.0728836059570312e-06, rel=0.00034613720184126256, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=1.1026859283447266e-06, rel=0.0003551108443373418, n=102336.0
    [validate_model] run onnxruntime fusion for 'bert'
    failed in shape inference <class 'AssertionError'>
    [validate_model] done 'bert' in 0.16673424700002215, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.bert.onnx'
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour='ortbert'
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour='ortbert'
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00038373230338287646, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=1.0132789611816406e-06, rel=0.00026781923006472165, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=1.0728836059570312e-06, rel=0.00034613720184126256, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=1.1026859283447266e-06, rel=0.0003551108443373418, n=102336.0
    [validate_model] run onnxruntime fusion for 'bert_keras'
    failed in shape inference <class 'AssertionError'>
    [validate_model] done 'bert_keras' in 0.2265212470001643, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.bert_keras.onnx'
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour='ortbert_keras'
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour='ortbert_keras'
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00038373230338287646, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=1.0132789611816406e-06, rel=0.00026781923006472165, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=1.0728836059570312e-06, rel=0.00034613720184126256, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=1.1026859283447266e-06, rel=0.0003551108443373418, n=102336.0
    [validate_model] run onnxruntime fusion for 'bert_tf'
    failed in shape inference <class 'AssertionError'>
    [validate_model] done 'bert_tf' in 0.10279864999984056, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.bert_tf.onnx'
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour='ortbert_tf'
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour='ortbert_tf'
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00038373230338287646, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=1.0132789611816406e-06, rel=0.00026781923006472165, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=1.0728836059570312e-06, rel=0.00034613720184126256, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=1.1026859283447266e-06, rel=0.0003551108443373418, n=102336.0
    [validate_model] run onnxruntime fusion for 'clip'
    failed in shape inference <class 'AssertionError'>
    [validate_model] done 'clip' in 0.16883487499990224, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.clip.onnx'
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour='ortclip'
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour='ortclip'
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00038373230338287646, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=1.0132789611816406e-06, rel=0.00026781923006472165, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=1.0728836059570312e-06, rel=0.00034613720184126256, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=1.1026859283447266e-06, rel=0.0003551108443373418, n=102336.0
    [validate_model] run onnxruntime fusion for 'conformer'
    failed in shape inference <class 'AssertionError'>
    [validate_model] done 'conformer' in 0.2126038750002408, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.conformer.onnx'
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour='ortconformer'
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour='ortconformer'
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00038373230338287646, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=1.0132789611816406e-06, rel=0.00026781923006472165, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=1.0728836059570312e-06, rel=0.00034613720184126256, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=1.1026859283447266e-06, rel=0.0003551108443373418, n=102336.0
    [validate_model] run onnxruntime fusion for 'gpt2'
    failed in shape inference <class 'AssertionError'>
    [validate_model] done 'gpt2' in 0.20293302700019922, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.gpt2.onnx'
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour='ortgpt2'
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour='ortgpt2'
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00038373230338287646, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=1.0132789611816406e-06, rel=0.00026781923006472165, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=1.0728836059570312e-06, rel=0.00034613720184126256, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=1.1026859283447266e-06, rel=0.0003551108443373418, n=102336.0
    [validate_model] run onnxruntime fusion for 'gpt2_tf'
    failed in shape inference <class 'AssertionError'>
    [validate_model] done 'gpt2_tf' in 0.1986968009996417, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.gpt2_tf.onnx'
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour='ortgpt2_tf'
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour='ortgpt2_tf'
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00038373230338287646, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=1.0132789611816406e-06, rel=0.00026781923006472165, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=1.0728836059570312e-06, rel=0.00034613720184126256, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=1.1026859283447266e-06, rel=0.0003551108443373418, n=102336.0
    [validate_model] run onnxruntime fusion for 'gpt_neox'
    failed in shape inference <class 'AssertionError'>
    [validate_model] done 'gpt_neox' in 0.19962834899979498, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.gpt_neox.onnx'
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour='ortgpt_neox'
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour='ortgpt_neox'
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00038373230338287646, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=1.0132789611816406e-06, rel=0.00026781923006472165, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=1.0728836059570312e-06, rel=0.00034613720184126256, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=1.1026859283447266e-06, rel=0.0003551108443373418, n=102336.0
    [validate_model] run onnxruntime fusion for 'mmdit'
    failed in shape inference <class 'AssertionError'>
    [validate_model] done 'mmdit' in 0.19610673299985137, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.mmdit.onnx'
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour='ortmmdit'
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour='ortmmdit'
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00038373230338287646, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=1.0132789611816406e-06, rel=0.00026781923006472165, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=1.0728836059570312e-06, rel=0.00034613720184126256, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=1.1026859283447266e-06, rel=0.0003551108443373418, n=102336.0
    [validate_model] run onnxruntime fusion for 'phi'
    [validate_model] done 'phi' in 0.06579585100007534, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.phi.onnx'
    [validate_onnx_model] missing 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.phi.onnx'
    [validate_model] run onnxruntime fusion for 'sam2'
    failed in shape inference <class 'AssertionError'>
    [validate_model] done 'sam2' in 0.1599450369999431, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.sam2.onnx'
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour='ortsam2'
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour='ortsam2'
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=7.748603820800781e-07, rel=0.00044309172106863606, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00032628376058705446, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=6.556510925292969e-07, rel=0.0002520801563113858, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=7.748603820800781e-07, rel=0.00037692802454639585, n=102336.0
    [validate_model] run onnxruntime fusion for 'swin'
    failed in shape inference <class 'AssertionError'>
    [validate_model] done 'swin' in 0.20968583400008356, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.swin.onnx'
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour='ortswin'
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour='ortswin'
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00038373230338287646, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=1.0132789611816406e-06, rel=0.00026781923006472165, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=1.0728836059570312e-06, rel=0.00034613720184126256, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=1.1026859283447266e-06, rel=0.0003551108443373418, n=102336.0
    [validate_model] run onnxruntime fusion for 't5'
    failed in shape inference <class 'AssertionError'>
    [validate_model] done 't5' in 0.23676425400026346, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.t5.onnx'
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour='ortt5'
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour='ortt5'
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00038373230338287646, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=1.0132789611816406e-06, rel=0.00026781923006472165, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=1.0728836059570312e-06, rel=0.00034613720184126256, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=1.1026859283447266e-06, rel=0.0003551108443373418, n=102336.0
    [validate_model] run onnxruntime fusion for 'tnlr'
    failed in shape inference <class 'AssertionError'>
    [validate_model] done 'tnlr' in 0.21958224299987705, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.tnlr.onnx'
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour='orttnlr'
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour='orttnlr'
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00038373230338287646, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=1.0132789611816406e-06, rel=0.00026781923006472165, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=1.0728836059570312e-06, rel=0.00034613720184126256, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=1.1026859283447266e-06, rel=0.0003551108443373418, n=102336.0
    [validate_model] run onnxruntime fusion for 'unet'
    failed in shape inference <class 'AssertionError'>
    [validate_model] done 'unet' in 0.20266016400000808, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.unet.onnx'
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour='ortunet'
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour='ortunet'
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=7.748603820800781e-07, rel=0.00044309172106863606, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00032628376058705446, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=6.556510925292969e-07, rel=0.0002520801563113858, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=7.748603820800781e-07, rel=0.00037692802454639585, n=102336.0
    [validate_model] run onnxruntime fusion for 'vae'
    failed in shape inference <class 'AssertionError'>
    [validate_model] done 'vae' in 0.19273402600038025, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.vae.onnx'
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour='ortvae'
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour='ortvae'
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=7.748603820800781e-07, rel=0.00044309172106863606, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00032628376058705446, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=6.556510925292969e-07, rel=0.0002520801563113858, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=7.748603820800781e-07, rel=0.00037692802454639585, n=102336.0
    [validate_model] run onnxruntime fusion for 'vit'
    failed in shape inference <class 'AssertionError'>
    [validate_model] done 'vit' in 0.16944560899992211, saved into 'dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.vit.onnx'
    [validate_onnx_model] verify onnx model with providers ['CPUExecutionProvider']..., flavour='ortvit'
    [validate_onnx_model] runtime is onnxruntime
    [validate_onnx_model] done (ort_session) flavour='ortvit'
    [validate_onnx_model] -- keys=[('inputs', 'run_expected', ''), ('inputs2', 'run_expected22', '22'), ('inputs_empty_cache', 'run_expected2_empty_cache', '2_empty_cache'), ('inputs_batch1', 'run_expected2_batch1', '2_batch1')]
    [validate_onnx_model] -- make_feeds for 'inputs'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96]
    [validate_onnx_model] discrepancies=abs=8.344650268554688e-07, rel=0.00038373230338287646, n=204672.0
    [validate_onnx_model] -- make_feeds for 'inputs2'...
    [validate_onnx_model] inputs=dict(input_ids:T7s3x4,attention_mask:T7s3x35,position_ids:T7s3x4,past_key_values:DynamicCache(key_cache=#1[T1s3x1x31x96], value_cache=#1[T1s3x1x31x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs22'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96]
    [validate_onnx_model] discrepancies=abs=1.0132789611816406e-06, rel=0.00026781923006472165, n=404160.0
    [validate_onnx_model] -- make_feeds for 'inputs_empty_cache'...
    [validate_onnx_model] inputs=dict(input_ids:T7s2x3,attention_mask:T7s2x3,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x0x96], value_cache=#1[T1s2x1x0x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_empty_cache'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96]
    [validate_onnx_model] discrepancies=abs=1.0728836059570312e-06, rel=0.00034613720184126256, n=193152.0
    [validate_onnx_model] -- make_feeds for 'inputs_batch1'...
    [validate_onnx_model] inputs=dict(input_ids:T7s1x3,attention_mask:T7s1x33,position_ids:T7s1x3,past_key_values:DynamicCache(key_cache=#1[T1s1x1x30x96], value_cache=#1[T1s1x1x30x96]))
    [validate_onnx_model] ort inputs=dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96)
    [validate_onnx_model] done (make_feeds)
    [validate_onnx_model] run session on inputs 'inputs2_batch1'...
    [validate_onnx_model] done (run)
    [validate_onnx_model] got=#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96]
    [validate_onnx_model] discrepancies=abs=1.1026859283447266e-06, rel=0.0003551108443373418, n=102336.0
    [validate_model] -- done (final)
    
    -- summary --
    :ERR_onnx_missing_ortphi,FileNotFoundError('dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.phi.onnx');
    :ERR_opt_ort_phi,'method' object is not iterable;
    :disc_onnx_ort_run22_abs,8.344650268554688e-07;
    :disc_onnx_ort_run22_abs_ortbart,1.0132789611816406e-06;
    :disc_onnx_ort_run22_abs_ortbert,1.0132789611816406e-06;
    :disc_onnx_ort_run22_abs_ortbert_keras,1.0132789611816406e-06;
    :disc_onnx_ort_run22_abs_ortbert_tf,1.0132789611816406e-06;
    :disc_onnx_ort_run22_abs_ortclip,1.0132789611816406e-06;
    :disc_onnx_ort_run22_abs_ortconformer,1.0132789611816406e-06;
    :disc_onnx_ort_run22_abs_ortgpt2,1.0132789611816406e-06;
    :disc_onnx_ort_run22_abs_ortgpt2_tf,1.0132789611816406e-06;
    :disc_onnx_ort_run22_abs_ortgpt_neox,1.0132789611816406e-06;
    :disc_onnx_ort_run22_abs_ortmmdit,1.0132789611816406e-06;
    :disc_onnx_ort_run22_abs_ortsam2,8.344650268554688e-07;
    :disc_onnx_ort_run22_abs_ortswin,1.0132789611816406e-06;
    :disc_onnx_ort_run22_abs_ortt5,1.0132789611816406e-06;
    :disc_onnx_ort_run22_abs_orttnlr,1.0132789611816406e-06;
    :disc_onnx_ort_run22_abs_ortunet,8.344650268554688e-07;
    :disc_onnx_ort_run22_abs_ortvae,8.344650268554688e-07;
    :disc_onnx_ort_run22_abs_ortvit,1.0132789611816406e-06;
    :disc_onnx_ort_run22_dnan,0;
    :disc_onnx_ort_run22_dnan_ortbart,0;
    :disc_onnx_ort_run22_dnan_ortbert,0;
    :disc_onnx_ort_run22_dnan_ortbert_keras,0;
    :disc_onnx_ort_run22_dnan_ortbert_tf,0;
    :disc_onnx_ort_run22_dnan_ortclip,0;
    :disc_onnx_ort_run22_dnan_ortconformer,0;
    :disc_onnx_ort_run22_dnan_ortgpt2,0;
    :disc_onnx_ort_run22_dnan_ortgpt2_tf,0;
    :disc_onnx_ort_run22_dnan_ortgpt_neox,0;
    :disc_onnx_ort_run22_dnan_ortmmdit,0;
    :disc_onnx_ort_run22_dnan_ortsam2,0;
    :disc_onnx_ort_run22_dnan_ortswin,0;
    :disc_onnx_ort_run22_dnan_ortt5,0;
    :disc_onnx_ort_run22_dnan_orttnlr,0;
    :disc_onnx_ort_run22_dnan_ortunet,0;
    :disc_onnx_ort_run22_dnan_ortvae,0;
    :disc_onnx_ort_run22_dnan_ortvit,0;
    :disc_onnx_ort_run22_n,404160.0;
    :disc_onnx_ort_run22_n_ortbart,404160.0;
    :disc_onnx_ort_run22_n_ortbert,404160.0;
    :disc_onnx_ort_run22_n_ortbert_keras,404160.0;
    :disc_onnx_ort_run22_n_ortbert_tf,404160.0;
    :disc_onnx_ort_run22_n_ortclip,404160.0;
    :disc_onnx_ort_run22_n_ortconformer,404160.0;
    :disc_onnx_ort_run22_n_ortgpt2,404160.0;
    :disc_onnx_ort_run22_n_ortgpt2_tf,404160.0;
    :disc_onnx_ort_run22_n_ortgpt_neox,404160.0;
    :disc_onnx_ort_run22_n_ortmmdit,404160.0;
    :disc_onnx_ort_run22_n_ortsam2,404160.0;
    :disc_onnx_ort_run22_n_ortswin,404160.0;
    :disc_onnx_ort_run22_n_ortt5,404160.0;
    :disc_onnx_ort_run22_n_orttnlr,404160.0;
    :disc_onnx_ort_run22_n_ortunet,404160.0;
    :disc_onnx_ort_run22_n_ortvae,404160.0;
    :disc_onnx_ort_run22_n_ortvit,404160.0;
    :disc_onnx_ort_run22_rel,0.00032628376058705446;
    :disc_onnx_ort_run22_rel_ortbart,0.00026781923006472165;
    :disc_onnx_ort_run22_rel_ortbert,0.00026781923006472165;
    :disc_onnx_ort_run22_rel_ortbert_keras,0.00026781923006472165;
    :disc_onnx_ort_run22_rel_ortbert_tf,0.00026781923006472165;
    :disc_onnx_ort_run22_rel_ortclip,0.00026781923006472165;
    :disc_onnx_ort_run22_rel_ortconformer,0.00026781923006472165;
    :disc_onnx_ort_run22_rel_ortgpt2,0.00026781923006472165;
    :disc_onnx_ort_run22_rel_ortgpt2_tf,0.00026781923006472165;
    :disc_onnx_ort_run22_rel_ortgpt_neox,0.00026781923006472165;
    :disc_onnx_ort_run22_rel_ortmmdit,0.00026781923006472165;
    :disc_onnx_ort_run22_rel_ortsam2,0.00032628376058705446;
    :disc_onnx_ort_run22_rel_ortswin,0.00026781923006472165;
    :disc_onnx_ort_run22_rel_ortt5,0.00026781923006472165;
    :disc_onnx_ort_run22_rel_orttnlr,0.00026781923006472165;
    :disc_onnx_ort_run22_rel_ortunet,0.00032628376058705446;
    :disc_onnx_ort_run22_rel_ortvae,0.00032628376058705446;
    :disc_onnx_ort_run22_rel_ortvit,0.00026781923006472165;
    :disc_onnx_ort_run22_sum,0.039355116007072866;
    :disc_onnx_ort_run22_sum_ortbart,0.042834832951484714;
    :disc_onnx_ort_run22_sum_ortbert,0.042834832951484714;
    :disc_onnx_ort_run22_sum_ortbert_keras,0.042834832951484714;
    :disc_onnx_ort_run22_sum_ortbert_tf,0.042834832951484714;
    :disc_onnx_ort_run22_sum_ortclip,0.042834832951484714;
    :disc_onnx_ort_run22_sum_ortconformer,0.042834832951484714;
    :disc_onnx_ort_run22_sum_ortgpt2,0.042834832951484714;
    :disc_onnx_ort_run22_sum_ortgpt2_tf,0.042834832951484714;
    :disc_onnx_ort_run22_sum_ortgpt_neox,0.042834832951484714;
    :disc_onnx_ort_run22_sum_ortmmdit,0.042834832951484714;
    :disc_onnx_ort_run22_sum_ortsam2,0.039355116007072866;
    :disc_onnx_ort_run22_sum_ortswin,0.042834832951484714;
    :disc_onnx_ort_run22_sum_ortt5,0.042834832951484714;
    :disc_onnx_ort_run22_sum_orttnlr,0.042834832951484714;
    :disc_onnx_ort_run22_sum_ortunet,0.039355116007072866;
    :disc_onnx_ort_run22_sum_ortvae,0.039355116007072866;
    :disc_onnx_ort_run22_sum_ortvit,0.042834832951484714;
    :disc_onnx_ort_run2_batch1_abs,7.748603820800781e-07;
    :disc_onnx_ort_run2_batch1_abs_ortbart,1.1026859283447266e-06;
    :disc_onnx_ort_run2_batch1_abs_ortbert,1.1026859283447266e-06;
    :disc_onnx_ort_run2_batch1_abs_ortbert_keras,1.1026859283447266e-06;
    :disc_onnx_ort_run2_batch1_abs_ortbert_tf,1.1026859283447266e-06;
    :disc_onnx_ort_run2_batch1_abs_ortclip,1.1026859283447266e-06;
    :disc_onnx_ort_run2_batch1_abs_ortconformer,1.1026859283447266e-06;
    :disc_onnx_ort_run2_batch1_abs_ortgpt2,1.1026859283447266e-06;
    :disc_onnx_ort_run2_batch1_abs_ortgpt2_tf,1.1026859283447266e-06;
    :disc_onnx_ort_run2_batch1_abs_ortgpt_neox,1.1026859283447266e-06;
    :disc_onnx_ort_run2_batch1_abs_ortmmdit,1.1026859283447266e-06;
    :disc_onnx_ort_run2_batch1_abs_ortsam2,7.748603820800781e-07;
    :disc_onnx_ort_run2_batch1_abs_ortswin,1.1026859283447266e-06;
    :disc_onnx_ort_run2_batch1_abs_ortt5,1.1026859283447266e-06;
    :disc_onnx_ort_run2_batch1_abs_orttnlr,1.1026859283447266e-06;
    :disc_onnx_ort_run2_batch1_abs_ortunet,7.748603820800781e-07;
    :disc_onnx_ort_run2_batch1_abs_ortvae,7.748603820800781e-07;
    :disc_onnx_ort_run2_batch1_abs_ortvit,1.1026859283447266e-06;
    :disc_onnx_ort_run2_batch1_dnan,0;
    :disc_onnx_ort_run2_batch1_dnan_ortbart,0;
    :disc_onnx_ort_run2_batch1_dnan_ortbert,0;
    :disc_onnx_ort_run2_batch1_dnan_ortbert_keras,0;
    :disc_onnx_ort_run2_batch1_dnan_ortbert_tf,0;
    :disc_onnx_ort_run2_batch1_dnan_ortclip,0;
    :disc_onnx_ort_run2_batch1_dnan_ortconformer,0;
    :disc_onnx_ort_run2_batch1_dnan_ortgpt2,0;
    :disc_onnx_ort_run2_batch1_dnan_ortgpt2_tf,0;
    :disc_onnx_ort_run2_batch1_dnan_ortgpt_neox,0;
    :disc_onnx_ort_run2_batch1_dnan_ortmmdit,0;
    :disc_onnx_ort_run2_batch1_dnan_ortsam2,0;
    :disc_onnx_ort_run2_batch1_dnan_ortswin,0;
    :disc_onnx_ort_run2_batch1_dnan_ortt5,0;
    :disc_onnx_ort_run2_batch1_dnan_orttnlr,0;
    :disc_onnx_ort_run2_batch1_dnan_ortunet,0;
    :disc_onnx_ort_run2_batch1_dnan_ortvae,0;
    :disc_onnx_ort_run2_batch1_dnan_ortvit,0;
    :disc_onnx_ort_run2_batch1_n,102336.0;
    :disc_onnx_ort_run2_batch1_n_ortbart,102336.0;
    :disc_onnx_ort_run2_batch1_n_ortbert,102336.0;
    :disc_onnx_ort_run2_batch1_n_ortbert_keras,102336.0;
    :disc_onnx_ort_run2_batch1_n_ortbert_tf,102336.0;
    :disc_onnx_ort_run2_batch1_n_ortclip,102336.0;
    :disc_onnx_ort_run2_batch1_n_ortconformer,102336.0;
    :disc_onnx_ort_run2_batch1_n_ortgpt2,102336.0;
    :disc_onnx_ort_run2_batch1_n_ortgpt2_tf,102336.0;
    :disc_onnx_ort_run2_batch1_n_ortgpt_neox,102336.0;
    :disc_onnx_ort_run2_batch1_n_ortmmdit,102336.0;
    :disc_onnx_ort_run2_batch1_n_ortsam2,102336.0;
    :disc_onnx_ort_run2_batch1_n_ortswin,102336.0;
    :disc_onnx_ort_run2_batch1_n_ortt5,102336.0;
    :disc_onnx_ort_run2_batch1_n_orttnlr,102336.0;
    :disc_onnx_ort_run2_batch1_n_ortunet,102336.0;
    :disc_onnx_ort_run2_batch1_n_ortvae,102336.0;
    :disc_onnx_ort_run2_batch1_n_ortvit,102336.0;
    :disc_onnx_ort_run2_batch1_rel,0.00037692802454639585;
    :disc_onnx_ort_run2_batch1_rel_ortbart,0.0003551108443373418;
    :disc_onnx_ort_run2_batch1_rel_ortbert,0.0003551108443373418;
    :disc_onnx_ort_run2_batch1_rel_ortbert_keras,0.0003551108443373418;
    :disc_onnx_ort_run2_batch1_rel_ortbert_tf,0.0003551108443373418;
    :disc_onnx_ort_run2_batch1_rel_ortclip,0.0003551108443373418;
    :disc_onnx_ort_run2_batch1_rel_ortconformer,0.0003551108443373418;
    :disc_onnx_ort_run2_batch1_rel_ortgpt2,0.0003551108443373418;
    :disc_onnx_ort_run2_batch1_rel_ortgpt2_tf,0.0003551108443373418;
    :disc_onnx_ort_run2_batch1_rel_ortgpt_neox,0.0003551108443373418;
    :disc_onnx_ort_run2_batch1_rel_ortmmdit,0.0003551108443373418;
    :disc_onnx_ort_run2_batch1_rel_ortsam2,0.00037692802454639585;
    :disc_onnx_ort_run2_batch1_rel_ortswin,0.0003551108443373418;
    :disc_onnx_ort_run2_batch1_rel_ortt5,0.0003551108443373418;
    :disc_onnx_ort_run2_batch1_rel_orttnlr,0.0003551108443373418;
    :disc_onnx_ort_run2_batch1_rel_ortunet,0.00037692802454639585;
    :disc_onnx_ort_run2_batch1_rel_ortvae,0.00037692802454639585;
    :disc_onnx_ort_run2_batch1_rel_ortvit,0.0003551108443373418;
    :disc_onnx_ort_run2_batch1_sum,0.011718470417235949;
    :disc_onnx_ort_run2_batch1_sum_ortbart,0.012457452539820224;
    :disc_onnx_ort_run2_batch1_sum_ortbert,0.012457452539820224;
    :disc_onnx_ort_run2_batch1_sum_ortbert_keras,0.012457452539820224;
    :disc_onnx_ort_run2_batch1_sum_ortbert_tf,0.012457452539820224;
    :disc_onnx_ort_run2_batch1_sum_ortclip,0.012457452539820224;
    :disc_onnx_ort_run2_batch1_sum_ortconformer,0.012457452539820224;
    :disc_onnx_ort_run2_batch1_sum_ortgpt2,0.012457452539820224;
    :disc_onnx_ort_run2_batch1_sum_ortgpt2_tf,0.012457452539820224;
    :disc_onnx_ort_run2_batch1_sum_ortgpt_neox,0.012457452539820224;
    :disc_onnx_ort_run2_batch1_sum_ortmmdit,0.012457452539820224;
    :disc_onnx_ort_run2_batch1_sum_ortsam2,0.011718470417235949;
    :disc_onnx_ort_run2_batch1_sum_ortswin,0.012457452539820224;
    :disc_onnx_ort_run2_batch1_sum_ortt5,0.012457452539820224;
    :disc_onnx_ort_run2_batch1_sum_orttnlr,0.012457452539820224;
    :disc_onnx_ort_run2_batch1_sum_ortunet,0.011718470417235949;
    :disc_onnx_ort_run2_batch1_sum_ortvae,0.011718470417235949;
    :disc_onnx_ort_run2_batch1_sum_ortvit,0.012457452539820224;
    :disc_onnx_ort_run2_empty_cache_abs,6.556510925292969e-07;
    :disc_onnx_ort_run2_empty_cache_abs_ortbart,1.0728836059570312e-06;
    :disc_onnx_ort_run2_empty_cache_abs_ortbert,1.0728836059570312e-06;
    :disc_onnx_ort_run2_empty_cache_abs_ortbert_keras,1.0728836059570312e-06;
    :disc_onnx_ort_run2_empty_cache_abs_ortbert_tf,1.0728836059570312e-06;
    :disc_onnx_ort_run2_empty_cache_abs_ortclip,1.0728836059570312e-06;
    :disc_onnx_ort_run2_empty_cache_abs_ortconformer,1.0728836059570312e-06;
    :disc_onnx_ort_run2_empty_cache_abs_ortgpt2,1.0728836059570312e-06;
    :disc_onnx_ort_run2_empty_cache_abs_ortgpt2_tf,1.0728836059570312e-06;
    :disc_onnx_ort_run2_empty_cache_abs_ortgpt_neox,1.0728836059570312e-06;
    :disc_onnx_ort_run2_empty_cache_abs_ortmmdit,1.0728836059570312e-06;
    :disc_onnx_ort_run2_empty_cache_abs_ortsam2,6.556510925292969e-07;
    :disc_onnx_ort_run2_empty_cache_abs_ortswin,1.0728836059570312e-06;
    :disc_onnx_ort_run2_empty_cache_abs_ortt5,1.0728836059570312e-06;
    :disc_onnx_ort_run2_empty_cache_abs_orttnlr,1.0728836059570312e-06;
    :disc_onnx_ort_run2_empty_cache_abs_ortunet,6.556510925292969e-07;
    :disc_onnx_ort_run2_empty_cache_abs_ortvae,6.556510925292969e-07;
    :disc_onnx_ort_run2_empty_cache_abs_ortvit,1.0728836059570312e-06;
    :disc_onnx_ort_run2_empty_cache_dnan,0;
    :disc_onnx_ort_run2_empty_cache_dnan_ortbart,0;
    :disc_onnx_ort_run2_empty_cache_dnan_ortbert,0;
    :disc_onnx_ort_run2_empty_cache_dnan_ortbert_keras,0;
    :disc_onnx_ort_run2_empty_cache_dnan_ortbert_tf,0;
    :disc_onnx_ort_run2_empty_cache_dnan_ortclip,0;
    :disc_onnx_ort_run2_empty_cache_dnan_ortconformer,0;
    :disc_onnx_ort_run2_empty_cache_dnan_ortgpt2,0;
    :disc_onnx_ort_run2_empty_cache_dnan_ortgpt2_tf,0;
    :disc_onnx_ort_run2_empty_cache_dnan_ortgpt_neox,0;
    :disc_onnx_ort_run2_empty_cache_dnan_ortmmdit,0;
    :disc_onnx_ort_run2_empty_cache_dnan_ortsam2,0;
    :disc_onnx_ort_run2_empty_cache_dnan_ortswin,0;
    :disc_onnx_ort_run2_empty_cache_dnan_ortt5,0;
    :disc_onnx_ort_run2_empty_cache_dnan_orttnlr,0;
    :disc_onnx_ort_run2_empty_cache_dnan_ortunet,0;
    :disc_onnx_ort_run2_empty_cache_dnan_ortvae,0;
    :disc_onnx_ort_run2_empty_cache_dnan_ortvit,0;
    :disc_onnx_ort_run2_empty_cache_n,193152.0;
    :disc_onnx_ort_run2_empty_cache_n_ortbart,193152.0;
    :disc_onnx_ort_run2_empty_cache_n_ortbert,193152.0;
    :disc_onnx_ort_run2_empty_cache_n_ortbert_keras,193152.0;
    :disc_onnx_ort_run2_empty_cache_n_ortbert_tf,193152.0;
    :disc_onnx_ort_run2_empty_cache_n_ortclip,193152.0;
    :disc_onnx_ort_run2_empty_cache_n_ortconformer,193152.0;
    :disc_onnx_ort_run2_empty_cache_n_ortgpt2,193152.0;
    :disc_onnx_ort_run2_empty_cache_n_ortgpt2_tf,193152.0;
    :disc_onnx_ort_run2_empty_cache_n_ortgpt_neox,193152.0;
    :disc_onnx_ort_run2_empty_cache_n_ortmmdit,193152.0;
    :disc_onnx_ort_run2_empty_cache_n_ortsam2,193152.0;
    :disc_onnx_ort_run2_empty_cache_n_ortswin,193152.0;
    :disc_onnx_ort_run2_empty_cache_n_ortt5,193152.0;
    :disc_onnx_ort_run2_empty_cache_n_orttnlr,193152.0;
    :disc_onnx_ort_run2_empty_cache_n_ortunet,193152.0;
    :disc_onnx_ort_run2_empty_cache_n_ortvae,193152.0;
    :disc_onnx_ort_run2_empty_cache_n_ortvit,193152.0;
    :disc_onnx_ort_run2_empty_cache_rel,0.0002520801563113858;
    :disc_onnx_ort_run2_empty_cache_rel_ortbart,0.00034613720184126256;
    :disc_onnx_ort_run2_empty_cache_rel_ortbert,0.00034613720184126256;
    :disc_onnx_ort_run2_empty_cache_rel_ortbert_keras,0.00034613720184126256;
    :disc_onnx_ort_run2_empty_cache_rel_ortbert_tf,0.00034613720184126256;
    :disc_onnx_ort_run2_empty_cache_rel_ortclip,0.00034613720184126256;
    :disc_onnx_ort_run2_empty_cache_rel_ortconformer,0.00034613720184126256;
    :disc_onnx_ort_run2_empty_cache_rel_ortgpt2,0.00034613720184126256;
    :disc_onnx_ort_run2_empty_cache_rel_ortgpt2_tf,0.00034613720184126256;
    :disc_onnx_ort_run2_empty_cache_rel_ortgpt_neox,0.00034613720184126256;
    :disc_onnx_ort_run2_empty_cache_rel_ortmmdit,0.00034613720184126256;
    :disc_onnx_ort_run2_empty_cache_rel_ortsam2,0.0002520801563113858;
    :disc_onnx_ort_run2_empty_cache_rel_ortswin,0.00034613720184126256;
    :disc_onnx_ort_run2_empty_cache_rel_ortt5,0.00034613720184126256;
    :disc_onnx_ort_run2_empty_cache_rel_orttnlr,0.00034613720184126256;
    :disc_onnx_ort_run2_empty_cache_rel_ortunet,0.0002520801563113858;
    :disc_onnx_ort_run2_empty_cache_rel_ortvae,0.0002520801563113858;
    :disc_onnx_ort_run2_empty_cache_rel_ortvit,0.00034613720184126256;
    :disc_onnx_ort_run2_empty_cache_sum,0.012811366097139398;
    :disc_onnx_ort_run2_empty_cache_sum_ortbart,0.020510222977463854;
    :disc_onnx_ort_run2_empty_cache_sum_ortbert,0.020510222977463854;
    :disc_onnx_ort_run2_empty_cache_sum_ortbert_keras,0.020510222977463854;
    :disc_onnx_ort_run2_empty_cache_sum_ortbert_tf,0.020510222977463854;
    :disc_onnx_ort_run2_empty_cache_sum_ortclip,0.020510222977463854;
    :disc_onnx_ort_run2_empty_cache_sum_ortconformer,0.020510222977463854;
    :disc_onnx_ort_run2_empty_cache_sum_ortgpt2,0.020510222977463854;
    :disc_onnx_ort_run2_empty_cache_sum_ortgpt2_tf,0.020510222977463854;
    :disc_onnx_ort_run2_empty_cache_sum_ortgpt_neox,0.020510222977463854;
    :disc_onnx_ort_run2_empty_cache_sum_ortmmdit,0.020510222977463854;
    :disc_onnx_ort_run2_empty_cache_sum_ortsam2,0.012811366097139398;
    :disc_onnx_ort_run2_empty_cache_sum_ortswin,0.020510222977463854;
    :disc_onnx_ort_run2_empty_cache_sum_ortt5,0.020510222977463854;
    :disc_onnx_ort_run2_empty_cache_sum_orttnlr,0.020510222977463854;
    :disc_onnx_ort_run2_empty_cache_sum_ortunet,0.012811366097139398;
    :disc_onnx_ort_run2_empty_cache_sum_ortvae,0.012811366097139398;
    :disc_onnx_ort_run2_empty_cache_sum_ortvit,0.020510222977463854;
    :disc_onnx_ort_run_abs,7.748603820800781e-07;
    :disc_onnx_ort_run_abs_ortbart,8.344650268554688e-07;
    :disc_onnx_ort_run_abs_ortbert,8.344650268554688e-07;
    :disc_onnx_ort_run_abs_ortbert_keras,8.344650268554688e-07;
    :disc_onnx_ort_run_abs_ortbert_tf,8.344650268554688e-07;
    :disc_onnx_ort_run_abs_ortclip,8.344650268554688e-07;
    :disc_onnx_ort_run_abs_ortconformer,8.344650268554688e-07;
    :disc_onnx_ort_run_abs_ortgpt2,8.344650268554688e-07;
    :disc_onnx_ort_run_abs_ortgpt2_tf,8.344650268554688e-07;
    :disc_onnx_ort_run_abs_ortgpt_neox,8.344650268554688e-07;
    :disc_onnx_ort_run_abs_ortmmdit,8.344650268554688e-07;
    :disc_onnx_ort_run_abs_ortsam2,7.748603820800781e-07;
    :disc_onnx_ort_run_abs_ortswin,8.344650268554688e-07;
    :disc_onnx_ort_run_abs_ortt5,8.344650268554688e-07;
    :disc_onnx_ort_run_abs_orttnlr,8.344650268554688e-07;
    :disc_onnx_ort_run_abs_ortunet,7.748603820800781e-07;
    :disc_onnx_ort_run_abs_ortvae,7.748603820800781e-07;
    :disc_onnx_ort_run_abs_ortvit,8.344650268554688e-07;
    :disc_onnx_ort_run_dnan,0;
    :disc_onnx_ort_run_dnan_ortbart,0;
    :disc_onnx_ort_run_dnan_ortbert,0;
    :disc_onnx_ort_run_dnan_ortbert_keras,0;
    :disc_onnx_ort_run_dnan_ortbert_tf,0;
    :disc_onnx_ort_run_dnan_ortclip,0;
    :disc_onnx_ort_run_dnan_ortconformer,0;
    :disc_onnx_ort_run_dnan_ortgpt2,0;
    :disc_onnx_ort_run_dnan_ortgpt2_tf,0;
    :disc_onnx_ort_run_dnan_ortgpt_neox,0;
    :disc_onnx_ort_run_dnan_ortmmdit,0;
    :disc_onnx_ort_run_dnan_ortsam2,0;
    :disc_onnx_ort_run_dnan_ortswin,0;
    :disc_onnx_ort_run_dnan_ortt5,0;
    :disc_onnx_ort_run_dnan_orttnlr,0;
    :disc_onnx_ort_run_dnan_ortunet,0;
    :disc_onnx_ort_run_dnan_ortvae,0;
    :disc_onnx_ort_run_dnan_ortvit,0;
    :disc_onnx_ort_run_n,204672.0;
    :disc_onnx_ort_run_n_ortbart,204672.0;
    :disc_onnx_ort_run_n_ortbert,204672.0;
    :disc_onnx_ort_run_n_ortbert_keras,204672.0;
    :disc_onnx_ort_run_n_ortbert_tf,204672.0;
    :disc_onnx_ort_run_n_ortclip,204672.0;
    :disc_onnx_ort_run_n_ortconformer,204672.0;
    :disc_onnx_ort_run_n_ortgpt2,204672.0;
    :disc_onnx_ort_run_n_ortgpt2_tf,204672.0;
    :disc_onnx_ort_run_n_ortgpt_neox,204672.0;
    :disc_onnx_ort_run_n_ortmmdit,204672.0;
    :disc_onnx_ort_run_n_ortsam2,204672.0;
    :disc_onnx_ort_run_n_ortswin,204672.0;
    :disc_onnx_ort_run_n_ortt5,204672.0;
    :disc_onnx_ort_run_n_orttnlr,204672.0;
    :disc_onnx_ort_run_n_ortunet,204672.0;
    :disc_onnx_ort_run_n_ortvae,204672.0;
    :disc_onnx_ort_run_n_ortvit,204672.0;
    :disc_onnx_ort_run_rel,0.00044309172106863606;
    :disc_onnx_ort_run_rel_ortbart,0.00038373230338287646;
    :disc_onnx_ort_run_rel_ortbert,0.00038373230338287646;
    :disc_onnx_ort_run_rel_ortbert_keras,0.00038373230338287646;
    :disc_onnx_ort_run_rel_ortbert_tf,0.00038373230338287646;
    :disc_onnx_ort_run_rel_ortclip,0.00038373230338287646;
    :disc_onnx_ort_run_rel_ortconformer,0.00038373230338287646;
    :disc_onnx_ort_run_rel_ortgpt2,0.00038373230338287646;
    :disc_onnx_ort_run_rel_ortgpt2_tf,0.00038373230338287646;
    :disc_onnx_ort_run_rel_ortgpt_neox,0.00038373230338287646;
    :disc_onnx_ort_run_rel_ortmmdit,0.00038373230338287646;
    :disc_onnx_ort_run_rel_ortsam2,0.00044309172106863606;
    :disc_onnx_ort_run_rel_ortswin,0.00038373230338287646;
    :disc_onnx_ort_run_rel_ortt5,0.00038373230338287646;
    :disc_onnx_ort_run_rel_orttnlr,0.00038373230338287646;
    :disc_onnx_ort_run_rel_ortunet,0.00044309172106863606;
    :disc_onnx_ort_run_rel_ortvae,0.00044309172106863606;
    :disc_onnx_ort_run_rel_ortvit,0.00038373230338287646;
    :disc_onnx_ort_run_sum,0.02031988672524676;
    :disc_onnx_ort_run_sum_ortbart,0.022044641082175076;
    :disc_onnx_ort_run_sum_ortbert,0.022044641082175076;
    :disc_onnx_ort_run_sum_ortbert_keras,0.022044641082175076;
    :disc_onnx_ort_run_sum_ortbert_tf,0.022044641082175076;
    :disc_onnx_ort_run_sum_ortclip,0.022044641082175076;
    :disc_onnx_ort_run_sum_ortconformer,0.022044641082175076;
    :disc_onnx_ort_run_sum_ortgpt2,0.022044641082175076;
    :disc_onnx_ort_run_sum_ortgpt2_tf,0.022044641082175076;
    :disc_onnx_ort_run_sum_ortgpt_neox,0.022044641082175076;
    :disc_onnx_ort_run_sum_ortmmdit,0.022044641082175076;
    :disc_onnx_ort_run_sum_ortsam2,0.02031988672524676;
    :disc_onnx_ort_run_sum_ortswin,0.022044641082175076;
    :disc_onnx_ort_run_sum_ortt5,0.022044641082175076;
    :disc_onnx_ort_run_sum_orttnlr,0.022044641082175076;
    :disc_onnx_ort_run_sum_ortunet,0.02031988672524676;
    :disc_onnx_ort_run_sum_ortvae,0.02031988672524676;
    :disc_onnx_ort_run_sum_ortvit,0.022044641082175076;
    :disc_patched_abs,0;
    :disc_patched_dnan,0;
    :disc_patched_n,204672.0;
    :disc_patched_rel,0;
    :disc_patched_sum,0.0;
    :dump_folder,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18;
    :dump_folder_name,arnir0_Tiny-LLM/onnx-dynamo/ir/op18;
    :export_args,();
    :export_dynamo,True;
    :export_exporter,onnx-dynamo;
    :export_kwargs,dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]));
    :export_opset,18;
    :export_optimization,ir;
    :model_class,LlamaForCausalLM;
    :model_config,{'vocab_size':32000,'max_position_embeddings':1024,'hidden_size':192,'intermediate_size':1024,'num_hidden_layers':1,'num_attention_heads':2,'num_key_value_heads':1,'hidden_act':'silu','initializer_range':0.02,'rms_norm_eps':1e-05,'pretraining_tp':1,'use_cache':True,'attention_bias':False,'attention_dropout':0.0,'mlp_bias':False,'head_dim':96,'rope_parameters':{'rope_type':'default','rope_theta':10000.0},'return_dict':True,'output_hidden_states':False,'dtype':'float32','tie_word_embeddings':False,'chunk_size_feed_forward':0,'is_encoder_decoder':False,'is_decoder':False,'cross_attention_hidden_size':None,'add_cross_attention':False,'tie_encoder_decoder':False,'architectures':['LlamaForCausalLM'],'finetuning_task':None,'id2label':{0:'LABEL_0',1:'LABEL_1'},'label2id':{'LABEL_0':0,'LABEL_1':1},'task_specific_params':None,'problem_type':None,'tokenizer_class':None,'prefix':None,'bos_token_id':1,'pad_token_id':None,'eos_token_id':2,'sep_token_id':None,'decoder_start_token_id':None,'max_length':20,'min_length':0,'do_sample':False,'early_stopping':False,'num_beams':1,'temperature':1.0,'top_k':50,'top_p':1.0,'typical_p':1.0,'repetition_penalty':1.0,'length_penalty':1.0,'no_repeat_ngram_size':0,'encoder_no_repeat_ngram_size':0,'bad_words_ids':None,'num_return_sequences':1,'output_scores':False,'return_dict_in_generate':False,'forced_bos_token_id':None,'forced_eos_token_id':None,'remove_invalid_values':False,'exponential_decay_length_penalty':None,'suppress_tokens':None,'begin_suppress_tokens':None,'num_beam_groups':1,'diversity_penalty':0.0,'_name_or_path':'','transformers_version':'5.0.0.dev0','model_type':'llama','rope_theta':10000.0,'subfolder':None,'output_attentions':False};
    :model_config_class,LlamaConfig;
    :model_file,~/github/transformers/src/transformers/models/llama/modeling_llama.py;
    :model_id,arnir0/Tiny-LLM;
    :model_inputs,dict(input_ids:T7s2x3,attention_mask:T7s2x33,position_ids:T7s2x3,past_key_values:DynamicCache(key_cache=#1[T1s2x1x30x96], value_cache=#1[T1s2x1x30x96]));
    :model_inputs_options,;
    :model_module,transformers.models.llama.modeling_llama;
    :model_nweights,12988992;
    :model_shapes,dict(input_ids:{0:DYN(batch),1:DYN(seq_length)},attention_mask:{0:DYN(batch),1:DYN(cache+seq)},position_ids:{0:DYN(batch),1:DYN(seq_length)},past_key_values:#2[#1[{0:DYN(batch),2:DYN(cache_length)}],#1[{0:DYN(batch),2:DYN(cache_length)}]]);
    :model_size,51955968;
    :model_subfolder,;
    :model_task,text-generation;
    :n_node_Add,10;
    :n_node_And,2;
    :n_node_Cast,2;
    :n_node_Concat,16;
    :n_node_Cos,1;
    :n_node_Expand,6;
    :n_node_Gather,1;
    :n_node_GatherND,1;
    :n_node_IsNaN,1;
    :n_node_LessOrEqual,1;
    :n_node_MatMul,11;
    :n_node_Mul,14;
    :n_node_Neg,2;
    :n_node_Pow,3;
    :n_node_Range,3;
    :n_node_Reciprocal,3;
    :n_node_ReduceMean,3;
    :n_node_Reshape,13;
    :n_node_Shape,5;
    :n_node_Sigmoid,1;
    :n_node_Sin,1;
    :n_node_Slice,7;
    :n_node_Softmax,1;
    :n_node_Sqrt,3;
    :n_node_Squeeze,4;
    :n_node_Transpose,6;
    :n_node_Unsqueeze,7;
    :n_node_Where,2;
    :n_node_functions,0;
    :n_node_initializer_1,16;
    :n_node_initializer_7,15;
    :n_node_initializer_9,1;
    :n_node_nodes,130;
    :n_node_nodes_nocst,130;
    :onnx_filename,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.onnx;
    :onnx_filename_ortbart,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.bart.onnx;
    :onnx_filename_ortbert,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.bert.onnx;
    :onnx_filename_ortbert_keras,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.bert_keras.onnx;
    :onnx_filename_ortbert_tf,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.bert_tf.onnx;
    :onnx_filename_ortclip,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.clip.onnx;
    :onnx_filename_ortconformer,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.conformer.onnx;
    :onnx_filename_ortgpt2,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.gpt2.onnx;
    :onnx_filename_ortgpt2_tf,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.gpt2_tf.onnx;
    :onnx_filename_ortgpt_neox,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.gpt_neox.onnx;
    :onnx_filename_ortmmdit,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.mmdit.onnx;
    :onnx_filename_ortsam2,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.sam2.onnx;
    :onnx_filename_ortswin,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.swin.onnx;
    :onnx_filename_ortt5,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.t5.onnx;
    :onnx_filename_orttnlr,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.tnlr.onnx;
    :onnx_filename_ortunet,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.unet.onnx;
    :onnx_filename_ortvae,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.vae.onnx;
    :onnx_filename_ortvit,dump_models/arnir0_Tiny-LLM/onnx-dynamo/ir/op18/arnir0_Tiny-LLM-onnx-dynamo-ir-op18.ort.vit.onnx;
    :onnx_ort_inputs,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs22,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs22_ortbart,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs22_ortbert,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs22_ortbert_keras,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs22_ortbert_tf,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs22_ortclip,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs22_ortconformer,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs22_ortgpt2,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs22_ortgpt2_tf,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs22_ortgpt_neox,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs22_ortmmdit,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs22_ortsam2,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs22_ortswin,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs22_ortt5,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs22_orttnlr,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs22_ortunet,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs22_ortvae,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs22_ortvit,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :onnx_ort_inputs2_batch1,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_batch1_ortbart,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_batch1_ortbert,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_batch1_ortbert_keras,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_batch1_ortbert_tf,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_batch1_ortclip,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_batch1_ortconformer,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_batch1_ortgpt2,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_batch1_ortgpt2_tf,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_batch1_ortgpt_neox,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_batch1_ortmmdit,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_batch1_ortsam2,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_batch1_ortswin,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_batch1_ortt5,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_batch1_orttnlr,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_batch1_ortunet,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_batch1_ortvae,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_batch1_ortvit,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :onnx_ort_inputs2_empty_cache,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs2_empty_cache_ortbart,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs2_empty_cache_ortbert,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs2_empty_cache_ortbert_keras,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs2_empty_cache_ortbert_tf,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs2_empty_cache_ortclip,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs2_empty_cache_ortconformer,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs2_empty_cache_ortgpt2,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs2_empty_cache_ortgpt2_tf,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs2_empty_cache_ortgpt_neox,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs2_empty_cache_ortmmdit,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs2_empty_cache_ortsam2,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs2_empty_cache_ortswin,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs2_empty_cache_ortt5,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs2_empty_cache_orttnlr,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs2_empty_cache_ortunet,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs2_empty_cache_ortvae,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs2_empty_cache_ortvit,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :onnx_ort_inputs_ortbart,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs_ortbert,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs_ortbert_keras,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs_ortbert_tf,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs_ortclip,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs_ortconformer,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs_ortgpt2,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs_ortgpt2_tf,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs_ortgpt_neox,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs_ortmmdit,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs_ortsam2,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs_ortswin,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs_ortt5,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs_orttnlr,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs_ortunet,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs_ortvae,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_ort_inputs_ortvit,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :onnx_size,200677;
    :onnx_size_ortbart,170127;
    :onnx_size_ortbert,170127;
    :onnx_size_ortbert_keras,170190;
    :onnx_size_ortbert_tf,170161;
    :onnx_size_ortclip,170127;
    :onnx_size_ortconformer,170179;
    :onnx_size_ortgpt2,170127;
    :onnx_size_ortgpt2_tf,170159;
    :onnx_size_ortgpt_neox,170168;
    :onnx_size_ortmmdit,170136;
    :onnx_size_ortsam2,201452;
    :onnx_size_ortswin,170127;
    :onnx_size_ortt5,170108;
    :onnx_size_orttnlr,170127;
    :onnx_size_ortunet,201452;
    :onnx_size_ortvae,201442;
    :onnx_size_ortvit,170117;
    :opt_ort_bart_delta_node,-18;
    :opt_ort_bart_duration,0.06871072900003128;
    :opt_ort_bart_duration_save,0.043745191999732924;
    :opt_ort_bart_n_nodes1,130;
    :opt_ort_bart_n_nodes2,112;
    :opt_ort_bert_delta_node,-18;
    :opt_ort_bert_duration,0.06676529100013795;
    :opt_ort_bert_duration_save,0.045182059000126173;
    :opt_ort_bert_keras_delta_node,-18;
    :opt_ort_bert_keras_duration,0.13777188500034754;
    :opt_ort_bert_keras_duration_save,0.03717316799975379;
    :opt_ort_bert_keras_n_nodes1,130;
    :opt_ort_bert_keras_n_nodes2,112;
    :opt_ort_bert_n_nodes1,130;
    :opt_ort_bert_n_nodes2,112;
    :opt_ort_bert_tf_delta_node,-18;
    :opt_ort_bert_tf_duration,0.02943789699975241;
    :opt_ort_bert_tf_duration_save,0.06269246100009696;
    :opt_ort_bert_tf_n_nodes1,130;
    :opt_ort_bert_tf_n_nodes2,112;
    :opt_ort_clip_delta_node,-18;
    :opt_ort_clip_duration,0.08001372300032017;
    :opt_ort_clip_duration_save,0.054731691000142746;
    :opt_ort_clip_n_nodes1,130;
    :opt_ort_clip_n_nodes2,112;
    :opt_ort_conformer_delta_node,-18;
    :opt_ort_conformer_duration,0.08355853700004445;
    :opt_ort_conformer_duration_save,0.07847254700027406;
    :opt_ort_conformer_n_nodes1,130;
    :opt_ort_conformer_n_nodes2,112;
    :opt_ort_gpt2_delta_node,-18;
    :opt_ort_gpt2_duration,0.0875048699999752;
    :opt_ort_gpt2_duration_save,0.055851423000149225;
    :opt_ort_gpt2_n_nodes1,130;
    :opt_ort_gpt2_n_nodes2,112;
    :opt_ort_gpt2_tf_delta_node,-18;
    :opt_ort_gpt2_tf_duration,0.08352191400035736;
    :opt_ort_gpt2_tf_duration_save,0.04383730000017749;
    :opt_ort_gpt2_tf_n_nodes1,130;
    :opt_ort_gpt2_tf_n_nodes2,112;
    :opt_ort_gpt_neox_delta_node,-18;
    :opt_ort_gpt_neox_duration,0.09340470500001175;
    :opt_ort_gpt_neox_duration_save,0.04682137999998304;
    :opt_ort_gpt_neox_n_nodes1,130;
    :opt_ort_gpt_neox_n_nodes2,112;
    :opt_ort_mmdit_delta_node,-18;
    :opt_ort_mmdit_duration,0.10243197199997667;
    :opt_ort_mmdit_duration_save,0.042310307000207104;
    :opt_ort_mmdit_n_nodes1,130;
    :opt_ort_mmdit_n_nodes2,112;
    :opt_ort_phi_duration,0.00011338500007695984;
    :opt_ort_sam2_delta_node,0;
    :opt_ort_sam2_duration,0.0813009459998284;
    :opt_ort_sam2_duration_save,0.032229560000359925;
    :opt_ort_sam2_n_nodes1,130;
    :opt_ort_sam2_n_nodes2,130;
    :opt_ort_swin_delta_node,-18;
    :opt_ort_swin_duration,0.0786684590002551;
    :opt_ort_swin_duration_save,0.04637126400029956;
    :opt_ort_swin_n_nodes1,130;
    :opt_ort_swin_n_nodes2,112;
    :opt_ort_t5_delta_node,-18;
    :opt_ort_t5_duration,0.0783139109998956;
    :opt_ort_t5_duration_save,0.05310510899971632;
    :opt_ort_t5_n_nodes1,130;
    :opt_ort_t5_n_nodes2,112;
    :opt_ort_tnlr_delta_node,-18;
    :opt_ort_tnlr_duration,0.07052623099980337;
    :opt_ort_tnlr_duration_save,0.0462378710003577;
    :opt_ort_tnlr_n_nodes1,130;
    :opt_ort_tnlr_n_nodes2,112;
    :opt_ort_unet_delta_node,0;
    :opt_ort_unet_duration,0.1166879649999828;
    :opt_ort_unet_duration_save,0.031566644999657;
    :opt_ort_unet_n_nodes1,130;
    :opt_ort_unet_n_nodes2,130;
    :opt_ort_vae_delta_node,0;
    :opt_ort_vae_duration,0.09508516599998984;
    :opt_ort_vae_duration_save,0.041896814000210725;
    :opt_ort_vae_n_nodes1,130;
    :opt_ort_vae_n_nodes2,130;
    :opt_ort_vit_delta_node,-18;
    :opt_ort_vit_duration,0.07645518099980109;
    :opt_ort_vit_duration_save,0.040547936000166374;
    :opt_ort_vit_n_nodes1,130;
    :opt_ort_vit_n_nodes2,112;
    :run_expected,CausalLMOutputWithPast(logits:T1s2x3x32000,past_key_values:DynamicCache(key_cache=#1[T1s2x1x33x96], value_cache=#1[T1s2x1x33x96]));
    :run_expected22,CausalLMOutputWithPast(logits:T1s3x4x32000,past_key_values:DynamicCache(key_cache=#1[T1s3x1x35x96], value_cache=#1[T1s3x1x35x96]));
    :run_expected2_batch1,CausalLMOutputWithPast(logits:T1s1x3x32000,past_key_values:DynamicCache(key_cache=#1[T1s1x1x33x96], value_cache=#1[T1s1x1x33x96]));
    :run_expected2_empty_cache,CausalLMOutputWithPast(logits:T1s2x3x32000,past_key_values:DynamicCache(key_cache=#1[T1s2x1x3x96], value_cache=#1[T1s2x1x3x96]));
    :run_feeds_inputs,dict(input_ids:A7s2x3,attention_mask:A7s2x33,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x30x96,past_key_values_value_cache_0:A1s2x1x30x96);
    :run_feeds_inputs2,dict(input_ids:A7s3x4,attention_mask:A7s3x35,position_ids:A7s3x4,past_key_values_key_cache_0:A1s3x1x31x96,past_key_values_value_cache_0:A1s3x1x31x96);
    :run_feeds_inputs_batch1,dict(input_ids:A7s1x3,attention_mask:A7s1x33,position_ids:A7s1x3,past_key_values_key_cache_0:A1s1x1x30x96,past_key_values_value_cache_0:A1s1x1x30x96);
    :run_feeds_inputs_empty_cache,dict(input_ids:A7s2x3,attention_mask:A7s2x3,position_ids:A7s2x3,past_key_values_key_cache_0:A1s2x1x0x96,past_key_values_value_cache_0:A1s2x1x0x96);
    :run_output_inputs,#3[A1s2x3x32000,A1s2x1x33x96,A1s2x1x33x96];
    :run_output_inputs2,#3[A1s3x4x32000,A1s3x1x35x96,A1s3x1x35x96];
    :run_output_inputs_batch1,#3[A1s1x3x32000,A1s1x1x33x96,A1s1x1x33x96];
    :run_output_inputs_empty_cache,#3[A1s2x3x32000,A1s2x1x3x96,A1s2x1x3x96];
    :second_input_keys,inputs2,inputs_empty_cache,inputs_batch1;
    :time_create_onnx_ort,0.04243475000021135;
    :time_create_onnx_ort_ortbart,0.021138034999694355;
    :time_create_onnx_ort_ortbert,0.0251042189997861;
    :time_create_onnx_ort_ortbert_keras,0.02258266300032119;
    :time_create_onnx_ort_ortbert_tf,0.02361042099983024;
    :time_create_onnx_ort_ortclip,0.021994427999743493;
    :time_create_onnx_ort_ortconformer,0.03874291500005711;
    :time_create_onnx_ort_ortgpt2,0.03893633999996382;
    :time_create_onnx_ort_ortgpt2_tf,0.023399002999667573;
    :time_create_onnx_ort_ortgpt_neox,0.03064009200033979;
    :time_create_onnx_ort_ortmmdit,0.0259936799998286;
    :time_create_onnx_ort_ortsam2,0.024399500000072294;
    :time_create_onnx_ort_ortswin,0.02504783500035046;
    :time_create_onnx_ort_ortt5,0.02286531699974148;
    :time_create_onnx_ort_orttnlr,0.02281566900001053;
    :time_create_onnx_ort_ortunet,0.027227439999933267;
    :time_create_onnx_ort_ortvae,0.038212310999824695;
    :time_create_onnx_ort_ortvit,0.04813481400015007;
    :time_create_torch_model,0.15771467800004757;
    :time_export_onnx,4.347349604000101;
    :time_export_onnx_opt_ir,0.041235744999994495;
    :time_onnx_save,0.19449118099964835;
    :time_ortfusion_ortbart,0.1720465699995657;
    :time_ortfusion_ortbert,0.16673424700002215;
    :time_ortfusion_ortbert_keras,0.2265212470001643;
    :time_ortfusion_ortbert_tf,0.10279864999984056;
    :time_ortfusion_ortclip,0.16883487499990224;
    :time_ortfusion_ortconformer,0.2126038750002408;
    :time_ortfusion_ortgpt2,0.20293302700019922;
    :time_ortfusion_ortgpt2_tf,0.1986968009996417;
    :time_ortfusion_ortgpt_neox,0.19962834899979498;
    :time_ortfusion_ortmmdit,0.19610673299985137;
    :time_ortfusion_ortphi,0.06579585100007534;
    :time_ortfusion_ortsam2,0.1599450369999431;
    :time_ortfusion_ortswin,0.20968583400008356;
    :time_ortfusion_ortt5,0.23676425400026346;
    :time_ortfusion_orttnlr,0.21958224299987705;
    :time_ortfusion_ortunet,0.20266016400000808;
    :time_ortfusion_ortvae,0.19273402600038025;
    :time_ortfusion_ortvit,0.16944560899992211;
    :time_preprocess_model_id,1.3420003597275354e-06;
    :time_run,0.050927087999752985;
    :time_run22,0.007231592000152887;
    :time_run2_batch1,0.005503260000295995;
    :time_run2_empty_cache,0.003326433999973233;
    :time_run_onnx_ort,0.009934372999850893;
    :time_run_onnx_ort22,0.0021273689999361522;
    :time_run_onnx_ort22_ortbart,0.001918444999773783;
    :time_run_onnx_ort22_ortbert,0.0017846390001068357;
    :time_run_onnx_ort22_ortbert_keras,0.0017331110002487549;
    :time_run_onnx_ort22_ortbert_tf,0.002920817000358511;
    :time_run_onnx_ort22_ortclip,0.0017232750001312525;
    :time_run_onnx_ort22_ortconformer,0.0020202440000502975;
    :time_run_onnx_ort22_ortgpt2,0.0024555700001656078;
    :time_run_onnx_ort22_ortgpt2_tf,0.001922232999731932;
    :time_run_onnx_ort22_ortgpt_neox,0.0017302609999205742;
    :time_run_onnx_ort22_ortmmdit,0.0017171800000141957;
    :time_run_onnx_ort22_ortsam2,0.0017724510003063187;
    :time_run_onnx_ort22_ortswin,0.0019098949996987358;
    :time_run_onnx_ort22_ortt5,0.0017552750000504602;
    :time_run_onnx_ort22_orttnlr,0.0018436299997119932;
    :time_run_onnx_ort22_ortunet,0.0018862360002458445;
    :time_run_onnx_ort22_ortvae,0.0031488159997934417;
    :time_run_onnx_ort22_ortvit,0.009258972999759862;
    :time_run_onnx_ort2_batch1,0.0010996850000992708;
    :time_run_onnx_ort2_batch1_ortbart,0.0010928280003099644;
    :time_run_onnx_ort2_batch1_ortbert,0.0011742690003302414;
    :time_run_onnx_ort2_batch1_ortbert_keras,0.0009669550004218763;
    :time_run_onnx_ort2_batch1_ortbert_tf,0.0014522559999932128;
    :time_run_onnx_ort2_batch1_ortclip,0.0011949229997298971;
    :time_run_onnx_ort2_batch1_ortconformer,0.0011417579999033478;
    :time_run_onnx_ort2_batch1_ortgpt2,0.0013865709997844533;
    :time_run_onnx_ort2_batch1_ortgpt2_tf,0.0011000019999301003;
    :time_run_onnx_ort2_batch1_ortgpt_neox,0.0010685219999686524;
    :time_run_onnx_ort2_batch1_ortmmdit,0.00106515000015861;
    :time_run_onnx_ort2_batch1_ortsam2,0.001110272999994777;
    :time_run_onnx_ort2_batch1_ortswin,0.0016524040001968388;
    :time_run_onnx_ort2_batch1_ortt5,0.0012124100003347849;
    :time_run_onnx_ort2_batch1_orttnlr,0.0010351509999964037;
    :time_run_onnx_ort2_batch1_ortunet,0.001642742000058206;
    :time_run_onnx_ort2_batch1_ortvae,0.0015353370004049793;
    :time_run_onnx_ort2_batch1_ortvit,0.0012128869998377922;
    :time_run_onnx_ort2_empty_cache,0.001358459999664774;
    :time_run_onnx_ort2_empty_cache_ortbart,0.0012955600000168488;
    :time_run_onnx_ort2_empty_cache_ortbert,0.0015221439998640562;
    :time_run_onnx_ort2_empty_cache_ortbert_keras,0.0012070949996996205;
    :time_run_onnx_ort2_empty_cache_ortbert_tf,0.0013570370001616539;
    :time_run_onnx_ort2_empty_cache_ortclip,0.0011988950000159093;
    :time_run_onnx_ort2_empty_cache_ortconformer,0.001245764000032068;
    :time_run_onnx_ort2_empty_cache_ortgpt2,0.001957451000180299;
    :time_run_onnx_ort2_empty_cache_ortgpt2_tf,0.0014118869999037997;
    :time_run_onnx_ort2_empty_cache_ortgpt_neox,0.001280779999888182;
    :time_run_onnx_ort2_empty_cache_ortmmdit,0.0013381649996517808;
    :time_run_onnx_ort2_empty_cache_ortsam2,0.001311544000145659;
    :time_run_onnx_ort2_empty_cache_ortswin,0.001347965999684675;
    :time_run_onnx_ort2_empty_cache_ortt5,0.001330986000084522;
    :time_run_onnx_ort2_empty_cache_orttnlr,0.001315321999754815;
    :time_run_onnx_ort2_empty_cache_ortunet,0.001336326000000554;
    :time_run_onnx_ort2_empty_cache_ortvae,0.0021984629997859884;
    :time_run_onnx_ort2_empty_cache_ortvit,0.01111250799976915;
    :time_run_onnx_ort_ortbart,0.0012059750001753855;
    :time_run_onnx_ort_ortbert,0.001577082000039809;
    :time_run_onnx_ort_ortbert_keras,0.001320847999977559;
    :time_run_onnx_ort_ortbert_tf,0.0012398359999679087;
    :time_run_onnx_ort_ortclip,0.006111495999903127;
    :time_run_onnx_ort_ortconformer,0.0019291050002721022;
    :time_run_onnx_ort_ortgpt2,0.002322327999991103;
    :time_run_onnx_ort_ortgpt2_tf,0.0017194130000461882;
    :time_run_onnx_ort_ortgpt_neox,0.0015608159997100302;
    :time_run_onnx_ort_ortmmdit,0.001626665999992838;
    :time_run_onnx_ort_ortsam2,0.001583689999733906;
    :time_run_onnx_ort_ortswin,0.002124379000179033;
    :time_run_onnx_ort_ortt5,0.0016111889999592677;
    :time_run_onnx_ort_orttnlr,0.0016405500000473694;
    :time_run_onnx_ort_ortunet,0.0016183930001716362;
    :time_run_onnx_ort_ortvae,0.0020812590000787168;
    :time_run_onnx_ort_ortvit,0.005197016000238364;
    :time_run_patched,0.030709064000348008;
    :time_torch_export_export,1.4399693660002413;
    :time_torch_export_export_n,1;
    :time_total,10.803030246000162;
    :time_total_exporter,5.28576296600022;
    :time_total_validation_onnx,0.09422458199969697;
    :time_total_validation_torch,0.06971565899993948;
    :version_date,2025-10-24T18:03:51;
    :version_device,;
    :version_do_run,True;
    :version_drop_inputs,[];
    :version_dtype,;
    :version_dump_folder,dump_models;
    :version_exporter,onnx-dynamo;
    :version_inputs2,1;
    :version_model_id,arnir0/Tiny-LLM;
    :version_numpy,2.3.4;
    :version_onnx,1.20.0;
    :version_onnx_diagnostic,0.7.16;
    :version_onnx_ir,0.1.12;
    :version_onnxruntime,1.24.0;
    :version_onnxscript,?;
    :version_opset,18;
    :version_optimization,ir;
    :version_ortbart_hidden_size,192;
    :version_ortbart_num_attention_heads,2;
    :version_ortbert_hidden_size,192;
    :version_ortbert_keras_hidden_size,192;
    :version_ortbert_keras_num_attention_heads,2;
    :version_ortbert_num_attention_heads,2;
    :version_ortbert_tf_hidden_size,192;
    :version_ortbert_tf_num_attention_heads,2;
    :version_ortclip_hidden_size,192;
    :version_ortclip_num_attention_heads,2;
    :version_ortconformer_hidden_size,192;
    :version_ortconformer_num_attention_heads,2;
    :version_ortfusiontype,ALL;
    :version_ortgpt2_hidden_size,192;
    :version_ortgpt2_num_attention_heads,2;
    :version_ortgpt2_tf_hidden_size,192;
    :version_ortgpt2_tf_num_attention_heads,2;
    :version_ortgpt_neox_hidden_size,192;
    :version_ortgpt_neox_num_attention_heads,2;
    :version_ortmmdit_hidden_size,192;
    :version_ortmmdit_num_attention_heads,2;
    :version_ortphi_hidden_size,192;
    :version_ortphi_num_attention_heads,2;
    :version_ortsam2_hidden_size,192;
    :version_ortsam2_num_attention_heads,2;
    :version_ortswin_hidden_size,192;
    :version_ortswin_num_attention_heads,2;
    :version_ortt5_hidden_size,192;
    :version_ortt5_num_attention_heads,2;
    :version_orttnlr_hidden_size,192;
    :version_orttnlr_num_attention_heads,2;
    :version_ortunet_hidden_size,192;
    :version_ortunet_num_attention_heads,2;
    :version_ortvae_hidden_size,192;
    :version_ortvae_num_attention_heads,2;
    :version_ortvit_hidden_size,192;
    :version_ortvit_num_attention_heads,2;
    :version_patch,{'patch': True};
    :version_patch_kwargs,{'patch':True,'patch_transformers':True,'patch_diffusers':True};
    :version_quiet,False;
    :version_rewrite,True;
    :version_runtime,onnxruntime;
    :version_same_as_pretrained,False;
    :version_scipy,1.16.2;
    :version_stop_if_static,0;
    :version_torch,2.10.0.dev20251022+cu130;
    :version_transformers,5.0.0.dev0;
    :version_use_pretrained,False;
    [runpythonerror]
    W1024 18:03:53.653000 10852 torch/fx/experimental/symbolic_shapes.py:6918] _maybe_guard_rel() was called on non-relation expression Eq(s72, 1) | Eq(Max(s44, s72), s72)
    W1024 18:03:53.658000 10852 torch/fx/experimental/symbolic_shapes.py:6918] _maybe_guard_rel() was called on non-relation expression Eq(s70, 1) | Eq(Max(s70, s9), s70)
    W1024 18:03:53.683000 10852 torch/fx/experimental/symbolic_shapes.py:6918] _maybe_guard_rel() was called on non-relation expression Eq(s44, 1) | Eq(Max(s44, s72), s44)
    W1024 18:03:53.686000 10852 torch/fx/experimental/symbolic_shapes.py:6918] _maybe_guard_rel() was called on non-relation expression Eq(s9, 1) | Eq(Max(s70, s9), s9)
    ~/vv/this312/lib/python3.12/site-packages/torch/onnx/_internal/exporter/_dynamic_shapes.py:272: UserWarning: # The axis name: batch will not be used, since it shares the same shape constraints with another axis: batch.
      warnings.warn(
    ~/vv/this312/lib/python3.12/site-packages/torch/onnx/_internal/exporter/_dynamic_shapes.py:272: UserWarning: # The axis name: seq_length will not be used, since it shares the same shape constraints with another axis: seq_length.
      warnings.warn(
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    Model producer not matched: Expected "keras2onnx", Got "pytorch".Please specify correct --model_type parameter.
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    Model producer not matched: Expected "tf2onnx", Got "pytorch".Please specify correct --model_type parameter.
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    Model producer not matched: Expected "tf2onnx", Got "pytorch".Please specify correct --model_type parameter.
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    
fusion:   0%|          | 0/5 [00:00<?, ?it/s]
                                             
The optimized model requires LayerNormalization with broadcast support. Please use onnxruntime-gpu>=1.21 for inference.
    
fusion:  20%|██        | 1/5 [00:00<00:00, 12.74it/s]
fusion: 100%|██████████| 5/5 [00:00<00:00, 58.24it/s]
    
sam2 fusion:   0%|          | 0/12 [00:00<?, ?it/s]
                                                   
symbolic shape inference disabled or failed.
    
sam2 fusion:  50%|█████     | 6/12 [00:00<00:00, 81.71it/s]
sam2 fusion: 100%|██████████| 12/12 [00:00<00:00, 157.66it/s]
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.
    
fusion:   0%|          | 0/18 [00:00<?, ?it/s]
                                              
symbolic shape inference disabled or failed.
    
fusion:  50%|█████     | 9/18 [00:00<00:00, 125.46it/s]
                                                       
SkipGroupNorm fusion will be skipped since symbolic shape inference disabled or failed.
    
fusion:  67%|██████▋   | 12/18 [00:00<00:00, 164.75it/s]
fusion: 100%|██████████| 18/18 [00:00<00:00, 161.85it/s]
fusion: 100%|██████████| 18/18 [00:00<00:00, 161.50it/s]
    
fusion:   0%|          | 0/18 [00:00<?, ?it/s]
                                              
symbolic shape inference disabled or failed.
    
fusion:  50%|█████     | 9/18 [00:00<00:00, 107.43it/s]
                                                       
SkipGroupNorm fusion will be skipped since symbolic shape inference disabled or failed.
    
fusion:  67%|██████▋   | 12/18 [00:00<00:00, 141.16it/s]
fusion: 100%|██████████| 18/18 [00:00<00:00, 201.24it/s]
    symbolic shape inference disabled or failed.
    symbolic shape inference disabled or failed.

Sdpa or Eager implementation or Use a StaticCache

Add --mop cache_implementation=static --iop cls_cache=StaticCache to use a StaticCache instead of a DynamicCache (default). Add --mop attn_implementation=eager to explicitly select eager implementation for attention.

python -m onnx_diagnostic validate \
            -m google/gemma-2b \
            --run \
            -v 1 \
            --export custom \
            -o dump_test \
            --dtype float16 \
            --device cpu \
            --patch \
            --no-quiet \
            --opt default \
            --rewrite \
            --mop attn_implementation=eager \
            --mop cache_implementation=static \
            --iop cls_cache=StaticCache