Pattern Optimizer¶

The pattern optimizer is implemented by class GraphBuilderPatternOptimization. It searches for a specific sequence of nodes in the graph and replaces it by another one without changing the inputs or the long_outputs of the graph. The goal of the optimizer is to make the whole computation graph more efficient. The goal of this implementation is to make this optimization as fast as possible. Assuming the nodes in an onnx graph are ordered in a way every input of a node was created by previous nodes, the optimizer must not require any global reordering. The cost should be in $O(N P I)$ in the worst case where N is the number of nodes, P is the number of patterns, I is the number of iterations.

It is difficult to foresee what a pattern needs in order to rewrite a part of the graph. This API tries to give as much freedom as it can without leaving too much to do to the developper which tries to add a new pattern.

Patterns¶

Patterns must inherit from PatternOptimization. This class defines two methods.

PatternOptimization.match¶

def match(
    self,
    g: "GraphBuilderPatternOptimization",
    node: NodeProto,
    matched: List[MatchResult],
) -> Optional[MatchResult]:

g is a GraphBuilderPatternOptimization, it holds all the existing nodes, is able to return any information about type, shape, the node before, the node after another one.
node: the matching must determine if some nodes around this one are part of set of nodes this pattern optimizer can rewrite. From there, the function explores wherever it needs, checking any condition it needs.
matched: usually unused, it returns of nodes already matching a pattern

The method must not modify the graph. The method returns None if no match is found or an instance of class MatchResult. It must contain:

a list of nodes involved in the rewriting. It does not mean all of them will be removed but all of them are needed to do the rewriting and must not be impacted by other pattern optimizer.
A function doing the rewriting (usually method apply of the pattern class).
An existing node where the rewritten nodes can be inserted. Knowing it makes it faster to rewriter. If not specified, the optimizer will automatically determine the position of the new nodes.

Debugging: method none

def none(
    self,
    node: Optional[NodeProto] = None,
    lineno: Optional[int] = None,
    msg: str = "",
):

It may be useful which reason made a pattern matching fail. Instead of returning None, method match can return the following expression:

return self.none(node, inspect.currentframe().f_lineno)

By setting the verbosity (see next Section), the user may then know which lines in the code returned None and which condition failed.

PatternOptimization.apply¶

@classmethod
def apply(
    cls, g: "GraphBuilder", *nodes: Sequence[NodeProto]
) -> List[NodeProto]:

The method does the rewriting. It assumes it can happen. It takes a list of nodes impacted by the rewriting. It assumes no other pattern optimizer modified them or will modify them. It receives the list of nodes returned by method apply. Since it is a list of argument, method match can include None values. The method returns the new nodes. The optimizer considers that any node given to this function is removed from the graph, and any node returned by it are added. If a received node must be kept, it must be added to the list of returned node.

Optimization Algorithm¶

It is implemented in method optimize

def optimize(
    self, max_iter=-1, remove_identity: bool = True
) -> List[Dict[str, Any]]:

The algorithm runs multiple iteration until the graph is not evolving or max_iter is reached. By default, it is equal to the number of nodes. An iteration is:

matches = []

builds all successors and predecessors

# Step 1: match

for all patterns P:

    for all nodes n:

        r = p.match(n)
        if r:
            if no node already scheduled to be rewritten by another match:
                matches.append(r)

# Step 2: apply

for all matches r:
    apply the match r

# Step 3: clean

remove unused nodes
remove identity nodes

This algorithm may apply more than one rewriting at each iteration but it guarantees the local structure when applying the rewriting was not altered by another one.

Adding a pattern¶

See #80 about the addition of a new pattern.

Example¶

Simple API¶

We consider the following simple model:

<<<

import torch
from experimental_experiment.helpers import pretty_onnx
from experimental_experiment.xbuilder import OptimizationOptions
from experimental_experiment.torch_interpreter import to_onnx


class MLP(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = torch.nn.Sequential(
            torch.nn.Linear(10, 32),
            torch.nn.ReLU(),
            torch.nn.Linear(32, 1),
        )

    def forward(self, x):
        return self.layers(x)


x = torch.rand(3, 10)
onx = to_onnx(
    MLP(), (x,), input_names=["x"], options=OptimizationOptions(patterns=None)
)
with open("temp_doc_mlp.onnx", "wb") as f:
    f.write(onx.SerializeToString())
print(pretty_onnx(onx))

>>>

    opset: domain='' version=18
    doc_string: large_model=False, inline=False, external_threshold=102...
    input: name='x' type=dtype('float32') shape=[3, 10]
    init: name='layers.0.weight' type=float32 shape=(32, 10)              -- DynamoInterpret.placeholder.1/P(layers.0.weight)
    init: name='layers.0.bias' type=float32 shape=(32,)                   -- DynamoInterpret.placeholder.1/P(layers.0.bias)
    init: name='layers.2.weight' type=float32 shape=(1, 32)               -- DynamoInterpret.placeholder.1/P(layers.2.weight)
    init: name='layers.2.bias' type=float32 shape=(1,) -- array([0.013], dtype=float32)-- DynamoInterpret.placeholder.1/P(layers.2.bias)
    Transpose(layers.0.weight, perm=[1,0]) -> _onx_transpose0
      MatMul(x, _onx_transpose0) -> _onx_matmul0
        Add(_onx_matmul0, layers.0.bias) -> linear
          Relu(linear) -> relu
    Transpose(layers.2.weight, perm=[1,0]) -> _onx_transpose02
      MatMul(relu, _onx_transpose02) -> _onx_matmul02
        Add(_onx_matmul02, layers.2.bias) -> output_0
    output: name='output_0' type=dtype('float32') shape=[3, 1]

Which we can renders as follows:

$digraph{ orientation=portrait; size=7; ranksep=0.25; nodesep=0.05; x [shape=box color=red label="x\nTensorProto.FLOAT\nshape=[3, 10]" fontsize=10]; output_0 [shape=box color=green label="output_0\nTensorProto.FLOAT\nshape=[3, 1]" fontsize=10]; layers_0_weight [shape=box label="layers_0_weight\nfloat32((32, 10))\n[[ 0.008 -0.192 0.219 0.008 0.154 -0.065 0.013..." fontsize=10]; layers_0_bias [shape=box label="layers_0_bias\nfloat32((32,))\n[ 0.028 -0.209 -0.152 -0.196 -0.151 0.097 -0.188 ..." fontsize=10]; layers_2_weight [shape=box label="layers_2_weight\nfloat32((1, 32))\n[[ 0.098 -0.027 0.155 -0.004 0.166 0.117 0.129..." fontsize=10]; layers_2_bias [shape=box label="layers_2_bias\nfloat32((1,))\n[0.013]" fontsize=10]; _onx_transpose0 [shape=box label="_onx_transpose0" fontsize=10]; linear [shape=box style="filled,rounded" color=orange label="Transpose\nperm=[1, 0]" fontsize=10]; layers_0_weight -> linear; linear -> _onx_transpose0; _onx_matmul0 [shape=box label="_onx_matmul0" fontsize=10]; Opset [shape=box style="filled,rounded" color=orange label="MatMul" fontsize=10]; x -> Opset; _onx_transpose0 -> Opset; Opset -> _onx_matmul0; linear [shape=box label="linear" fontsize=10]; Opset2 [shape=box style="filled,rounded" color=orange label="Add" fontsize=10]; _onx_matmul0 -> Opset2; layers_0_bias -> Opset2; Opset2 -> linear; relu [shape=box label="relu" fontsize=10]; relu [shape=box style="filled,rounded" color=orange label="Relu" fontsize=10]; linear -> relu; relu -> relu; _onx_transpose02 [shape=box label="_onx_transpose02" fontsize=10]; linear2 [shape=box style="filled,rounded" color=orange label="Transpose\nperm=[1, 0]" fontsize=10]; layers_2_weight -> linear2; linear2 -> _onx_transpose02; _onx_matmul02 [shape=box label="_onx_matmul02" fontsize=10]; Opset3 [shape=box style="filled,rounded" color=orange label="MatMul" fontsize=10]; relu -> Opset3; _onx_transpose02 -> Opset3; Opset3 -> _onx_matmul02; Opset4 [shape=box style="filled,rounded" color=orange label="Add" fontsize=10]; _onx_matmul02 -> Opset4; layers_2_bias -> Opset4; Opset4 -> output_0; }$

We then apply the optimizations by writing the following code:

<<<

import onnx
from experimental_experiment.helpers import pretty_onnx
from experimental_experiment.xbuilder import GraphBuilder

onx = onnx.load("temp_doc_mlp.onnx")

# The model is placed in a GraphBuilder.
# It creates dictionnaires to store shapes, ranks, types
# to make it easier to the optimizers to find the information
# they need. It still uses NodeProto to store nodes
gr = GraphBuilder(onx, infer_shapes=True)

# Let's optimize.
opt_onx = gr.to_onnx(optimize=True)
with open("temp_doc_mlp_opt.onnx", "wb") as f:
    f.write(opt_onx.SerializeToString())
print(pretty_onnx(opt_onx))

>>>

    opset: domain='' version=18
    doc_string: large_model=False, inline=False, external_threshold=102...
    input: name='x' type=dtype('float32') shape=[3, 10]
    init: name='layers.0.weight' type=float32 shape=(32, 10)              -- DynamoInterpret.placeholder.1/P(layers.0.weight)GraphBuilder._update_structures_with_proto.1/from(layers.0.weight)
    init: name='layers.0.bias' type=float32 shape=(32,)                   -- DynamoInterpret.placeholder.1/P(layers.0.bias)GraphBuilder._update_structures_with_proto.1/from(layers.0.bias)
    init: name='layers.2.weight' type=float32 shape=(1, 32)               -- DynamoInterpret.placeholder.1/P(layers.2.weight)GraphBuilder._update_structures_with_proto.1/from(layers.2.weight)
    init: name='layers.2.bias' type=float32 shape=(1,) -- array([0.013], dtype=float32)-- DynamoInterpret.placeholder.1/P(layers.2.bias)GraphBuilder._update_structures_with_proto.1/from(layers.2.bias)
    Gemm(x, layers.0.weight, layers.0.bias, transB=1) -> linear
      Relu(linear) -> relu
        Gemm(relu, layers.2.weight, layers.2.bias, transB=1) -> output_0
    output: name='output_0' type=dtype('float32') shape=[3, 1]

Which renders as follows:

$digraph{ orientation=portrait; size=7; ranksep=0.25; nodesep=0.05; x [shape=box color=red label="x\nTensorProto.FLOAT\nshape=[3, 10]" fontsize=10]; output_0 [shape=box color=green label="output_0\nTensorProto.FLOAT\nshape=[3, 1]" fontsize=10]; layers_0_weight [shape=box label="layers_0_weight\nfloat32((32, 10))\n[[ 0.008 -0.192 0.219 0.008 0.154 -0.065 0.013..." fontsize=10]; layers_0_bias [shape=box label="layers_0_bias\nfloat32((32,))\n[ 0.028 -0.209 -0.152 -0.196 -0.151 0.097 -0.188 ..." fontsize=10]; layers_2_weight [shape=box label="layers_2_weight\nfloat32((1, 32))\n[[ 0.098 -0.027 0.155 -0.004 0.166 0.117 0.129..." fontsize=10]; layers_2_bias [shape=box label="layers_2_bias\nfloat32((1,))\n[0.013]" fontsize=10]; linear [shape=box label="linear" fontsize=10]; GemmTransposePattern__MatMulAddPattern__Opset2 [shape=box style="filled,rounded" color=orange label="Gemm\ntransB=1" fontsize=10]; x -> GemmTransposePattern__MatMulAddPattern__Opset2; layers_0_weight -> GemmTransposePattern__MatMulAddPattern__Opset2; layers_0_bias -> GemmTransposePattern__MatMulAddPattern__Opset2; GemmTransposePattern__MatMulAddPattern__Opset2 -> linear; relu [shape=box label="relu" fontsize=10]; relu [shape=box style="filled,rounded" color=orange label="Relu" fontsize=10]; linear -> relu; relu -> relu; GemmTransposePattern__MatMulAddPattern__Opset32 [shape=box style="filled,rounded" color=orange label="Gemm\ntransB=1" fontsize=10]; relu -> GemmTransposePattern__MatMulAddPattern__Opset32; layers_2_weight -> GemmTransposePattern__MatMulAddPattern__Opset32; layers_2_bias -> GemmTransposePattern__MatMulAddPattern__Opset32; GemmTransposePattern__MatMulAddPattern__Opset32 -> output_0; }$

Verbosity¶

<<<

import onnx
from experimental_experiment.xbuilder import GraphBuilder

onx = onnx.load("temp_doc_mlp.onnx")

gr = GraphBuilder(onx, infer_shapes=True, verbose=1)
opt_onx = gr.to_onnx(optimize=True)

>>>

    [GraphBuilder.optimize] start with 7 nodes
    [GraphBuilder.optimize] #patterns=44
    [GraphBuilderPatternOptimization.optimize] start with 7 nodes, 4 initializers, 44 patterns, priorities=[0, 1]
    [GraphBuilderPatternOptimization.optimize] iteration 0: 7 nodes, priority=0
    [GraphBuilderPatternOptimization.optimize] increase priority to 1
    [GraphBuilderPatternOptimization.optimize] iteration 1: 7 nodes, priority=1
    [GraphBuilderPatternOptimization.optimize] applies 2 matches, 2*MatMulAddPattern - time=0.001 | max_time=IdentityPattern:0.000
    [GraphBuilderPatternOptimization.optimize] iteration 2: 5 nodes, priority=1
    [GraphBuilderPatternOptimization.optimize] applies 2 matches, 2*GemmTransposePattern - time=0.000 | max_time=TransposeMatMulPattern:0.000
    [GraphBuilderPatternOptimization.optimize] iteration 3: 7 nodes, priority=1
    [GraphBuilderPatternOptimization.optimize] applies 2 matches, 2*TransposeTransposePattern - time=0.000 | max_time=TransposeTransposePattern:0.000
    [GraphBuilderPatternOptimization.optimize] iteration 4: 3 nodes, priority=1
    [GraphBuilderPatternOptimization.optimize] done after 5 iterations with 3 nodes in 0.007
    [GraphBuilder.optimize] done with 3 nodes in 0.008
    [GraphBuilder-BLO.to_onnx] make_model 4 inits 0 params
    [GraphBuilder-BLO.time_evaluation_constants_] 0
    [GraphBuilder-BLO._build_initializers] start with 4 initializers, large_model=False, external_threshold=1024
    [GraphBuilder-BLO._build_initializers] switch low/high order
    [GraphBuilder-BLO._build_initializers] done in 8.370006980840117e-07s with 4 initializers, 0 large initializers

With more verbosity:

<<<

import onnx
from experimental_experiment.xbuilder import GraphBuilder

onx = onnx.load("temp_doc_mlp.onnx")

gr = GraphBuilder(onx, infer_shapes=True, verbose=11)
opt_onx = gr.to_onnx(optimize=True)

>>>

    [GraphBuilder._update_structures_with_proto] starts with 7 nodes
    [GraphBuilder-TBC.set_shape] layers.0.weight:(32, 10)
    [GraphBuilder-TBC.set_rank] layers.0.weight:2
    [GraphBuilder-TBC.set_type] layers.0.weight:1
    [GraphBuilder-TBC.make_initializer] layers.0.weight[1:(32, 10)]
    [GraphBuilder.update_node_constant] new constant 'layers.0.weight', node=None
    [GraphBuilder-TBC.set_shape] layers.0.bias:(32,)
    [GraphBuilder-TBC.set_rank] layers.0.bias:1
    [GraphBuilder-TBC.set_type] layers.0.bias:1
    [GraphBuilder-TBC.make_initializer] layers.0.bias[1:(32,)]
    [GraphBuilder.update_node_constant] new constant 'layers.0.bias', node=None
    [GraphBuilder-TBC.set_shape] layers.2.weight:(1, 32)
    [GraphBuilder-TBC.set_rank] layers.2.weight:2
    [GraphBuilder-TBC.set_type] layers.2.weight:1
    [GraphBuilder-TBC.make_initializer] layers.2.weight[1:(1, 32)]
    [GraphBuilder.update_node_constant] new constant 'layers.2.weight', node=None
    [GraphBuilder-TBC.set_shape] layers.2.bias:(1,)
    [GraphBuilder-TBC.set_rank] layers.2.bias:1
    [GraphBuilder-TBC.set_type] layers.2.bias:1
    [GraphBuilder-TBC.make_initializer] layers.2.bias[1:(1,)]
    [GraphBuilder.update_node_constant] new constant 'layers.2.bias', node=None
    [GraphBuilder-TBC.set_type] x:1
    [GraphBuilder-TBC.set_shape] x:(3, 10)
    [GraphBuilder-TBC.set_rank] x:2
    [GraphBuilder-TBC.set_type] output_0:1
    [GraphBuilder-TBC.set_shape] output_0:(3, 1)
    [GraphBuilder-TBC.set_rank] output_0:2
    [GraphBuilder.update_node_constant] new constant '_onx_transpose0', node=Transpose
    [GraphBuilder-TBC.set_type] _onx_transpose0:1
    [GraphBuilder-TBC.set_shape] _onx_transpose0:(10, 32)
    [GraphBuilder-TBC.set_rank] _onx_transpose0:2
    [GraphBuilder-TBC.set_type] _onx_transpose0:1
    [GraphBuilder-TBC.set_type] _onx_matmul0:1
    [GraphBuilder-TBC.set_shape] _onx_matmul0:(3, 32)
    [GraphBuilder-TBC.set_rank] _onx_matmul0:2
    [GraphBuilder-TBC.set_type] _onx_matmul0:1
    [GraphBuilder-TBC.set_type] linear:1
    [GraphBuilder-TBC.set_shape] linear:(3, 32)
    [GraphBuilder-TBC.set_rank] linear:2
    [GraphBuilder-TBC.set_type] linear:1
    [GraphBuilder-TBC.set_type] relu:1
    [GraphBuilder-TBC.set_shape] relu:(3, 32)
    [GraphBuilder-TBC.set_rank] relu:2
    [GraphBuilder-TBC.set_type] relu:1
    [GraphBuilder.update_node_constant] new constant '_onx_transpose02', node=Transpose
    [GraphBuilder-TBC.set_type] _onx_transpose02:1
    [GraphBuilder-TBC.set_shape] _onx_transpose02:(32, 1)
    [GraphBuilder-TBC.set_rank] _onx_transpose02:2
    [GraphBuilder-TBC.set_type] _onx_transpose02:1
    [GraphBuilder-TBC.set_type] _onx_matmul02:1
    [GraphBuilder-TBC.set_shape] _onx_matmul02:(3, 1)
    [GraphBuilder-TBC.set_rank] _onx_matmul02:2
    [GraphBuilder-TBC.set_type] _onx_matmul02:1
    [GraphBuilder-TBC.set_type] output_0:1
    [GraphBuilder._update_structures_with_proto] ends with 7 nodes in 0.0006945059976715129
    [GraphBuilder.constant_folding] starts with 6 constants and 7 nodes.
    [GraphBuilder.constant_folding] cst:: . :: _onx_matmul02
    [GraphBuilder.constant_folding] cst:: 1 :: _onx_transpose0
    [GraphBuilder.constant_folding] cst:: . :: _onx_matmul0
    [GraphBuilder.constant_folding] cst:: 1 :: layers.2.bias
    [GraphBuilder.constant_folding] cst:: . :: output_0
    [GraphBuilder.constant_folding] cst:: 1 :: layers.0.bias
    [GraphBuilder.constant_folding] cst:: 1 :: layers.2.weight
    [GraphBuilder.constant_folding] cst:: . :: relu
    [GraphBuilder.constant_folding] cst:: . :: x
    [GraphBuilder.constant_folding] cst:: 1 :: layers.0.weight
    [GraphBuilder.constant_folding] cst:: . :: linear
    [GraphBuilder.constant_folding] cst:: 1 :: _onx_transpose02
    [GraphBuilder.constant_folding] initializer: layers.0.weight
    [GraphBuilder.constant_folding] initializer: layers.0.bias
    [GraphBuilder.constant_folding] initializer: layers.2.weight
    [GraphBuilder.constant_folding] initializer: layers.2.bias
    [GraphBuilder.constant_folding] from: Transpose(_onx_transpose0)
    [GraphBuilder.constant_folding] fold_constant:Transpose:_onx_transpose0[torch.float32:torch.Size([10, 32])]:from:layers.0.weight
    [GraphBuilder.constant_folding] from: Transpose(_onx_transpose02)
    [GraphBuilder.constant_folding] fold_constant:Transpose:_onx_transpose02[torch.float32:torch.Size([32, 1])]:from:layers.2.weight
    [GraphBuilder.update_node_constant] new constant '_onx_transpose0', node=Transpose
    [GraphBuilder.update_node_constant] new constant '_onx_transpose02', node=Transpose
    [GraphBuilder.constant_folding] ends with 6 constants and 7 nodes in 0.0003197229998477269 seconds
    [GraphBuilder._update_shape_types_with_proto] starts with 7 nodes and 11 shapes.
    [GraphBuilder._update_shape_types_with_proto] infer shapes
    [GraphBuilder._update_shape_types_with_proto] infer shapes done 0.0003379400004632771 seconds
    [GraphBuilder._update_shape_types_with_proto] _clean_shapes after 0.0004058300000906456 seconds
    [GraphBuilder._update_shape_types_with_proto] walk through 11 shapes.
    [GraphBuilder-TBC.set_type] p_layers_0_weight:1
    [GraphBuilder-TBC.set_shape] p_layers_0_weight:(32, 10)
    [GraphBuilder-TBC.set_rank] p_layers_0_weight:2
    [GraphBuilder-TBC.set_type] _onx_matmul02:1
    [GraphBuilder-TBC.set_type] _onx_transpose0:1
    [GraphBuilder-TBC.set_type] _onx_matmul0:1
    [GraphBuilder-TBC.set_type] p_layers_2_weight:1
    [GraphBuilder-TBC.set_shape] p_layers_2_weight:(1, 32)
    [GraphBuilder-TBC.set_rank] p_layers_2_weight:2
    [GraphBuilder-TBC.set_type] relu:1
    [GraphBuilder-TBC.set_type] p_layers_0_bias:1
    [GraphBuilder-TBC.set_shape] p_layers_0_bias:(32,)
    [GraphBuilder-TBC.set_rank] p_layers_0_bias:1
    [GraphBuilder-TBC.set_type] p_layers_2_bias:1
    [GraphBuilder-TBC.set_shape] p_layers_2_bias:(1,)
    [GraphBuilder-TBC.set_rank] p_layers_2_bias:1
    [GraphBuilder-TBC.set_type] linear:1
    [GraphBuilder-TBC.set_type] linear_1:1
    [GraphBuilder-TBC.set_shape] linear_1:(3, 1)
    [GraphBuilder-TBC.set_rank] linear_1:2
    [GraphBuilder-TBC.set_type] _onx_transpose02:1
    [GraphBuilder._update_shape_types_with_proto] ends in 0.0001459359991713427 seconds.
    [GraphBuilder.optimize] start with 7 nodes
    [GraphBuilder.optimize] options=OptimizationOptions(remove_unused=True, remove_identity=True,
        constant_folding=False, constant_size=1024, constant_fusing=True,
        verbose=11, max_iter=-1, recursive=False, processor=CPU, order=None,
        patterns=['BatchNormalizationPattern', 'BatchNormalizationTrainingPattern',
        'CastLayerNormalizationCastPattern', 'CastPattern', 'CastCastBinaryPattern',
        'CastOpCastPattern', 'ClipClipPattern', 'ComputationCastOpCastPattern',
        'ConvBiasNullPattern', 'DropoutPattern', 'ExpandPattern',
        'ExpandBroadcastPattern', 'ExpandSwapPattern', 'GeluPattern',
        'IdentityPattern', 'LayerNormalizationPattern',
        'LayerNormalizationScalePattern', 'LeakyReluPattern',
        'MulMulMulScalarPattern', 'ReduceReshapePattern',
        'ReduceSumNormalizePattern', 'ReshapePattern',
        'ReshapeMatMulReshapePattern', 'Reshape2Of3Pattern',
        'ReshapeReshapeBinaryPattern', 'MatMulAddPattern', 'GemmTransposePattern',
        'MatMulReshape2Of3Pattern', 'MulMulMatMulPattern', 'ReshapeReshapePattern',
        'RotaryConcatPartPattern', 'SameChildrenPattern',
        'SequenceConstructAtPattern', 'SliceSlicePattern', 'SlicesSplitPattern',
        'SoftmaxCrossEntropyLossCastPattern', 'Sub1MulPattern',
        'SwitchOrderBinaryPattern', 'TransposeMatMulPattern',
        'TransposeReshapeMatMulPattern', 'TransposeReshapeTransposePattern',
        'TransposeTransposePattern', 'UnsqueezeEqualPattern',
        'UnsqueezeUnsqueezePattern'])
    [GraphBuilder.remove_identity_nodes] starts with 7
    [GraphBuilder.remove_identity_nodes] found 0 replacements
    [GraphBuilder.remove_identity_nodes] kept 7 nodes
    [GraphBuilder.remove_identity_nodes] ends with 7 nodes in 2.7557001885725185e-05 seconds
    [GraphBuilderPatternOptimization.optimize] start with 7 nodes, 4 initializers, 44 patterns, priorities=[0, 1]
    [GraphBuilderPatternOptimization.optimize] use pattern   1/44 - P0 - BatchNormalizationPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern   2/44 - P0 - BatchNormalizationTrainingPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern   3/44 - P0 - CastPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern   4/44 - P0 - ConvBiasNullPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern   5/44 - P0 - ExpandPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern   6/44 - P0 - GeluPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern   7/44 - P0 - IdentityPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern   8/44 - P0 - LeakyReluPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern   9/44 - P0 - ReshapePattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  10/44 - P0 - ReshapeReshapePattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  11/44 - P0 - SameChildrenPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  12/44 - P0 - SoftmaxCrossEntropyLossCastPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  13/44 - P0 - TransposeReshapeTransposePattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  14/44 - P0 - TransposeTransposePattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  15/44 - P0 - UnsqueezeUnsqueezePattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  16/44 - P1 - CastCastBinaryPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  17/44 - P1 - CastLayerNormalizationCastPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  18/44 - P1 - CastOpCastPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  19/44 - P1 - ClipClipPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  20/44 - P1 - ComputationCastOpCastPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  21/44 - P1 - DropoutPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  22/44 - P1 - ExpandBroadcastPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  23/44 - P1 - ExpandSwapPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  24/44 - P1 - GemmTransposePattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  25/44 - P1 - LayerNormalizationPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  26/44 - P1 - LayerNormalizationScalePattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  27/44 - P1 - MatMulAddPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  28/44 - P1 - MatMulReshape2Of3Pattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  29/44 - P1 - MulMulMatMulPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  30/44 - P1 - MulMulMulScalarPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  31/44 - P1 - ReduceReshapePattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  32/44 - P1 - ReduceSumNormalizePattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  33/44 - P1 - Reshape2Of3Pattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  34/44 - P1 - ReshapeMatMulReshapePattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  35/44 - P1 - ReshapeReshapeBinaryPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  36/44 - P1 - RotaryConcatPartPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  37/44 - P1 - SequenceConstructAtPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  38/44 - P1 - SliceSlicePattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  39/44 - P1 - SlicesSplitPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  40/44 - P1 - Sub1MulPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  41/44 - P1 - SwitchOrderBinaryPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  42/44 - P1 - TransposeMatMulPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  43/44 - P1 - TransposeReshapeMatMulPattern()
    [GraphBuilderPatternOptimization.optimize] use pattern  44/44 - P1 - UnsqueezeEqualPattern()
    --
    
    opset: : 18
    init: layers.0.weight: ?: ?                                            -- GraphBuilder._update_structures_with_proto.1/from(layers.0.weight)
    init: layers.0.bias: ?: ?                                              -- GraphBuilder._update_structures_with_proto.1/from(layers.0.bias)
    init: layers.2.weight: ?: ?                                            -- GraphBuilder._update_structures_with_proto.1/from(layers.2.weight)
    init: layers.2.bias: ?: ?                                              -- GraphBuilder._update_structures_with_proto.1/from(layers.2.bias)
    input:: x                                                                       |T1: 3 x 10
    Transpose: layers.0.weight -> _onx_transpose0                                   |T1: 10 x 32                  - linear
    MatMul: x, _onx_transpose0 -> _onx_matmul0                                      |T1: 3 x 32                   - Opset
    Add: _onx_matmul0, layers.0.bias -> linear                                      |T1: 3 x 32                   - Opset2
    Relu: linear -> relu                                                            |T1: 3 x 32                   - relu
    Transpose: layers.2.weight -> _onx_transpose02                                  |T1: 32 x 1                   - linear2
    MatMul: relu, _onx_transpose02 -> _onx_matmul02                                 |T1: 3 x 1                    - Opset3
    Add: _onx_matmul02, layers.2.bias -> output_0                                   |T1: 3 x 1                    - Opset4
    output:: output_0                                                               |T1: 3 x 1
    --
    [GraphBuilderPatternOptimization.optimize] iteration 0: 7 nodes, priority=0
    [IdentityPattern.match] NONE - line: 165:experimental_experiment.xoptim.patterns.onnx_any, op_type=Transpose, name=linear
    [IdentityPattern.match] NONE - line: 197:experimental_experiment.xoptim.patterns.onnx_any, op_type=Add, name=Opset2
    [IdentityPattern.match] NONE - line: 165:experimental_experiment.xoptim.patterns.onnx_any, op_type=Transpose, name=linear2
    [IdentityPattern.match] NONE - line: 210:experimental_experiment.xoptim.patterns.onnx_any, op_type=Add, name=Opset4
    [GraphBuilder-OXK.make_tensor_input] x[0:None] -- marker=_build_pattern1_x
    [GraphBuilder-OXK.set_type] x:-1
    [GraphBuilder-OXK.make_tensor_input] zero[0:None] -- marker=_build_pattern1_zero
    [GraphBuilder-OXK.set_type] zero:-1
    [GraphBuilder-OXK.make_tensor_input] slope[0:None] -- marker=_build_pattern1_slope
    [GraphBuilder-OXK.set_type] slope:-1
    [GraphBuilder-OXK.make_node] [TT:-] Greater: ['x', 'zero']->['_onx_greater0']
    [GraphBuilder-OXK.set_type] _onx_greater0:9
    [GraphBuilder-OXK.make_node] [TT:-] Mul: ['x', 'slope']->['_onx_mul0']
    [GraphBuilder-OXK.set_type] _onx_mul0:-1
    [GraphBuilder-OXK.make_node] [TTT:-] Where: ['_onx_greater0', 'x', '_onx_mul0']->['_onx_where0']
    [GraphBuilder-OXK.set_type] _onx_where0:-1
    [GraphBuilder-OXK.make_tensor_output] _onx_where0[0: None]
    [GraphBuilder-PVK.make_tensor_input] X[0:None] -- marker=_build_pattern1_X
    [GraphBuilder-PVK.set_type] X:-1
    [GraphBuilder-PVK.make_tensor_input] indices[0:None] -- marker=_build_pattern1_indices
    [GraphBuilder-PVK.set_type] indices:-1
    [GraphBuilder-PVK.make_tensor_input] axis[0:None] -- marker=_build_pattern1_axis
    [GraphBuilder-PVK.set_type] axis:-1
    [GraphBuilder-PVK.make_tensor_input] zerof[0:None] -- marker=_build_pattern1_zerof
    [GraphBuilder-PVK.set_type] zerof:-1
    [GraphBuilder-PVK.make_tensor_input] zeroi[0:None] -- marker=_build_pattern1_zeroi
    [GraphBuilder-PVK.set_type] zeroi:-1
    [GraphBuilder-PVK.make_tensor_input] b[0:None] -- marker=_build_pattern1_b
    [GraphBuilder-PVK.set_type] b:-1
    [GraphBuilder-PVK.make_node] [TT:-] Equal: ['indices', 'b']->['_onx_equal0']
    [GraphBuilder-PVK.set_type] _onx_equal0:9
    [GraphBuilder-PVK.make_node] [T:-] Not: ['_onx_equal0']->['_onx_not0']
    [GraphBuilder-PVK.set_type] _onx_not0:9
    [GraphBuilder-PVK.make_node] [TTT:-] Where: ['_onx_not0', 'indices', 'zeroi']->['_onx_where0']
    [GraphBuilder-PVK.set_type] _onx_where0:-1
    [GraphBuilder-PVK.make_node] [TT:-] Unsqueeze: ['_onx_where0', 'axis']->['_onx_unsqueeze0']
    [GraphBuilder-PVK.set_type] _onx_unsqueeze0:-1
    [GraphBuilder-PVK.make_node] [T:-] LogSoftmax: ['X']->['_onx_logsoftmax0']
    [GraphBuilder-PVK.set_type] _onx_logsoftmax0:-1
    [GraphBuilder-PVK.set_type] _onx_gatherelements0:-1
    [GraphBuilder-PVK.make_node] [TT:T] GatherElements: ['_onx_logsoftmax0', '_onx_unsqueeze0']->['_onx_gatherelements0']
    [GraphBuilder-PVK.set_type] _onx_gatherelements0:-1
    [GraphBuilder-PVK.make_node] [TT:-] Squeeze: ['_onx_gatherelements0', 'axis']->['_onx_squeeze0']
    [GraphBuilder-PVK.set_type] _onx_squeeze0:-1
    [GraphBuilder-PVK.make_node] [T:-] Neg: ['_onx_squeeze0']->['_onx_neg0']
    [GraphBuilder-PVK.set_type] _onx_neg0:-1
    [GraphBuilder-PVK.make_node] [TTT:-] Where: ['_onx_not0', '_onx_neg0', 'zerof']->['_onx_where02']
    [GraphBuilder-PVK.set_type] _onx_where02:-1
    [GraphBuilder-PVK.make_node] [T:-] Cast: ['_onx_not0']->['_onx_cast0']
    [GraphBuilder-PVK.set_type] _onx_cast0:1
    [GraphBuilder-PVK.make_node] [T:-] ReduceSum: ['_onx_cast0']->['_onx_reducesum0']
    [GraphBuilder-PVK.set_type] _onx_reducesum0:1
    [GraphBuilder-PVK.set_shape] _onx_reducesum0:()
    [GraphBuilder-PVK.set_rank] _onx_reducesum0:0
    [GraphBuilder-PVK.make_node] [#:-] Cast: ['_onx_reducesum0']->['_onx_cast02']
    [GraphBuilder-PVK.set_type] _onx_cast02:10
    [GraphBuilder-PVK.set_shape] _onx_cast02:()
    [GraphBuilder-PVK.set_rank] _onx_cast02:0
    [GraphBuilder-PVK.make_node] [T:-] Cast: ['_onx_where02']->['_onx_cast03']
    [GraphBuilder-PVK.set_type] _onx_cast03:1
    [GraphBuilder-PVK.make_node] [T:-] ReduceSum: ['_onx_cast03']->['_onx_reducesum02']
    [GraphBuilder-PVK.set_type] _onx_reducesum02:1
    [GraphBuilder-PVK.set_shape] _onx_reducesum02:()
    [GraphBuilder-PVK.set_rank] _onx_reducesum02:0
    [GraphBuilder-PVK.make_node] [#:-] Cast: ['_onx_reducesum02']->['_onx_cast04']
    [GraphBuilder-PVK.set_type] _onx_cast04:10
    [GraphBuilder-PVK.set_shape] _onx_cast04:()
    [GraphBuilder-PVK.set_rank] _onx_cast04:0
    [GraphBuilder-PVK.make_node] [##:-] Div: ['_onx_cast04', '_onx_cast02']->['_onx_div0']
    [GraphBuilder-PVK.set_type] _onx_div0:10
    [GraphBuilder-PVK.set_shape] _onx_div0:()
    [GraphBuilder-PVK.set_rank] _onx_div0:0
    [GraphBuilder-PVK.make_tensor_output] _onx_div0[0: None]
    [TransposeReshapeTransposePattern.match] NONE - line: 140:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=linear
    [TransposeReshapeTransposePattern.match] NONE - line: 140:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=linear2
    [TransposeTransposePattern.match] NONE - line: 51:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=linear
    [TransposeTransposePattern.match] NONE - line: 51:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=linear2
    [GraphBuilderPatternOptimization.optimize] done all: -0 +0 nodes
    [GraphBuilder.remove_identity_nodes] starts with 7
    [GraphBuilder.remove_identity_nodes] found 0 replacements
    [GraphBuilder.remove_identity_nodes] kept 7 nodes
    [GraphBuilder.remove_identity_nodes] ends with 7 nodes in 3.063800249947235e-05 seconds
    [GraphBuilderPatternOptimization.optimize] increase priority to 1
    [GraphBuilderPatternOptimization.optimize] iteration 1: 7 nodes, priority=1
    [CastCastBinaryPattern.match] NONE - line: 86:experimental_experiment.xoptim.patterns.onnx_cast, op_type=Add, name=Opset2
    [CastCastBinaryPattern.match] NONE - line: 86:experimental_experiment.xoptim.patterns.onnx_cast, op_type=Add, name=Opset4
    [CastOpCastPattern.match] NONE - line: 162:experimental_experiment.xoptim.patterns.onnx_cast, op_type=Add, name=Opset2
    [CastOpCastPattern.match] NONE - line: 159:experimental_experiment.xoptim.patterns.onnx_cast, op_type=Add, name=Opset4
    [ComputationCastOpCastPattern.match] NONE - line: 303:experimental_experiment.xoptim.patterns.onnx_cast, op_type=Add, name=Opset2
    [ComputationCastOpCastPattern.match] NONE - line: 303:experimental_experiment.xoptim.patterns.onnx_cast, op_type=Add, name=Opset4
    [IdentityPattern.match] NONE - line: 165:experimental_experiment.xoptim.patterns.onnx_any, op_type=Transpose, name=linear
    [IdentityPattern.match] NONE - line: 197:experimental_experiment.xoptim.patterns.onnx_any, op_type=Add, name=Opset2
    [IdentityPattern.match] NONE - line: 165:experimental_experiment.xoptim.patterns.onnx_any, op_type=Transpose, name=linear2
    [IdentityPattern.match] NONE - line: 210:experimental_experiment.xoptim.patterns.onnx_any, op_type=Add, name=Opset4
    [ReshapeMatMulReshapePattern.match] NONE - line: 738:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=MatMul, name=Opset
    [ReshapeMatMulReshapePattern.match] NONE - line: 738:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=MatMul, name=Opset3
    [Reshape2Of3Pattern.match] NONE - line: 227:experimental_experiment.xoptim.patterns.onnx_reshape, op_type=Add, name=Opset2
    [Reshape2Of3Pattern.match] NONE - line: 227:experimental_experiment.xoptim.patterns.onnx_reshape, op_type=Add, name=Opset4
    [ReshapeReshapeBinaryPattern.match] NONE - line: 389:experimental_experiment.xoptim.patterns.onnx_reshape, op_type=Add, name=Opset2
    [ReshapeReshapeBinaryPattern.match] NONE - line: 389:experimental_experiment.xoptim.patterns.onnx_reshape, op_type=Add, name=Opset4
    [GraphBuilderPatternOptimization.optimize] match=MatchResult: MatMulAddPattern replaces ['MatMul', 'Add']
    [GraphBuilderPatternOptimization.optimize] match=MatchResult: MatMulAddPattern replaces ['MatMul', 'Add']
    [MatMulReshape2Of3Pattern.match] NONE - line: 373:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=MatMul, name=Opset
    [MatMulReshape2Of3Pattern.match] NONE - line: 373:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=MatMul, name=Opset3
    [MulMulMatMulPattern.match] NONE - line: 674:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=MatMul, name=Opset
    [MulMulMatMulPattern.match] NONE - line: 677:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=MatMul, name=Opset3
    [GraphBuilderPatternOptimization.match] OVERLAP match=MatchResult: TransposeMatMulPattern replaces ['Transpose', 'MatMul'] #marked: 4)
    [GraphBuilderPatternOptimization.match] OVERLAP match=MatchResult: TransposeMatMulPattern replaces ['Transpose', 'MatMul'] #marked: 4)
    [TransposeReshapeMatMulPattern.match] NONE - line: 991:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=MatMul, name=Opset
    [TransposeReshapeMatMulPattern.match] NONE - line: 991:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=MatMul, name=Opset3
    [TransposeReshapeTransposePattern.match] NONE - line: 140:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=linear
    [TransposeReshapeTransposePattern.match] NONE - line: 140:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=linear2
    [TransposeTransposePattern.match] NONE - line: 51:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=linear
    [TransposeTransposePattern.match] NONE - line: 51:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=linear2
    [GraphBuilderPatternOptimization.optimize] applies 2 matches, 2*MatMulAddPattern - time=0.001 | max_time=IdentityPattern:0.000
    [GraphBuilderPatternOptimization.optimize] apply MatchResult: MatMulAddPattern replaces ['MatMul', 'Add'], inputs: {'x', '_onx_transpose0', 'layers.0.bias', '_onx_matmul0'}, outputs: {'linear', '_onx_matmul0'}
    [GraphBuilderPatternOptimization.apply_match] MatchResult: MatMulAddPattern replaces ['MatMul', 'Add']
      - MatMul: ['x', '_onx_transpose0'] -> ['_onx_matmul0']
      - Add: ['_onx_matmul0', 'layers.0.bias'] -> ['linear']
      + Gemm: ['x', '_onx_transpose0', 'layers.0.bias'] -> ['linear']
    [GraphBuilder-TBC.set_type] linear:1
    [GraphBuilderPatternOptimization.apply_match] MatchResult: MatMulAddPattern replaces ['MatMul', 'Add'] applied.
    [GraphBuilderPatternOptimization.optimize] - add ['Gemm']
    [GraphBuilderPatternOptimization.optimize] done MatchResult: MatMulAddPattern replaces ['MatMul', 'Add']: -2 +1 nodes
    [GraphBuilderPatternOptimization.optimize] removed outputs {'_onx_matmul0'}
    [GraphBuilderPatternOptimization.optimize] apply MatchResult: MatMulAddPattern replaces ['MatMul', 'Add'], inputs: {'_onx_transpose02', '_onx_matmul02', 'relu', 'layers.2.bias'}, outputs: {'_onx_matmul02', 'output_0'}
    [GraphBuilderPatternOptimization.apply_match] MatchResult: MatMulAddPattern replaces ['MatMul', 'Add']
      - MatMul: ['relu', '_onx_transpose02'] -> ['_onx_matmul02']
      - Add: ['_onx_matmul02', 'layers.2.bias'] -> ['output_0']
      + Gemm: ['relu', '_onx_transpose02', 'layers.2.bias'] -> ['output_0']
    [GraphBuilder-TBC.set_type] output_0:1
    [GraphBuilderPatternOptimization.apply_match] MatchResult: MatMulAddPattern replaces ['MatMul', 'Add'] applied.
    [GraphBuilderPatternOptimization.optimize] - add ['Gemm']
    [GraphBuilderPatternOptimization.optimize] done MatchResult: MatMulAddPattern replaces ['MatMul', 'Add']: -2 +1 nodes
    [GraphBuilderPatternOptimization.optimize] removed outputs {'_onx_matmul02'}
    [GraphBuilderPatternOptimization.optimize] done all: -4 +2 nodes
    [GraphBuilder.remove_identity_nodes] starts with 5
    [GraphBuilder.remove_identity_nodes] found 0 replacements
    [GraphBuilder.remove_identity_nodes] kept 5 nodes
    [GraphBuilder.remove_identity_nodes] ends with 5 nodes in 1.9203998817829415e-05 seconds
    [GraphBuilderPatternOptimization.optimize] iteration 2: 5 nodes, priority=1
    [IdentityPattern.match] NONE - line: 165:experimental_experiment.xoptim.patterns.onnx_any, op_type=Transpose, name=linear
    [IdentityPattern.match] NONE - line: 165:experimental_experiment.xoptim.patterns.onnx_any, op_type=Transpose, name=linear2
    [MatMulAddPattern.match] NONE - line: 48:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=Gemm, name=MatMulAddPattern--Opset
    [MatMulAddPattern.match] NONE - line: 45:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=Gemm, name=MatMulAddPattern--Opset3
    [GraphBuilderPatternOptimization.optimize] match=MatchResult: GemmTransposePattern replaces ['Gemm']
    [GraphBuilderPatternOptimization.optimize] match=MatchResult: GemmTransposePattern replaces ['Gemm']
    [GraphBuilderPatternOptimization.match] OVERLAP match=MatchResult: TransposeMatMulPattern replaces ['Transpose', 'Gemm'] #marked: 2)
    [GraphBuilderPatternOptimization.match] OVERLAP match=MatchResult: TransposeMatMulPattern replaces ['Transpose', 'Gemm'] #marked: 2)
    [TransposeReshapeTransposePattern.match] NONE - line: 140:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=linear
    [TransposeReshapeTransposePattern.match] NONE - line: 140:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=linear2
    [TransposeTransposePattern.match] NONE - line: 51:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=linear
    [TransposeTransposePattern.match] NONE - line: 51:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=linear2
    [GraphBuilderPatternOptimization.optimize] applies 2 matches, 2*GemmTransposePattern - time=0.000 | max_time=TransposeMatMulPattern:0.000
    [GraphBuilderPatternOptimization.optimize] apply MatchResult: GemmTransposePattern replaces ['Gemm'], inputs: {'x', '_onx_transpose0', 'layers.0.bias'}, outputs: {'linear'}
    [GraphBuilder.update_node_constant] new constant 'GemmTransposePattern--_onx_transpose0', node=Transpose
    [GraphBuilderPatternOptimization.apply_match] MatchResult: GemmTransposePattern replaces ['Gemm']
      - Gemm: ['x', '_onx_transpose0', 'layers.0.bias'] -> ['linear']
      + Transpose: ['_onx_transpose0'] -> ['GemmTransposePattern--_onx_transpose0']
      + Gemm: ['x', 'GemmTransposePattern--_onx_transpose0', 'layers.0.bias'] -> ['linear']
    [GraphBuilder.update_node_constant] new constant 'GemmTransposePattern--_onx_transpose0', node=Transpose
    [GraphBuilder-TBC.set_type] GemmTransposePattern--_onx_transpose0:1
    [GraphBuilder-TBC.set_shape] GemmTransposePattern--_onx_transpose0:(32, 10)
    [GraphBuilder-TBC.set_rank] GemmTransposePattern--_onx_transpose0:2
    [GraphBuilder-TBC.set_type] linear:1
    [GraphBuilderPatternOptimization.apply_match] MatchResult: GemmTransposePattern replaces ['Gemm'] applied.
    [GraphBuilderPatternOptimization.optimize] - add ['Transpose', 'Gemm']
    [GraphBuilderPatternOptimization.optimize] done MatchResult: GemmTransposePattern replaces ['Gemm']: -1 +2 nodes
    [GraphBuilderPatternOptimization.optimize] apply MatchResult: GemmTransposePattern replaces ['Gemm'], inputs: {'_onx_transpose02', 'relu', 'layers.2.bias'}, outputs: {'output_0'}
    [GraphBuilder.update_node_constant] new constant 'GemmTransposePattern--_onx_transpose02', node=Transpose
    [GraphBuilderPatternOptimization.apply_match] MatchResult: GemmTransposePattern replaces ['Gemm']
      - Gemm: ['relu', '_onx_transpose02', 'layers.2.bias'] -> ['output_0']
      + Transpose: ['_onx_transpose02'] -> ['GemmTransposePattern--_onx_transpose02']
      + Gemm: ['relu', 'GemmTransposePattern--_onx_transpose02', 'layers.2.bias'] -> ['output_0']
    [GraphBuilder.update_node_constant] new constant 'GemmTransposePattern--_onx_transpose02', node=Transpose
    [GraphBuilder-TBC.set_type] GemmTransposePattern--_onx_transpose02:1
    [GraphBuilder-TBC.set_shape] GemmTransposePattern--_onx_transpose02:(1, 32)
    [GraphBuilder-TBC.set_rank] GemmTransposePattern--_onx_transpose02:2
    [GraphBuilder-TBC.set_type] output_0:1
    [GraphBuilderPatternOptimization.apply_match] MatchResult: GemmTransposePattern replaces ['Gemm'] applied.
    [GraphBuilderPatternOptimization.optimize] - add ['Transpose', 'Gemm']
    [GraphBuilderPatternOptimization.optimize] done MatchResult: GemmTransposePattern replaces ['Gemm']: -1 +2 nodes
    [GraphBuilderPatternOptimization.optimize] done all: -2 +4 nodes
    [GraphBuilder.remove_identity_nodes] starts with 7
    [GraphBuilder.remove_identity_nodes] found 0 replacements
    [GraphBuilder.remove_identity_nodes] kept 7 nodes
    [GraphBuilder.remove_identity_nodes] ends with 7 nodes in 2.3068998416420072e-05 seconds
    [GraphBuilderPatternOptimization.optimize] iteration 3: 7 nodes, priority=1
    [IdentityPattern.match] NONE - line: 165:experimental_experiment.xoptim.patterns.onnx_any, op_type=Transpose, name=linear
    [IdentityPattern.match] NONE - line: 165:experimental_experiment.xoptim.patterns.onnx_any, op_type=Transpose, name=GemmTransposePattern--MatMulAddPattern--Opset
    [IdentityPattern.match] NONE - line: 165:experimental_experiment.xoptim.patterns.onnx_any, op_type=Transpose, name=linear2
    [IdentityPattern.match] NONE - line: 165:experimental_experiment.xoptim.patterns.onnx_any, op_type=Transpose, name=GemmTransposePattern--MatMulAddPattern--Opset3
    [MatMulAddPattern.match] NONE - line: 48:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=Gemm, name=GemmTransposePattern--MatMulAddPattern--Opset2
    [MatMulAddPattern.match] NONE - line: 45:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=Gemm, name=GemmTransposePattern--MatMulAddPattern--Opset32
    [GemmTransposePattern.match] NONE - line: 284:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=Gemm, name=GemmTransposePattern--MatMulAddPattern--Opset2
    [GemmTransposePattern.match] NONE - line: 284:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=Gemm, name=GemmTransposePattern--MatMulAddPattern--Opset32
    [TransposeMatMulPattern.match] NONE - line: 886:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=Gemm, name=GemmTransposePattern--MatMulAddPattern--Opset2
    [TransposeMatMulPattern.match] NONE - line: 886:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=Gemm, name=GemmTransposePattern--MatMulAddPattern--Opset32
    [TransposeReshapeTransposePattern.match] NONE - line: 140:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=linear
    [TransposeReshapeTransposePattern.match] NONE - line: 140:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=GemmTransposePattern--MatMulAddPattern--Opset
    [TransposeReshapeTransposePattern.match] NONE - line: 140:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=linear2
    [TransposeReshapeTransposePattern.match] NONE - line: 140:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=GemmTransposePattern--MatMulAddPattern--Opset3
    [GraphBuilderPatternOptimization.optimize] match=MatchResult: TransposeTransposePattern replaces ['Transpose', 'Transpose']
    [TransposeTransposePattern.match] NONE - line: 51:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=GemmTransposePattern--MatMulAddPattern--Opset
    [GraphBuilderPatternOptimization.optimize] match=MatchResult: TransposeTransposePattern replaces ['Transpose', 'Transpose']
    [TransposeTransposePattern.match] NONE - line: 51:experimental_experiment.xoptim.patterns.onnx_transpose, op_type=Transpose, name=GemmTransposePattern--MatMulAddPattern--Opset3
    [GraphBuilderPatternOptimization.optimize] applies 2 matches, 2*TransposeTransposePattern - time=0.000 | max_time=TransposeTransposePattern:0.000
    [GraphBuilderPatternOptimization.optimize] apply MatchResult: TransposeTransposePattern replaces ['Transpose', 'Transpose'], inputs: {'_onx_transpose0', 'layers.0.weight'}, outputs: {'_onx_transpose0', 'GemmTransposePattern--_onx_transpose0'}
    [GraphBuilder.update_node_constant] new constant 'GemmTransposePattern--_onx_transpose0', node=Identity
    [GraphBuilderPatternOptimization.apply_match] MatchResult: TransposeTransposePattern replaces ['Transpose', 'Transpose']
      - Transpose: ['layers.0.weight'] -> ['_onx_transpose0']
      - Transpose: ['_onx_transpose0'] -> ['GemmTransposePattern--_onx_transpose0']
      + Identity: ['layers.0.weight'] -> ['GemmTransposePattern--_onx_transpose0']
    [GraphBuilder.update_node_constant] new constant 'GemmTransposePattern--_onx_transpose0', node=Identity
    [GraphBuilder-TBC.set_type] GemmTransposePattern--_onx_transpose0:1
    [GraphBuilder.update_node_constant] new constant 'GemmTransposePattern--_onx_transpose0', node=Identity
    [GraphBuilderPatternOptimization.apply_match] MatchResult: TransposeTransposePattern replaces ['Transpose', 'Transpose'] applied.
    [GraphBuilderPatternOptimization.optimize] - add ['Identity']
    [GraphBuilderPatternOptimization.optimize] done MatchResult: TransposeTransposePattern replaces ['Transpose', 'Transpose']: -2 +1 nodes
    [GraphBuilderPatternOptimization.optimize] removed outputs {'_onx_transpose0'}
    [GraphBuilderPatternOptimization.optimize] apply MatchResult: TransposeTransposePattern replaces ['Transpose', 'Transpose'], inputs: {'_onx_transpose02', 'layers.2.weight'}, outputs: {'_onx_transpose02', 'GemmTransposePattern--_onx_transpose02'}
    [GraphBuilder.update_node_constant] new constant 'GemmTransposePattern--_onx_transpose02', node=Identity
    [GraphBuilderPatternOptimization.apply_match] MatchResult: TransposeTransposePattern replaces ['Transpose', 'Transpose']
      - Transpose: ['layers.2.weight'] -> ['_onx_transpose02']
      - Transpose: ['_onx_transpose02'] -> ['GemmTransposePattern--_onx_transpose02']
      + Identity: ['layers.2.weight'] -> ['GemmTransposePattern--_onx_transpose02']
    [GraphBuilder.update_node_constant] new constant 'GemmTransposePattern--_onx_transpose02', node=Identity
    [GraphBuilder-TBC.set_type] GemmTransposePattern--_onx_transpose02:1
    [GraphBuilder.update_node_constant] new constant 'GemmTransposePattern--_onx_transpose02', node=Identity
    [GraphBuilderPatternOptimization.apply_match] MatchResult: TransposeTransposePattern replaces ['Transpose', 'Transpose'] applied.
    [GraphBuilderPatternOptimization.optimize] - add ['Identity']
    [GraphBuilderPatternOptimization.optimize] done MatchResult: TransposeTransposePattern replaces ['Transpose', 'Transpose']: -2 +1 nodes
    [GraphBuilderPatternOptimization.optimize] removed outputs {'_onx_transpose02'}
    [GraphBuilderPatternOptimization.optimize] done all: -4 +2 nodes
    [GraphBuilder.remove_identity_nodes] starts with 5
    [GraphBuilder.remove_identity_nodes] found 2 replacements
    [GraphBuilder.remove_identity_nodes] kept 3 nodes
    [GraphBuilder.remove_identity_nodes] node Gemm-GemmTransposePattern--MatMulAddPattern--Opset2:['x', 'GemmTransposePattern--_onx_transpose0', 'layers.0.bias']->['x', 'layers.0.weight', 'layers.0.bias']:['linear']->['linear']
    [GraphBuilder.remove_identity_nodes] node Gemm-GemmTransposePattern--MatMulAddPattern--Opset32:['relu', 'GemmTransposePattern--_onx_transpose02', 'layers.2.bias']->['relu', 'layers.2.weight', 'layers.2.bias']:['output_0']->['output_0']
    [GraphBuilder.remove_identity_nodes] ends with 3 nodes in 6.21190010861028e-05 seconds
    [GraphBuilderPatternOptimization.optimize] iteration 4: 3 nodes, priority=1
    [MatMulAddPattern.match] NONE - line: 48:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=Gemm, name=GemmTransposePattern--MatMulAddPattern--Opset2
    [MatMulAddPattern.match] NONE - line: 45:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=Gemm, name=GemmTransposePattern--MatMulAddPattern--Opset32
    [GemmTransposePattern.match] NONE - line: 284:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=Gemm, name=GemmTransposePattern--MatMulAddPattern--Opset2
    [GemmTransposePattern.match] NONE - line: 284:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=Gemm, name=GemmTransposePattern--MatMulAddPattern--Opset32
    [TransposeMatMulPattern.match] NONE - line: 848:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=Gemm, name=GemmTransposePattern--MatMulAddPattern--Opset2
    [TransposeMatMulPattern.match] NONE - line: 848:experimental_experiment.xoptim.patterns.onnx_matmul, op_type=Gemm, name=GemmTransposePattern--MatMulAddPattern--Opset32
    [GraphBuilderPatternOptimization.optimize] done all: -0 +0 nodes
    [GraphBuilderPatternOptimization.optimize] done after 5 iterations with 3 nodes in 0.006
        STAT apply_GemmTransposePattern +4 -2 #it=1 maxmatch=1 i=2 - time=0.0003327369995531626
        STAT apply_MatMulAddPattern +2 -4 #it=1 maxmatch=1 i=2 - time=0.00020908800070174038
        STAT apply_TransposeTransposePattern +2 -4 #it=1 maxmatch=1 i=2 - time=0.00022536699907504953
        STAT build_graph_for_pattern +0 -0 #it=5 maxmatch=0 i=0 - time=0.00014327100143418647
        STAT check_pattern_00 +0 -0 #it=1 maxmatch=0 i=0 - time=1.605400029802695e-05
        STAT check_pattern_A0 +0 -0 #it=3 maxmatch=0 i=0 - time=8.254599379142746e-05
        STAT check_pattern_B0 +0 -0 #it=4 maxmatch=0 i=0 - time=5.0224000005982816e-05
        STAT match_BatchNormalizationPattern +0 -0 #it=5 maxmatch=0 i=0 - time=4.306400296627544e-05
        STAT match_BatchNormalizationTrainingPattern +0 -0 #it=5 maxmatch=0 i=0 - time=2.4255001335404813e-05
        STAT match_CastCastBinaryPattern +0 -0 #it=4 maxmatch=0 i=0 - time=4.104700201423839e-05
        STAT match_CastLayerNormalizationCastPattern +0 -0 #it=4 maxmatch=0 i=0 - time=2.0181003492325544e-05
        STAT match_CastOpCastPattern +0 -0 #it=4 maxmatch=0 i=0 - time=3.865600228891708e-05
        STAT match_CastPattern +0 -0 #it=5 maxmatch=0 i=0 - time=2.383299943176098e-05
        STAT match_ClipClipPattern +0 -0 #it=4 maxmatch=0 i=0 - time=1.9169998267898336e-05
        STAT match_ComputationCastOpCastPattern +0 -0 #it=4 maxmatch=0 i=0 - time=0.0001033730004564859
        STAT match_ConvBiasNullPattern +0 -0 #it=5 maxmatch=0 i=0 - time=2.7209000108996406e-05
        STAT match_DropoutPattern +0 -0 #it=4 maxmatch=0 i=0 - time=1.6662004782119766e-05
        STAT match_ExpandBroadcastPattern +0 -0 #it=4 maxmatch=0 i=0 - time=1.6551002772757784e-05
        STAT match_ExpandPattern +0 -0 #it=5 maxmatch=0 i=0 - time=2.3043998226057738e-05
        STAT match_ExpandSwapPattern +0 -0 #it=4 maxmatch=0 i=0 - time=1.6948000848060474e-05
        STAT match_GeluPattern +0 -0 #it=5 maxmatch=0 i=0 - time=3.5999983083456755e-06
        STAT match_GemmTransposePattern +0 -0 #it=4 maxmatch=2 i=2 - time=7.471699427696876e-05
        STAT match_IdentityPattern +0 -0 #it=5 maxmatch=0 i=0 - time=0.00042024600043077953
        STAT match_LayerNormalizationPattern +0 -0 #it=4 maxmatch=0 i=0 - time=2.0667001081164926e-05
        STAT match_LayerNormalizationScalePattern +0 -0 #it=4 maxmatch=0 i=0 - time=1.8404000002192333e-05
        STAT match_LeakyReluPattern +0 -0 #it=5 maxmatch=0 i=0 - time=0.0007750050026515964
        STAT match_MatMulAddPattern +0 -0 #it=4 maxmatch=2 i=2 - time=0.00011396699846955016
        STAT match_MatMulReshape2Of3Pattern +0 -0 #it=4 maxmatch=2 i=0 - time=3.724699854501523e-05
        STAT match_MulMulMatMulPattern +0 -0 #it=4 maxmatch=2 i=0 - time=3.2002004445530474e-05
        STAT match_MulMulMulScalarPattern +0 -0 #it=4 maxmatch=0 i=0 - time=2.10210018849466e-05
        STAT match_ReduceReshapePattern +0 -0 #it=4 maxmatch=0 i=0 - time=2.295699596288614e-05
        STAT match_ReduceSumNormalizePattern +0 -0 #it=4 maxmatch=0 i=0 - time=1.929100471897982e-05
        STAT match_Reshape2Of3Pattern +0 -0 #it=4 maxmatch=0 i=0 - time=4.199899922241457e-05
        STAT match_ReshapeMatMulReshapePattern +0 -0 #it=4 maxmatch=0 i=0 - time=3.7881000025663525e-05
        STAT match_ReshapePattern +0 -0 #it=5 maxmatch=0 i=0 - time=2.580499858595431e-05
        STAT match_ReshapeReshapeBinaryPattern +0 -0 #it=4 maxmatch=0 i=0 - time=3.5246001061750576e-05
        STAT match_ReshapeReshapePattern +0 -0 #it=5 maxmatch=2 i=0 - time=2.27799937420059e-05
        STAT match_RotaryConcatPartPattern +0 -0 #it=4 maxmatch=2 i=0 - time=1.9571001757867634e-05
        STAT match_SameChildrenPattern +0 -0 #it=5 maxmatch=2 i=0 - time=4.7129997255979106e-05
        STAT match_SequenceConstructAtPattern +0 -0 #it=4 maxmatch=2 i=0 - time=1.7833997844718397e-05
        STAT match_SliceSlicePattern +0 -0 #it=4 maxmatch=2 i=0 - time=1.87539953913074e-05
        STAT match_SlicesSplitPattern +0 -0 #it=4 maxmatch=2 i=0 - time=1.723600144032389e-05
        STAT match_SoftmaxCrossEntropyLossCastPattern +0 -0 #it=5 maxmatch=2 i=0 - time=0.0018088959986926056
        STAT match_Sub1MulPattern +0 -0 #it=4 maxmatch=2 i=0 - time=1.909900311147794e-05
        STAT match_SwitchOrderBinaryPattern +0 -0 #it=4 maxmatch=2 i=0 - time=2.3594995582243428e-05
        STAT match_TransposeMatMulPattern +0 -0 #it=4 maxmatch=2 i=0 - time=0.00014759800251340494
        STAT match_TransposeReshapeMatMulPattern +0 -0 #it=4 maxmatch=2 i=0 - time=3.168400144204497e-05
        STAT match_TransposeReshapeTransposePattern +0 -0 #it=5 maxmatch=2 i=0 - time=8.608399730292149e-05
        STAT match_TransposeTransposePattern +0 -0 #it=5 maxmatch=2 i=2 - time=9.404599768458866e-05
        STAT match_UnsqueezeEqualPattern +0 -0 #it=4 maxmatch=2 i=0 - time=1.855400114436634e-05
        STAT match_UnsqueezeUnsqueezePattern +0 -0 #it=5 maxmatch=2 i=0 - time=2.4615997972432524e-05
        STAT remove_identity_nodes +2 -4 #it=4 maxmatch=0 i=0 - time=0.00019398500080569647
    --MODEL: 3 nodes, 1 inputs, 1 outputs, 4 initializers--
         INPUT:   1 x 1t
        OUTPUT:   1 x 1t
          INIT:   4 x 1t
          NODE:   2 x Gemm
          NODE:   1 x Relu
    --MODEL: 3 nodes, 1 inputs, 1 outputs, 4 initializers--DETAILED--
         INPUT:   1 x 1t[3x10]
        OUTPUT:   1 x 1t[3x1]
          INIT:   1 x 1t[1]
          INIT:   1 x 1t[1x32]
          INIT:   1 x 1t[32]
          INIT:   1 x 1t[32x10]
          NODE:   1 x Gemm -SIG- 1t[3x10], 1t[32x10], 1t[32]
          NODE:   1 x Gemm -SIG- 1t[3x32], 1t[1x32], 1t[1]
          NODE:   1 x Relu -SIG- 1t[3x32]
    [GraphBuilder.optimize] done with 3 nodes in 0.008
        STAT apply_GemmTransposePattern +4 -2 #it=1 maxmatch=1 i=2 - time=0.0003327369995531626
        STAT apply_MatMulAddPattern +2 -4 #it=1 maxmatch=1 i=2 - time=0.00020908800070174038
        STAT apply_TransposeTransposePattern +2 -4 #it=1 maxmatch=1 i=2 - time=0.00022536699907504953
        STAT build_graph_for_pattern +0 -0 #it=5 maxmatch=0 i=0 - time=0.00014327100143418647
        STAT check_A +0 -0 #it=0 maxmatch=0 i=0 - time=2.4860997655196115e-05
        STAT check_B +0 -0 #it=0 maxmatch=0 i=0 - time=1.4296001609181985e-05
        STAT check_C +0 -0 #it=0 maxmatch=0 i=0 - time=1.2853997759521008e-05
        STAT check_F +0 -0 #it=0 maxmatch=0 i=0 - time=1.466499816160649e-05
        STAT check_G +0 -0 #it=0 maxmatch=0 i=0 - time=9.174997103400528e-06
        STAT check_pattern_00 +0 -0 #it=1 maxmatch=0 i=0 - time=1.605400029802695e-05
        STAT check_pattern_A0 +0 -0 #it=3 maxmatch=0 i=0 - time=8.254599379142746e-05
        STAT check_pattern_B0 +0 -0 #it=4 maxmatch=0 i=0 - time=5.0224000005982816e-05
        STAT match_BatchNormalizationPattern +0 -0 #it=5 maxmatch=0 i=0 - time=4.306400296627544e-05
        STAT match_BatchNormalizationTrainingPattern +0 -0 #it=5 maxmatch=0 i=0 - time=2.4255001335404813e-05
        STAT match_CastCastBinaryPattern +0 -0 #it=4 maxmatch=0 i=0 - time=4.104700201423839e-05
        STAT match_CastLayerNormalizationCastPattern +0 -0 #it=4 maxmatch=0 i=0 - time=2.0181003492325544e-05
        STAT match_CastOpCastPattern +0 -0 #it=4 maxmatch=0 i=0 - time=3.865600228891708e-05
        STAT match_CastPattern +0 -0 #it=5 maxmatch=0 i=0 - time=2.383299943176098e-05
        STAT match_ClipClipPattern +0 -0 #it=4 maxmatch=0 i=0 - time=1.9169998267898336e-05
        STAT match_ComputationCastOpCastPattern +0 -0 #it=4 maxmatch=0 i=0 - time=0.0001033730004564859
        STAT match_ConvBiasNullPattern +0 -0 #it=5 maxmatch=0 i=0 - time=2.7209000108996406e-05
        STAT match_DropoutPattern +0 -0 #it=4 maxmatch=0 i=0 - time=1.6662004782119766e-05
        STAT match_ExpandBroadcastPattern +0 -0 #it=4 maxmatch=0 i=0 - time=1.6551002772757784e-05
        STAT match_ExpandPattern +0 -0 #it=5 maxmatch=0 i=0 - time=2.3043998226057738e-05
        STAT match_ExpandSwapPattern +0 -0 #it=4 maxmatch=0 i=0 - time=1.6948000848060474e-05
        STAT match_GeluPattern +0 -0 #it=5 maxmatch=0 i=0 - time=3.5999983083456755e-06
        STAT match_GemmTransposePattern +0 -0 #it=4 maxmatch=2 i=2 - time=7.471699427696876e-05
        STAT match_IdentityPattern +0 -0 #it=5 maxmatch=0 i=0 - time=0.00042024600043077953
        STAT match_LayerNormalizationPattern +0 -0 #it=4 maxmatch=0 i=0 - time=2.0667001081164926e-05
        STAT match_LayerNormalizationScalePattern +0 -0 #it=4 maxmatch=0 i=0 - time=1.8404000002192333e-05
        STAT match_LeakyReluPattern +0 -0 #it=5 maxmatch=0 i=0 - time=0.0007750050026515964
        STAT match_MatMulAddPattern +0 -0 #it=4 maxmatch=2 i=2 - time=0.00011396699846955016
        STAT match_MatMulReshape2Of3Pattern +0 -0 #it=4 maxmatch=2 i=0 - time=3.724699854501523e-05
        STAT match_MulMulMatMulPattern +0 -0 #it=4 maxmatch=2 i=0 - time=3.2002004445530474e-05
        STAT match_MulMulMulScalarPattern +0 -0 #it=4 maxmatch=0 i=0 - time=2.10210018849466e-05
        STAT match_ReduceReshapePattern +0 -0 #it=4 maxmatch=0 i=0 - time=2.295699596288614e-05
        STAT match_ReduceSumNormalizePattern +0 -0 #it=4 maxmatch=0 i=0 - time=1.929100471897982e-05
        STAT match_Reshape2Of3Pattern +0 -0 #it=4 maxmatch=0 i=0 - time=4.199899922241457e-05
        STAT match_ReshapeMatMulReshapePattern +0 -0 #it=4 maxmatch=0 i=0 - time=3.7881000025663525e-05
        STAT match_ReshapePattern +0 -0 #it=5 maxmatch=0 i=0 - time=2.580499858595431e-05
        STAT match_ReshapeReshapeBinaryPattern +0 -0 #it=4 maxmatch=0 i=0 - time=3.5246001061750576e-05
        STAT match_ReshapeReshapePattern +0 -0 #it=5 maxmatch=2 i=0 - time=2.27799937420059e-05
        STAT match_RotaryConcatPartPattern +0 -0 #it=4 maxmatch=2 i=0 - time=1.9571001757867634e-05
        STAT match_SameChildrenPattern +0 -0 #it=5 maxmatch=2 i=0 - time=4.7129997255979106e-05
        STAT match_SequenceConstructAtPattern +0 -0 #it=4 maxmatch=2 i=0 - time=1.7833997844718397e-05
        STAT match_SliceSlicePattern +0 -0 #it=4 maxmatch=2 i=0 - time=1.87539953913074e-05
        STAT match_SlicesSplitPattern +0 -0 #it=4 maxmatch=2 i=0 - time=1.723600144032389e-05
        STAT match_SoftmaxCrossEntropyLossCastPattern +0 -0 #it=5 maxmatch=2 i=0 - time=0.0018088959986926056
        STAT match_Sub1MulPattern +0 -0 #it=4 maxmatch=2 i=0 - time=1.909900311147794e-05
        STAT match_SwitchOrderBinaryPattern +0 -0 #it=4 maxmatch=2 i=0 - time=2.3594995582243428e-05
        STAT match_TransposeMatMulPattern +0 -0 #it=4 maxmatch=2 i=0 - time=0.00014759800251340494
        STAT match_TransposeReshapeMatMulPattern +0 -0 #it=4 maxmatch=2 i=0 - time=3.168400144204497e-05
        STAT match_TransposeReshapeTransposePattern +0 -0 #it=5 maxmatch=2 i=0 - time=8.608399730292149e-05
        STAT match_TransposeTransposePattern +0 -0 #it=5 maxmatch=2 i=2 - time=9.404599768458866e-05
        STAT match_UnsqueezeEqualPattern +0 -0 #it=4 maxmatch=2 i=0 - time=1.855400114436634e-05
        STAT match_UnsqueezeUnsqueezePattern +0 -0 #it=5 maxmatch=2 i=0 - time=2.4615997972432524e-05
        STAT pattern_optimization +0 -4 #it=0 maxmatch=0 i=0 - time=0.007263799998327158
        STAT remove_identity_nodes +2 -4 #it=4 maxmatch=0 i=0 - time=0.00023608799892826937
        STAT remove_unused +0 -0 #it=0 maxmatch=0 i=0 - time=7.145300332922488e-05
    --MODEL: 3 nodes, 1 inputs, 1 outputs, 4 initializers--
         INPUT:   1 x 1t
        OUTPUT:   1 x 1t
          INIT:   4 x 1t
          NODE:   2 x Gemm
          NODE:   1 x Relu
    --MODEL: 3 nodes, 1 inputs, 1 outputs, 4 initializers--DETAILED--
         INPUT:   1 x 1t[3x10]
        OUTPUT:   1 x 1t[3x1]
          INIT:   1 x 1t[1]
          INIT:   1 x 1t[1x32]
          INIT:   1 x 1t[32]
          INIT:   1 x 1t[32x10]
          NODE:   1 x Gemm -SIG- 1t[3x10], 1t[32x10], 1t[32]
          NODE:   1 x Gemm -SIG- 1t[3x32], 1t[1x32], 1t[1]
          NODE:   1 x Relu -SIG- 1t[3x32]
    [GraphBuilder-TBC.to_onnx] make_model 4 inits 0 params
    [GraphBuilder-TBC.time_evaluation_constants_] 0
    [GraphBuilder-TBC._build_initializers] start with 4 initializers, large_model=False, external_threshold=1024
    [GraphBuilder-TBC._build_initializers] switch low/high order
    [GraphBuilder-TBC._build_initializers] TensorProto-layers.0.weight:1[(32, 10)]
    [GraphBuilder-TBC._build_initializers] TensorProto-layers.0.bias:1[(32,)]
    [GraphBuilder-TBC._build_initializers] TensorProto-layers.2.weight:1[(1, 32)]
    [GraphBuilder-TBC._build_initializers] TensorProto-layers.2.bias:1[(1,)]
    [GraphBuilder-TBC._build_initializers] done in 8.729984983801842e-07s with 4 initializers, 0 large initializers

Select the pattern to use¶

Class OptimizationOptions is used to enable or disable patterns.

<<<

import onnx
from experimental_experiment.xbuilder import GraphBuilder, OptimizationOptions

onx = onnx.load("temp_doc_mlp.onnx")

gr = GraphBuilder(
    onx,
    infer_shapes=True,
    optimization_options=OptimizationOptions(
        patterns="TransposeTranspose,TransposeMatMul", verbose=1
    ),
)
opt_onx = gr.to_onnx(optimize=True)

>>>

    [GraphBuilder.optimize] start with 7 nodes
    [GraphBuilder.optimize] #patterns=2
    [GraphBuilderPatternOptimization.optimize] start with 7 nodes, 4 initializers, 2 patterns, priorities=[0, 1]
    [GraphBuilderPatternOptimization.optimize] iteration 0: 7 nodes, priority=0
    [GraphBuilderPatternOptimization.optimize] increase priority to 1
    [GraphBuilderPatternOptimization.optimize] iteration 1: 7 nodes, priority=1
    [GraphBuilderPatternOptimization.optimize] applies 2 matches, 2*TransposeMatMulPattern - time=0.000 | max_time=TransposeMatMulPattern:0.000
    [GraphBuilderPatternOptimization.optimize] iteration 2: 5 nodes, priority=1
    [GraphBuilderPatternOptimization.optimize] done after 3 iterations with 5 nodes in 0.001
    [GraphBuilder.optimize] done with 5 nodes in 0.001

There exists some predefined lists of patterns:

default: includes all patterns using only standard onnx patterns.
onnxruntime: patterns specific to onnxruntime, the final model may be executed by onnxruntime and possibly only onnxruntime as it may introduce patterns from Supported Operators and Data Types.

<<<

import onnx
from experimental_experiment.xbuilder import GraphBuilder, OptimizationOptions

onx = onnx.load("temp_doc_mlp.onnx")

gr = GraphBuilder(
    onx,
    infer_shapes=True,
    optimization_options=OptimizationOptions(patterns="default+onnxruntime", verbose=1),
)
opt_onx = gr.to_onnx(optimize=True)

>>>

    [GraphBuilder.optimize] start with 7 nodes
    [GraphBuilder.optimize] #patterns=56
    [GraphBuilderPatternOptimization.optimize] start with 7 nodes, 4 initializers, 56 patterns, priorities=[0, 1, 2, 3]
    [GraphBuilderPatternOptimization.optimize] iteration 0: 7 nodes, priority=0
    [GraphBuilderPatternOptimization.optimize] increase priority to 1
    [GraphBuilderPatternOptimization.optimize] iteration 1: 7 nodes, priority=1
    [GraphBuilderPatternOptimization.optimize] applies 2 matches, 2*MatMulAddPattern - time=0.001 | max_time=IdentityPattern:0.000
    [GraphBuilderPatternOptimization.optimize] iteration 2: 5 nodes, priority=1
    [GraphBuilderPatternOptimization.optimize] applies 2 matches, 2*GemmTransposePattern - time=0.000 | max_time=TransposeMatMulPattern:0.000
    [GraphBuilderPatternOptimization.optimize] iteration 3: 7 nodes, priority=1
    [GraphBuilderPatternOptimization.optimize] applies 2 matches, 2*TransposeTransposePattern - time=0.000 | max_time=TransposeTransposePattern:0.000
    [GraphBuilderPatternOptimization.optimize] iteration 4: 3 nodes, priority=1
    [GraphBuilderPatternOptimization.optimize] increase priority to 2
    [GraphBuilderPatternOptimization.optimize] iteration 5: 3 nodes, priority=2
    [GraphBuilderPatternOptimization.optimize] increase priority to 3
    [GraphBuilderPatternOptimization.optimize] iteration 6: 3 nodes, priority=3
    [GraphBuilderPatternOptimization.optimize] done after 7 iterations with 3 nodes in 0.008
    [GraphBuilder.optimize] done with 3 nodes in 0.009

Statistics¶

This can be used to see when a pattern is applied and how long it takes.

<<<

import pandas
import onnx
from experimental_experiment.xbuilder import GraphBuilder, OptimizationOptions

onx = onnx.load("temp_doc_mlp.onnx")

gr = GraphBuilder(
    onx,
    infer_shapes=True,
    optimization_options=OptimizationOptions(patterns="default"),
)
stat = gr.optimize()

print(pandas.DataFrame(stat))

>>>

                         pattern   time_in  removed  added  iteration  instances  match_index
                  check_A  0.000022      NaN    NaN        NaN        NaN          NaN
    remove_identity_nodes  0.000035      0.0    0.0        NaN        NaN          NaN
                  check_B  0.000013      NaN    NaN        NaN        NaN          NaN
            remove_unused  0.000036      0.0    NaN        NaN        NaN          NaN
                  check_C  0.000012      NaN    NaN        NaN        NaN          NaN
    ..                       ...       ...      ...    ...        ...        ...          ...
build_graph_for_pattern  0.000017      NaN    NaN        4.0        NaN          NaN
   pattern_optimization  0.004985      4.0    NaN        NaN        NaN          NaN
                check_F  0.000010      NaN    NaN        NaN        NaN          NaN
          remove_unused  0.000024      0.0    NaN        NaN        NaN          NaN
                check_G  0.000008      NaN    NaN        NaN        NaN          NaN
    
    [226 rows x 7 columns]

It can be aggregated:

<<<

import pandas
import onnx
from experimental_experiment.xbuilder import GraphBuilder, OptimizationOptions

onx = onnx.load("temp_doc_mlp.onnx")

gr = GraphBuilder(
    onx,
    infer_shapes=True,
    optimization_options=OptimizationOptions(patterns="default"),
)
stat = gr.optimize()

df = pandas.DataFrame(stat)
for c in df.columns:
    if "time" not in c and "pattern" not in c:
        df[c] = df[c].fillna(0).astype(int)
aggs = {
    "time_in": "sum",
    "added": "sum",
    "removed": "sum",
    "iteration": "max",
    "match_index": "max",
    "instances": "sum",
}
print(df.groupby("pattern").agg(aggs))

>>>

                                               time_in  added  removed  iteration  match_index  instances
    pattern                                                                                              
    apply_GemmTransposePattern                0.000714      4        2          2            1          2
    apply_MatMulAddPattern                    0.000471      2        4          1            1          2
    apply_TransposeTransposePattern           0.000511      2        4          3            1          2
    build_graph_for_pattern                   0.000334      0        0          4            0          0
    check_A                                   0.000046      0        0          0            0          0
    check_B                                   0.000028      0        0          0            0          0
    check_C                                   0.000026      0        0          0            0          0
    check_F                                   0.000022      0        0          0            0          0
    check_G                                   0.000018      0        0          0            0          0
    check_pattern_00                          0.000042      0        0         -1            0          0
    check_pattern_A0                          0.000186      0        0          3            0          0
    check_pattern_B0                          0.000106      0        0          3            0          0
    match_BatchNormalizationPattern           0.000095      0        0          4            0          0
    match_BatchNormalizationTrainingPattern   0.000053      0        0          4            0          0
    match_CastCastBinaryPattern               0.000414      0        0          4            0          0
    match_CastLayerNormalizationCastPattern   0.000045      0        0          4            0          0
    match_CastOpCastPattern                   0.000140      0        0          4            0          0
    match_CastPattern                         0.000050      0        0          4            0          0
    match_ClipClipPattern                     0.000059      0        0          4            0          0
    match_ComputationCastOpCastPattern        0.000084      0        0          4            0          0
    match_ConvBiasNullPattern                 0.000063      0        0          4            0          0
    match_DropoutPattern                      0.000047      0        0          4            0          0
    match_ExpandBroadcastPattern              0.000047      0        0          4            0          0
    match_ExpandPattern                       0.000061      0        0          4            0          0
    match_ExpandSwapPattern                   0.000047      0        0          4            0          0
    match_GeluPattern                         0.000009      0        0          4            0          0
    match_GemmTransposePattern                0.000154      0        0          4            2          2
    match_IdentityPattern                     0.000805      0        0          4            0          0
    match_LayerNormalizationPattern           0.000054      0        0          4            0          0
    match_LayerNormalizationScalePattern      0.000050      0        0          4            0          0
    match_LeakyReluPattern                    0.001209      0        0          4            0          0
    match_MatMulAddPattern                    0.000366      0        0          4            2          2
    match_MatMulReshape2Of3Pattern            0.000119      0        0          4            2          0
    match_MulMulMatMulPattern                 0.000108      0        0          4            2          0
    match_MulMulMulScalarPattern              0.000055      0        0          4            0          0
    match_ReduceReshapePattern                0.000062      0        0          4            0          0
    match_ReduceSumNormalizePattern           0.000050      0        0          4            0          0
    match_Reshape2Of3Pattern                  0.000106      0        0          4            0          0
    match_ReshapeMatMulReshapePattern         0.000087      0        0          4            0          0
    match_ReshapePattern                      0.000059      0        0          4            0          0
    match_ReshapeReshapeBinaryPattern         0.000081      0        0          4            0          0
    match_ReshapeReshapePattern               0.000060      0        0          4            2          0
    match_RotaryConcatPartPattern             0.000060      0        0          4            2          0
    match_SameChildrenPattern                 0.000133      0        0          4            2          0
    match_SequenceConstructAtPattern          0.000103      0        0          4            2          0
    match_SliceSlicePattern                   0.000069      0        0          4            2          0
    match_SlicesSplitPattern                  0.000055      0        0          4            2          0
    match_SoftmaxCrossEntropyLossCastPattern  0.003354      0        0          4            2          0
    match_Sub1MulPattern                      0.000063      0        0          4            2          0
    match_SwitchOrderBinaryPattern            0.000065      0        0          4            2          0
    match_TransposeMatMulPattern              0.000533      0        0          4            2          0
    match_TransposeReshapeMatMulPattern       0.000254      0        0          4            2          0
    match_TransposeReshapeTransposePattern    0.000141      0        0          4            2          0
    match_TransposeTransposePattern           0.000163      0        0          4            2          2
    match_UnsqueezeEqualPattern               0.000043      0        0          4            2          0
    match_UnsqueezeUnsqueezePattern           0.000050      0        0          4            2          0
    pattern_optimization                      0.013868      0        4          0            0          0
    remove_identity_nodes                     0.000476      2        4          3            0          0
    remove_unused                             0.000124      0        0          0            0          0

Shape inference¶

The optimizers require to know the shapes to ensure they can rewrite some nodes and avoid producing a model which does not return the same results. If it is missing, some patterns cannot match for sure and they will not match.

This information can be built by running shape inference on the onnx models. That’s what is done is the previous examples. However, the best case is when this information comes from torch.

Function to_onnx converts a torch model into ONNX. While doing so, it stores the shape information coming from torch. There is no need to run shape inference on the onnx model it generates before optimizing it.

Available Patterns and API¶

All patterns may be found at experimental_experiment.xoptim.patterns and experimental_experiment.xoptim.patterns_ort.

When writing a pattern, walking along the graph or checking the shape is very common. Class GraphBuilderPatternOptimization provides the following methods.

Opsets¶

Patterns must rewrite using the nodes of the opset defined in the model.

main_opset: returns the opset

Shapes, Types¶

has_type: tells if a result type is known
get_type: returns a result type, fails if not known
has_shape: tells if a result shape is known
get_shape: returns a result shape, fails if not known
has_rank: tells if a result rank is known
get_rank: returns a result rank, fails if not known
try_infer_type: returns a type if it can be guessed
try_infer_shape: returns a shape if it can be guessed

Constants¶

is_constant: tells if a node is a constant (it may be a constant, an initializer or any value built on other constants)
is_constant_scalar: checks a constant is a scalar and compares its value to a number
get_computed_constant: returns the constant, computes it is a constant built from other constants
get_attribute: returns an attribute of a node

Graph¶

next_node: returns the next node only if there is only one
next_nodes: returns the node consuming this result
node_before: returns the node producing the result
is_output: tells if a result is an output
is_used_by_subgraph: tells if a result is used by a subgraph
is_used_more_than_once: tells if a result is used more than once
is_used_only_by: tells if a result is only used by specific nodes

Nodes¶

make_node: creates a node without adding it to the graph
make_node_check_opset: creates a node without adding it to the graph, deals with some constraints related to opset version