yobx.xoptim.patterns_ort.activation#

class yobx.xoptim.patterns_ort.activation.BiasGeluPattern(verbose: int = 0, priority: int = 1, min_opset: int = 1)[source]#

Replaces by y = BiasGelu(x, B):

t = x + B
y = t ( Erf(1 / t) + 1)

Model with nodes to be fused:

        graph TD

    classDef ioNode fill:#dfd,stroke:#333,color:#333
    classDef initNode fill:#cccc00,stroke:#333,color:#333
    classDef constNode fill:#f9f,stroke:#333,stroke-width:2px,color:#333
    classDef opNode fill:#bbf,stroke:#333,stroke-width:2px,color:#333

    I_B(["B FLOAT(8)"])
    I_X(["X FLOAT(2, 2, 4, 8)"])

    Constant_0[["Constant() -#gt; B"]]
    Add_1[["Add(., .)"]]
    Div_2[["Div(., [1.4140625])"]]
    Erf_3[["Erf(.)"]]
    Add_4[["Add(., [1.0])"]]
    Mul_5[["Mul(., .)"]]
    Mul_6[["Mul(., [0.5])"]]

    I_X -->|"FLOAT(2, 2, 4, 8)"| Add_1
    Constant_0 -->|"FLOAT(8)"| Add_1
    Add_1 -->|"FLOAT(2, 2, 4, 8)"| Div_2
    Div_2 -->|"FLOAT(2, 2, 4, 8)"| Erf_3
    Erf_3 -->|"FLOAT(2, 2, 4, 8)"| Add_4
    Add_1 -->|"FLOAT(2, 2, 4, 8)"| Mul_5
    Add_4 -->|"FLOAT(2, 2, 4, 8)"| Mul_5
    Mul_5 -->|"FLOAT(2, 2, 4, 8)"| Mul_6

    O_Y(["Y FLOAT(2, 2, 4, 8)"])
    Mul_6 --> O_Y

    class I_B,I_X,O_Y ioNode
    class Constant_0 constNode
    class Add_1,Div_2,Erf_3,Add_4,Mul_5,Mul_6 opNode

Outcome of the fusion:

        graph TD

    classDef ioNode fill:#dfd,stroke:#333,color:#333
    classDef initNode fill:#cccc00,stroke:#333,color:#333
    classDef constNode fill:#f9f,stroke:#333,stroke-width:2px,color:#333
    classDef opNode fill:#bbf,stroke:#333,stroke-width:2px,color:#333

    I_B(["B FLOAT(8)"])
    I_X(["X FLOAT(2, 2, 4, 8)"])

    BiasGelu_0[["com.microsoft.BiasGelu(., .)"]]

    I_X -->|"FLOAT(2, 2, 4, 8)"| BiasGelu_0
    I_B -->|"FLOAT(8)"| BiasGelu_0

    O_Y(["Y FLOAT(2, 2, 4, 8)"])
    BiasGelu_0 --> O_Y

    class I_B,I_X,O_Y ioNode
    class BiasGelu_0 opNode
apply(g: GraphBuilder, add_node: NodeProto, div_node: NodeProto, erf_node: NodeProto, add_1_node: NodeProto, mul_node: NodeProto, half_node: NodeProto) → List[NodeProto][source]#

The method does the rewriting. It assumes it can happen. It takes a list of nodes impacted by the rewriting assumes no other pattern optimizer will be modify them. It receives the list of nodes returned by method apply. Since it is a list of argument, method match can include None values. The method returns the new nodes. The optimizer considers that any node given to this function is removed from the graph, and any node returned by it are added. If a received node must be kept, it must be added to the list of returned node.

Parameters:

nodes – nodes returned by method match, there are then removed

Returns:

nodes to add to graph.

match(g: GraphBuilderPatternOptimization, node: NodeProto, matched: List[MatchResult]) → MatchResult | None[source]#

Determines nodes around node which can be rewritten.

Parameters:

  • g – is a GraphBuilderPatternOptimization, it holds all the existing nodes, is able to return any information about type, shape, the node before, the node after another one.

  • node – the matching must determine if some nodes around this one are part of set of nodes this pattern optimizer can rewrite. From there, the function explores wherever it needs, checking any condition it needs.

  • matched – usually unused, it returns of nodes already matching a pattern

The method must not modify the graph. The method returns None if no match is found or an instance of class MatchResult. It must contain:

  • a list of nodes involved in the rewriting. It does not mean all of them will be removed but all of them are needed to do the rewriting and must not be impacted by other pattern optimizer.

  • A function doing the rewriting (usually method apply of the pattern class).

  • An existing node where the rewritten nodes can be inserted. Knowing it makes it faster to rewriter. If not specified, the optimizer will automatically determine the position of the new nodes.

class yobx.xoptim.patterns_ort.activation.BiasSoftmaxPattern(verbose: int = 0, priority: int = 1, min_opset: int = 1)[source]#

Replaces Softmax(Add(x,y), axis=-1) by BiasSoftmax(x,y,axis=-1)

Model with nodes to be fused:

        graph TD

    classDef ioNode fill:#dfd,stroke:#333,color:#333
    classDef initNode fill:#cccc00,stroke:#333,color:#333
    classDef constNode fill:#f9f,stroke:#333,stroke-width:2px,color:#333
    classDef opNode fill:#bbf,stroke:#333,stroke-width:2px,color:#333

    I_X(["X FLOAT(16, 8, 4, 8)"])
    I_Y(["Y FLOAT(16, 1, 4, 8)"])

    Add_0[["Add(., .)"]]
    Softmax_1[["Softmax(., axis=-1)"]]

    I_X -->|"FLOAT(16, 8, 4, 8)"| Add_0
    I_Y -->|"FLOAT(16, 1, 4, 8)"| Add_0
    Add_0 -->|"FLOAT(16, 8, 4, 8)"| Softmax_1

    O_Z(["Z FLOAT(16, 8, 4, 8)"])
    Softmax_1 --> O_Z

    class I_X,I_Y,O_Z ioNode
    class Add_0,Softmax_1 opNode

Outcome of the fusion:

        graph TD

    classDef ioNode fill:#dfd,stroke:#333,color:#333
    classDef initNode fill:#cccc00,stroke:#333,color:#333
    classDef constNode fill:#f9f,stroke:#333,stroke-width:2px,color:#333
    classDef opNode fill:#bbf,stroke:#333,stroke-width:2px,color:#333

    I_X(["X FLOAT(16, 8, 4, 8)"])
    I_Y(["Y FLOAT(16, 1, 4, 8)"])

    BiasSoftmax_0[["com.microsoft.BiasSoftmax(., ., axis=-1)"]]

    I_X -->|"FLOAT(16, 8, 4, 8)"| BiasSoftmax_0
    I_Y -->|"FLOAT(16, 1, 4, 8)"| BiasSoftmax_0

    O_Z(["Z FLOAT(16, 8, 4, 8)"])
    BiasSoftmax_0 --> O_Z

    class I_X,I_Y,O_Z ioNode
    class BiasSoftmax_0 opNode
apply(g: GraphBuilder, add_node: NodeProto, softmax_node: NodeProto) → List[NodeProto][source]#

The method does the rewriting. It assumes it can happen. It takes a list of nodes impacted by the rewriting assumes no other pattern optimizer will be modify them. It receives the list of nodes returned by method apply. Since it is a list of argument, method match can include None values. The method returns the new nodes. The optimizer considers that any node given to this function is removed from the graph, and any node returned by it are added. If a received node must be kept, it must be added to the list of returned node.

Parameters:

nodes – nodes returned by method match, there are then removed

Returns:

nodes to add to graph.

match(g: GraphBuilderPatternOptimization, node: NodeProto, matched: List[MatchResult]) → MatchResult | None[source]#

Determines nodes around node which can be rewritten.

Parameters:

  • g – is a GraphBuilderPatternOptimization, it holds all the existing nodes, is able to return any information about type, shape, the node before, the node after another one.

  • node – the matching must determine if some nodes around this one are part of set of nodes this pattern optimizer can rewrite. From there, the function explores wherever it needs, checking any condition it needs.

  • matched – usually unused, it returns of nodes already matching a pattern

The method must not modify the graph. The method returns None if no match is found or an instance of class MatchResult. It must contain:

  • a list of nodes involved in the rewriting. It does not mean all of them will be removed but all of them are needed to do the rewriting and must not be impacted by other pattern optimizer.

  • A function doing the rewriting (usually method apply of the pattern class).

  • An existing node where the rewritten nodes can be inserted. Knowing it makes it faster to rewriter. If not specified, the optimizer will automatically determine the position of the new nodes.

class yobx.xoptim.patterns_ort.activation.FastGeluPattern(verbose: int = 0, priority: int = 1, min_opset: int = 1)[source]#

Replaces Gelu by FastGelu.

Model with nodes to be fused:

        graph TD

    classDef ioNode fill:#dfd,stroke:#333,color:#333
    classDef initNode fill:#cccc00,stroke:#333,color:#333
    classDef constNode fill:#f9f,stroke:#333,stroke-width:2px,color:#333
    classDef opNode fill:#bbf,stroke:#333,stroke-width:2px,color:#333

    I_linear_65(["linear_65 FLOAT16(4, 512, 16384)"])

    Gelu_0[["Gelu(.)"]]

    I_linear_65 -->|"FLOAT16(4, 512, 16384)"| Gelu_0

    O_mul_44(["mul_44 FLOAT16(4, 512, 16384)"])
    Gelu_0 --> O_mul_44

    class I_linear_65,O_mul_44 ioNode
    class Gelu_0 opNode

Outcome of the fusion:

        graph TD

    classDef ioNode fill:#dfd,stroke:#333,color:#333
    classDef initNode fill:#cccc00,stroke:#333,color:#333
    classDef constNode fill:#f9f,stroke:#333,stroke-width:2px,color:#333
    classDef opNode fill:#bbf,stroke:#333,stroke-width:2px,color:#333

    I_linear_65(["linear_65 FLOAT16(4, 512, 16384)"])

    FastGelu_0[["com.microsoft.FastGelu(.)"]]

    I_linear_65 -->|"FLOAT16(4, 512, 16384)"| FastGelu_0

    O_mul_44(["mul_44 FLOAT16(4, 512, 16384)"])
    FastGelu_0 --> O_mul_44

    class I_linear_65,O_mul_44 ioNode
    class FastGelu_0 opNode
apply(g: GraphBuilder, gelu_node: NodeProto) → List[NodeProto][source]#

The method does the rewriting. It assumes it can happen. It takes a list of nodes impacted by the rewriting assumes no other pattern optimizer will be modify them. It receives the list of nodes returned by method apply. Since it is a list of argument, method match can include None values. The method returns the new nodes. The optimizer considers that any node given to this function is removed from the graph, and any node returned by it are added. If a received node must be kept, it must be added to the list of returned node.

Parameters:

nodes – nodes returned by method match, there are then removed

Returns:

nodes to add to graph.

match(g: GraphBuilderPatternOptimization, node: NodeProto, matched: List[MatchResult]) → MatchResult | None[source]#

Determines nodes around node which can be rewritten.

Parameters:

  • g – is a GraphBuilderPatternOptimization, it holds all the existing nodes, is able to return any information about type, shape, the node before, the node after another one.

  • node – the matching must determine if some nodes around this one are part of set of nodes this pattern optimizer can rewrite. From there, the function explores wherever it needs, checking any condition it needs.

  • matched – usually unused, it returns of nodes already matching a pattern

The method must not modify the graph. The method returns None if no match is found or an instance of class MatchResult. It must contain:

  • a list of nodes involved in the rewriting. It does not mean all of them will be removed but all of them are needed to do the rewriting and must not be impacted by other pattern optimizer.

  • A function doing the rewriting (usually method apply of the pattern class).

  • An existing node where the rewritten nodes can be inserted. Knowing it makes it faster to rewriter. If not specified, the optimizer will automatically determine the position of the new nodes.

class yobx.xoptim.patterns_ort.activation.GeluErfPattern(verbose: int = 0, priority: int = 0, min_opset: int = 1)[source]#

Detects the decomposed version of Gelu with Erf.

Model with nodes to be fused:

        graph TD

    classDef ioNode fill:#dfd,stroke:#333,color:#333
    classDef initNode fill:#cccc00,stroke:#333,color:#333
    classDef constNode fill:#f9f,stroke:#333,stroke-width:2px,color:#333
    classDef opNode fill:#bbf,stroke:#333,stroke-width:2px,color:#333

    I_X(["X FLOAT(2, 2, 4, 8)"])

    Div_0[["Div(., [1.4140625])"]]
    Erf_1[["Erf(.)"]]
    Add_2[["Add(., [1.0])"]]
    Mul_3[["Mul(., .)"]]
    Mul_4[["Mul([0.5], .)"]]

    I_X -->|"FLOAT(2, 2, 4, 8)"| Div_0
    Div_0 -->|"FLOAT(2, 2, 4, 8)"| Erf_1
    Erf_1 -->|"FLOAT(2, 2, 4, 8)"| Add_2
    I_X -->|"FLOAT(2, 2, 4, 8)"| Mul_3
    Add_2 -->|"FLOAT(2, 2, 4, 8)"| Mul_3
    Mul_3 -->|"FLOAT(2, 2, 4, 8)"| Mul_4

    O_Y(["Y FLOAT(2, 2, 4, 8)"])
    Mul_4 --> O_Y

    class I_X,O_Y ioNode
    class Div_0,Erf_1,Add_2,Mul_3,Mul_4 opNode

Outcome of the fusion:

        graph TD

    classDef ioNode fill:#dfd,stroke:#333,color:#333
    classDef initNode fill:#cccc00,stroke:#333,color:#333
    classDef constNode fill:#f9f,stroke:#333,stroke-width:2px,color:#333
    classDef opNode fill:#bbf,stroke:#333,stroke-width:2px,color:#333

    I_X(["X FLOAT(2, 2, 4, 8)"])

    Gelu_0[["com.microsoft.Gelu(.)"]]

    I_X -->|"FLOAT(2, 2, 4, 8)"| Gelu_0

    O_Y(["Y FLOAT(2, 2, 4, 8)"])
    Gelu_0 --> O_Y

    class I_X,O_Y ioNode
    class Gelu_0 opNode
apply_pattern(g: GraphBuilder, x, cst2, one, c05)[source]#

Applies the replacement.

match_pattern(g: GraphBuilder, x, cst2, one, c05)[source]#

Builds the pattern to match.

validate_mapping(g: GraphBuilderPatternOptimization, deleted_nodes: List[NodeProto], pattern_nodes: List[NodeProto] | None = None) → bool[source]#

Validates the mapping.

Parameters:

  • g – GraphBuilder

  • deleted_nodes – matched nodes from the model (to be deleted)

  • pattern_nodes – matched nodes coming from the pattern

Returns:

validate the mapping or not, default is True

class yobx.xoptim.patterns_ort.activation.GeluOrtPattern(verbose: int = 0, priority: int = 0, min_opset: int = 1, domain: str = 'com.microsoft')[source]#

Detects the decomposed version of Gelu with Tanh

y = \frac{x}{2} \left(1 + \tanh\left(\sqrt{\frac{2}{\pi}} (x + 0.044715 * x^3)\right)\right)

Model with nodes to be fused:

        graph TD

    classDef ioNode fill:#dfd,stroke:#333,color:#333
    classDef initNode fill:#cccc00,stroke:#333,color:#333
    classDef constNode fill:#f9f,stroke:#333,stroke-width:2px,color:#333
    classDef opNode fill:#bbf,stroke:#333,stroke-width:2px,color:#333

    I_linear_73(["linear_73 FLOAT16(4, 512, 128)"])

    Pow_0[["Pow(., [3.0])"]]
    Mul_1[["Mul(., 0.0447)"]]
    Add_2[["Add(., .)"]]
    Mul_3[["Mul(., 0.798)"]]
    Tanh_4[["Tanh(.)"]]
    Add_5[["Add(., 1.0)"]]
    Mul_6[["Mul(., 0.5)"]]
    Mul_7[["Mul(., .)"]]

    I_linear_73 -->|"FLOAT16(4, 512, 128)"| Pow_0
    Pow_0 -->|"FLOAT16(4, 512, 128)"| Mul_1
    I_linear_73 -->|"FLOAT16(4, 512, 128)"| Add_2
    Mul_1 -->|"FLOAT16(4, 512, 128)"| Add_2
    Add_2 -->|"FLOAT16(4, 512, 128)"| Mul_3
    Mul_3 -->|"FLOAT16(4, 512, 128)"| Tanh_4
    Tanh_4 -->|"FLOAT16(4, 512, 128)"| Add_5
    I_linear_73 -->|"FLOAT16(4, 512, 128)"| Mul_6
    Mul_6 -->|"FLOAT16(4, 512, 128)"| Mul_7
    Add_5 -->|"FLOAT16(4, 512, 128)"| Mul_7

    O_mul_52(["mul_52 FLOAT16(4, 512, 128)"])
    Mul_7 --> O_mul_52

    class I_linear_73,O_mul_52 ioNode
    class Pow_0,Mul_1,Add_2,Mul_3,Tanh_4,Add_5,Mul_6,Mul_7 opNode

Outcome of the fusion:

        graph TD

    classDef ioNode fill:#dfd,stroke:#333,color:#333
    classDef initNode fill:#cccc00,stroke:#333,color:#333
    classDef constNode fill:#f9f,stroke:#333,stroke-width:2px,color:#333
    classDef opNode fill:#bbf,stroke:#333,stroke-width:2px,color:#333

    I_linear_73(["linear_73 FLOAT16(4, 512, 128)"])

    Gelu_0[["com.microsoft.Gelu(.)"]]

    I_linear_73 -->|"FLOAT16(4, 512, 128)"| Gelu_0

    O_mul_52(["mul_52 FLOAT16(4, 512, 128)"])
    Gelu_0 --> O_mul_52

    class I_linear_73,O_mul_52 ioNode
    class Gelu_0 opNode
class yobx.xoptim.patterns_ort.activation.QuickGeluPattern(verbose: int = 0, priority: int = 1, min_opset: int = 1)[source]#

Replaces Mul(x, Sigmoid(x)) by QuickGelu(x, alpha=1)

Model with nodes to be fused:

        graph TD

    classDef ioNode fill:#dfd,stroke:#333,color:#333
    classDef initNode fill:#cccc00,stroke:#333,color:#333
    classDef constNode fill:#f9f,stroke:#333,stroke-width:2px,color:#333
    classDef opNode fill:#bbf,stroke:#333,stroke-width:2px,color:#333

    I_X(["X FLOAT(1, 8, 6, 6)"])

    Sigmoid_0[["Sigmoid(.)"]]
    Mul_1[["Mul(., .)"]]

    I_X -->|"FLOAT(1, 8, 6, 6)"| Sigmoid_0
    I_X -->|"FLOAT(1, 8, 6, 6)"| Mul_1
    Sigmoid_0 -->|"FLOAT(1, 8, 6, 6)"| Mul_1

    O_Y(["Y FLOAT(1, 8, 6, 6)"])
    Mul_1 --> O_Y

    class I_X,O_Y ioNode
    class Sigmoid_0,Mul_1 opNode

Outcome of the fusion:

        graph TD

    classDef ioNode fill:#dfd,stroke:#333,color:#333
    classDef initNode fill:#cccc00,stroke:#333,color:#333
    classDef constNode fill:#f9f,stroke:#333,stroke-width:2px,color:#333
    classDef opNode fill:#bbf,stroke:#333,stroke-width:2px,color:#333

    I_X(["X FLOAT(1, 8, 6, 6)"])

    QuickGelu_0[["com.microsoft.QuickGelu(.)"]]

    I_X -->|"FLOAT(1, 8, 6, 6)"| QuickGelu_0

    O_Y(["Y FLOAT(1, 8, 6, 6)"])
    QuickGelu_0 --> O_Y

    class I_X,O_Y ioNode
    class QuickGelu_0 opNode
apply(g: GraphBuilder, sigmoid: NodeProto, mul_node: NodeProto) → List[NodeProto][source]#

The method does the rewriting. It assumes it can happen. It takes a list of nodes impacted by the rewriting assumes no other pattern optimizer will be modify them. It receives the list of nodes returned by method apply. Since it is a list of argument, method match can include None values. The method returns the new nodes. The optimizer considers that any node given to this function is removed from the graph, and any node returned by it are added. If a received node must be kept, it must be added to the list of returned node.

Parameters:

nodes – nodes returned by method match, there are then removed

Returns:

nodes to add to graph.

match(g: GraphBuilderPatternOptimization, node: NodeProto, matched: List[MatchResult]) → MatchResult | None[source]#

Determines nodes around node which can be rewritten.

Parameters:

  • g – is a GraphBuilderPatternOptimization, it holds all the existing nodes, is able to return any information about type, shape, the node before, the node after another one.

  • node – the matching must determine if some nodes around this one are part of set of nodes this pattern optimizer can rewrite. From there, the function explores wherever it needs, checking any condition it needs.

  • matched – usually unused, it returns of nodes already matching a pattern

The method must not modify the graph. The method returns None if no match is found or an instance of class MatchResult. It must contain:

  • a list of nodes involved in the rewriting. It does not mean all of them will be removed but all of them are needed to do the rewriting and must not be impacted by other pattern optimizer.

  • A function doing the rewriting (usually method apply of the pattern class).

  • An existing node where the rewritten nodes can be inserted. Knowing it makes it faster to rewriter. If not specified, the optimizer will automatically determine the position of the new nodes.