Un arbre de décision en réseaux de neurones#

L’idée est de convertir sous la forme d’un réseaux de neurones un arbre de décision puis de continuer l’apprentissage de façon à obtenir un assemblage de régression logistique plutôt que de décision binaire.

[1]:

%matplotlib inline
import numpy
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
from tqdm import tqdm

Un exemple sur Iris#

La méthode ne marche que sur un problème de classification binaire.

[2]:

from sklearn.datasets import load_iris

data = load_iris()
X, y = data.data[:, :2], data.target
y = y % 2

[3]:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=11)

[4]:

from sklearn.tree import DecisionTreeClassifier

dec = DecisionTreeClassifier(max_depth=2, random_state=11)
dec.fit(X_train, y_train)
dec.score(X_test, y_test)

[4]:

0.6052631578947368

[5]:

from sklearn.tree import export_graphviz
from mlstatpy.render_js_dot import RenderJsDot

dot = export_graphviz(dec, filled=True)
dot = dot.replace("shape=box, ", "shape=box, fontsize=10, ")
RenderJsDot(dot)
print(dot)

digraph Tree {
node [shape=box, fontsize=10, style="filled", color="black", fontname="helvetica"] ;
edge [fontname="helvetica"] ;
0 [label="x[1] <= 2.95\ngini = 0.454\nsamples = 112\nvalue = [73, 39]", fillcolor="#f3c4a3"] ;
1 [label="x[0] <= 7.05\ngini = 0.429\nsamples = 45\nvalue = [14, 31]", fillcolor="#92c9f1"] ;
0 -> 1 [labeldistance=2.5, labelangle=45, headlabel="True"] ;
2 [label="gini = 0.402\nsamples = 43\nvalue = [12.0, 31.0]", fillcolor="#86c3ef"] ;
1 -> 2 ;
3 [label="gini = 0.0\nsamples = 2\nvalue = [2, 0]", fillcolor="#e58139"] ;
1 -> 3 ;
4 [label="x[1] <= 3.25\ngini = 0.21\nsamples = 67\nvalue = [59, 8]", fillcolor="#e99254"] ;
0 -> 4 [labeldistance=2.5, labelangle=-45, headlabel="False"] ;
5 [label="gini = 0.375\nsamples = 32\nvalue = [24, 8]", fillcolor="#eeab7b"] ;
4 -> 5 ;
6 [label="gini = 0.0\nsamples = 35\nvalue = [35, 0]", fillcolor="#e58139"] ;
4 -> 6 ;
}

L’arbre de décision est petit donc visuellement réduit et il est perfectible aussi.

Même exemple en réseau de neurones#

Chaque noeud de l’arbre de décision est converti en deux neurones : * un qui le relie à l’entrée et qui évalue la décision, il produit la valeur \(o_1\) * un autre qui associe le résultat du premier noeud avec celui le précède dans la structure de l’arbre de décision, il produit la valeur \(o_2\) La décision finale est quelque chose comme \(sigmoid(o_1 + o_2 - 1)\). Un neurone agrège le résultat de toutes les feuilles.

[6]:

from mlstatpy.ml.neural_tree import NeuralTreeNet

net = NeuralTreeNet.create_from_tree(dec)
RenderJsDot(net.to_dot())

[6]:

On considère une entrée en particulier.

[7]:

n = 60
dec.predict_proba(X[n : n + 1])

[7]:

array([[0.27906977, 0.72093023]])

Les sorties du réseau de neurones :

[8]:

net.predict(X[n : n + 1])[:, -2:]

[8]:

array([[0.12536069, 0.87463931]])

Et on trace les valeurs intermédiaires.

[9]:

RenderJsDot(net.to_dot(X=X[n]))

[9]:

On poursuit la comparaison :

[10]:

dec.predict_proba(X_test)[:5]

[10]:

array([[0.75      , 0.25      ],
       [0.75      , 0.25      ],
       [0.27906977, 0.72093023],
       [1.        , 0.        ],
       [0.27906977, 0.72093023]])

[11]:

net.predict(X_test)[:5, -2:]

[11]:

array([[0.79156817, 0.20843183],
       [0.73646978, 0.26353022],
       [0.29946111, 0.70053889],
       [0.94070094, 0.05929906],
       [0.24924737, 0.75075263]])

[12]:

dec.predict_proba(X_test)[-5:]

[12]:

array([[1.  , 0.  ],
       [0.75, 0.25],
       [1.  , 0.  ],
       [0.75, 0.25],
       [0.75, 0.25]])

[13]:

net.predict(X_test)[-5:, -2:]

[13]:

array([[0.93247891, 0.06752109],
       [0.86338585, 0.13661415],
       [0.98219036, 0.01780964],
       [0.98352807, 0.01647193],
       [0.73646978, 0.26353022]])

[14]:

numpy.argmax(net.predict(X_test)[-5:, -2:], axis=1)

[14]:

array([0, 0, 0, 0, 0])

On compare visuellement les deux frontières de classification.

[15]:

def plot_grid(X, y, fct, title, ax=None):
    cmap_light = ListedColormap(["orange", "cyan", "cornflowerblue"])
    cmap_bold = ListedColormap(["darkorange", "c", "darkblue"])

    h = 0.05
    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx, yy = numpy.meshgrid(
        numpy.arange(x_min, x_max, h), numpy.arange(y_min, y_max, h)
    )
    Z = fct(numpy.c_[xx.ravel(), yy.ravel()])

    Z = Z.reshape(xx.shape)
    if ax is None:
        _, ax = plt.subplots(1, 1)
    ax.pcolormesh(xx, yy, Z, cmap=cmap_light)

    ax.scatter(X[:, 0], X[:, 1], c=y, cmap=cmap_bold, edgecolor="k", s=20)
    ax.set_xlim(xx.min(), xx.max())
    ax.set_ylim(yy.min(), yy.max())
    ax.set_title(title)


fig, ax = plt.subplots(1, 2, figsize=(12, 4))
plot_grid(X, y, dec.predict, dec.__class__.__name__, ax=ax[0])
plot_grid(
    X,
    y,
    lambda x: numpy.argmax(net.predict(x)[:, -2:], axis=1),
    net.__class__.__name__,
    ax=ax[1],
)
../../_images/notebooks_ml_neural_tree_23_0.png

Le code qui produit les prédictions du réseau de neurones est assez long à exécuter mais il produit à peu près les mêmes frontières excepté qu’elles sont plus arrondies.

Intermède de simples neurones de régression#

Avant d’apprendre ou plutôt de continuer l’apprentissage des coefficients du réseaux de neurones, voyons comment un neurone se débrouille sur un problème de régression. Le neurone n’est pas converti, il est appris.

[16]:

regX = numpy.empty((150, 1), dtype=numpy.float64)
regX[:50, 0] = numpy.random.randn(50) - 4
regX[50:100, 0] = numpy.random.randn(50)
regX[100:, 0] = numpy.random.randn(50) + 4
noise = numpy.random.randn(regX.shape[0]) / 10
regY = regX[:, 0] * -0.5 * 0.2 + noise
regY[regX[:, 0] > 0.3] = noise[regX[:, 0] > 0.3]

fig, ax = plt.subplots(1, 1, figsize=(8, 4))
ax.scatter(regX[:, 0], regY)
ax.set_title("Nuage de points linéaire par morceaux");
../../_images/notebooks_ml_neural_tree_26_0.png

On cale une régression avec scikit-learn.

[17]:

from sklearn.linear_model import LinearRegression

lr = LinearRegression()
lr.fit(regX, regY)

fig, ax = plt.subplots(1, 1, figsize=(8, 4))
ax.scatter(regX[:, 0], regY)
ax.scatter(regX[:, 0], lr.predict(regX))
ax.set_title("Régression scikit-learn");
../../_images/notebooks_ml_neural_tree_28_0.png

Et maintenant un neurone avec une fonction d’activation « identity ».

[18]:

from mlstatpy.ml.neural_tree import NeuralTreeNode

neu = NeuralTreeNode(1, activation="identity")
neu

[18]:

NeuralTreeNode(weights=array([-0.2630011]), bias=np.float64(0.009885644406795709), activation='identity')

[19]:

neu.fit(regX, regY, verbose=True, max_iter=20)

0/20: loss: 84.07 lr=0.002 max(coef): 0.26 l1=0/0.27 l2=0/0.069
1/20: loss: 4.075 lr=0.000163 max(coef): 0.092 l1=1.4/0.17 l2=1.2/0.014
2/20: loss: 2.51 lr=0.000115 max(coef): 0.12 l1=0.6/0.17 l2=0.22/0.017
3/20: loss: 2.661 lr=9.42e-05 max(coef): 0.13 l1=0.34/0.19 l2=0.072/0.021
4/20: loss: 2.493 lr=8.16e-05 max(coef): 0.13 l1=0.59/0.18 l2=0.23/0.019
5/20: loss: 2.477 lr=7.3e-05 max(coef): 0.13 l1=3.1/0.18 l2=6.8/0.019
6/20: loss: 2.48 lr=6.66e-05 max(coef): 0.13 l1=0.32/0.18 l2=0.069/0.019
7/20: loss: 2.48 lr=6.17e-05 max(coef): 0.14 l1=0.46/0.19 l2=0.13/0.021
8/20: loss: 2.567 lr=5.77e-05 max(coef): 0.14 l1=1.3/0.19 l2=1.1/0.022
9/20: loss: 2.46 lr=5.44e-05 max(coef): 0.14 l1=0.27/0.19 l2=0.048/0.022
10/20: loss: 2.476 lr=5.16e-05 max(coef): 0.14 l1=2.7/0.19 l2=5.5/0.022
11/20: loss: 2.478 lr=4.92e-05 max(coef): 0.14 l1=1.8/0.19 l2=2.2/0.021
12/20: loss: 2.465 lr=4.71e-05 max(coef): 0.14 l1=2.5/0.19 l2=4.5/0.022
13/20: loss: 2.48 lr=4.53e-05 max(coef): 0.14 l1=0.19/0.19 l2=0.024/0.022
14/20: loss: 2.464 lr=4.36e-05 max(coef): 0.14 l1=0.12/0.19 l2=0.0072/0.023
15/20: loss: 2.467 lr=4.22e-05 max(coef): 0.14 l1=0.85/0.19 l2=0.49/0.023
16/20: loss: 2.472 lr=4.08e-05 max(coef): 0.14 l1=0.61/0.19 l2=0.34/0.023
17/20: loss: 2.463 lr=3.96e-05 max(coef): 0.14 l1=0.42/0.19 l2=0.11/0.023
18/20: loss: 2.46 lr=3.85e-05 max(coef): 0.14 l1=0.6/0.19 l2=0.18/0.022
19/20: loss: 2.483 lr=3.75e-05 max(coef): 0.14 l1=0.1/0.19 l2=0.0064/0.022
20/20: loss: 2.46 lr=3.65e-05 max(coef): 0.14 l1=1.5/0.19 l2=1.6/0.022

[19]:

NeuralTreeNode(weights=array([-0.05022479]), bias=np.float64(0.1388943013680868), activation='identity')

[20]:

fig, ax = plt.subplots(1, 1, figsize=(8, 4))
ax.scatter(regX[:, 0], regY)
ax.scatter(regX[:, 0], lr.predict(regX), label="sklearn")
ax.scatter(regX[:, 0], neu.predict(regX), label="NeuralTreeNode")
ax.legend()
ax.set_title("Régression et neurones");
../../_images/notebooks_ml_neural_tree_32_0.png

Ca marche. Et avec d’autres fonctions d’activation…

[21]:

neus = {"identity": neu}
for act in tqdm(["relu", "leakyrelu", "sigmoid"]):
    nact = NeuralTreeNode(1, activation=act)
    nact.fit(regX, regY)
    neus[act] = nact

100%|██████████| 3/3 [00:00<00:00,  3.22it/s]

[22]:

neus["relu"], neus["leakyrelu"]

[22]:

(NeuralTreeNode(weights=array([-0.17040281]), bias=np.float64(-0.286333299470671), activation='relu'),
 NeuralTreeNode(weights=array([-0.21481952]), bias=np.float64(-0.48562866410120015), activation='leakyrelu'))

[23]:

fig, ax = plt.subplots(1, 1, figsize=(8, 4))
ax.scatter(regX[:, 0], regY)
ax.scatter(regX[:, 0], lr.predict(regX), label="sklearn")
for k, v in neus.items():
    ax.scatter(regX[:, 0], v.predict(regX), label=k)
ax.legend()
ax.set_title("Régression, neurone\nactivation");
../../_images/notebooks_ml_neural_tree_36_0.png

Rien de surprenant. La fonction sigmoïde prend ses valeurs entre 0 et 1. La fonction relu est parfois nulle sur une demi-droite, dès que la fonction est nulle sur l’ensemble du nuage de points, le gradient est nul partout (voir Rectifier (neural networks)). La fonction leaky relu est définie comme suit :

\[\begin{split}f(x) = \left\{\begin{array}{l} x \, si \, x > 0 \\ \frac{x}{100} \, sinon \end{array}\right.\end{split}\]

Le gradient n’est pas nul sur la partie la plus plate.

Intermède de simples neurones de classification#

Avant d’apprendre ou plutôt de continuer l’apprentissage des coefficients du réseaux de neurones, voyons comment un neurone se débrouille sur un problème de classification. Le neurone n’est pas converti mais appris.

[24]:

from sklearn.linear_model import LogisticRegression


clsX = numpy.empty((100, 2), dtype=numpy.float64)
clsX[:50] = numpy.random.randn(50, 2)
clsX[50:] = numpy.random.randn(50, 2) + 2
clsy = numpy.zeros(100, dtype=numpy.int64)
clsy[50:] = 1

logr = LogisticRegression()
logr.fit(clsX, clsy)
pred1 = logr.predict(clsX)

[25]:

def line_cls(x0, x1, coef, bias):
    y0 = -(coef[0, 0] * x0 + bias) / coef[0, 1]
    y1 = -(coef[0, 0] * x1 + bias) / coef[0, 1]
    return x0, y0, x1, y1


x0, y0, x1, y1 = line_cls(-5, 5, logr.coef_, logr.intercept_)

[26]:

h = 0.1
fig, ax = plt.subplots(1, 1, figsize=(8, 4))
ax.scatter(clsX[clsy == 0, 0], clsX[clsy == 0, 1], label="cl0")
ax.scatter(clsX[clsy == 1, 0], clsX[clsy == 1, 1], label="cl1")
ax.scatter(clsX[pred1 == 0, 0] + h, clsX[pred1 == 0, 1] + h, label="LR0")
ax.scatter(clsX[pred1 == 1, 0] + h, clsX[pred1 == 1, 1] + h, label="LR1")
ax.plot([x0, x1], [y0, y1], "y--", lw=4, label="frontière LR")
ax.set_ylim([-3, 3])
ax.legend()
ax.set_title("Classification et neurones");
../../_images/notebooks_ml_neural_tree_41_0.png

Un neurone de classification binaire produit deux sorties, une pour chaque classe, et sont normalisées à 1. La fonction d’activation est la fonction softmax.

[27]:

clsY = numpy.empty((clsy.shape[0], 2), dtype=numpy.float64)
clsY[:, 1] = clsy
clsY[:, 0] = 1 - clsy

[28]:

softneu = NeuralTreeNode(2, activation="softmax")
softneu

[28]:

NeuralTreeNode(weights=array([[1.30014733, 1.48516886],
       [0.12365284, 0.50958825]]), bias=array([-1.00277994, -0.24843673]), activation='softmax')

[29]:

softneu.fit(clsX, clsY, verbose=True, max_iter=20, lr=0.001)

0/20: loss: 1225 lr=0.001 max(coef): 1.5 l1=0/4.7 l2=0/5.2
1/20: loss: 285.3 lr=9.95e-05 max(coef): 6.1 l1=41/26 l2=3.4e+02/1.2e+02
2/20: loss: 262.8 lr=7.05e-05 max(coef): 6.9 l1=30/30 l2=2e+02/1.6e+02
3/20: loss: 255.5 lr=5.76e-05 max(coef): 7.1 l1=36/31 l2=2.5e+02/1.7e+02
4/20: loss: 253 lr=4.99e-05 max(coef): 7.3 l1=32/32 l2=2e+02/1.9e+02
5/20: loss: 245.7 lr=4.47e-05 max(coef): 7.5 l1=26/33 l2=1.5e+02/2e+02
6/20: loss: 243.7 lr=4.08e-05 max(coef): 7.7 l1=34/34 l2=2.2e+02/2.1e+02
7/20: loss: 240.7 lr=3.78e-05 max(coef): 7.8 l1=32/35 l2=2.7e+02/2.2e+02
8/20: loss: 238.3 lr=3.53e-05 max(coef): 7.9 l1=27/36 l2=1.5e+02/2.2e+02
9/20: loss: 232.6 lr=3.33e-05 max(coef): 8 l1=11/36 l2=27/2.3e+02
10/20: loss: 232.2 lr=3.16e-05 max(coef): 8.1 l1=12/37 l2=32/2.4e+02
11/20: loss: 228.4 lr=3.01e-05 max(coef): 8.2 l1=27/37 l2=1.5e+02/2.4e+02
12/20: loss: 223.3 lr=2.89e-05 max(coef): 8.3 l1=32/38 l2=2e+02/2.5e+02
13/20: loss: 221.1 lr=2.77e-05 max(coef): 8.4 l1=27/38 l2=1.5e+02/2.6e+02
14/20: loss: 219.5 lr=2.67e-05 max(coef): 8.5 l1=15/39 l2=58/2.6e+02
15/20: loss: 216.1 lr=2.58e-05 max(coef): 8.5 l1=37/39 l2=2.7e+02/2.7e+02
16/20: loss: 214.6 lr=2.5e-05 max(coef): 8.6 l1=19/40 l2=98/2.7e+02
17/20: loss: 212.4 lr=2.42e-05 max(coef): 8.7 l1=18/40 l2=83/2.8e+02
18/20: loss: 210.7 lr=2.36e-05 max(coef): 8.7 l1=20/40 l2=85/2.9e+02
19/20: loss: 208.3 lr=2.29e-05 max(coef): 8.8 l1=19/41 l2=1.1e+02/2.9e+02
20/20: loss: 206.4 lr=2.24e-05 max(coef): 8.8 l1=35/41 l2=2.7e+02/3e+02

[29]:

NeuralTreeNode(weights=array([[5.54615581, 5.83619756],
       [8.82929618, 8.48175986]]), bias=array([7.52191022, 4.91851196]), activation='softmax')

[30]:

pred = softneu.predict(clsX)
pred[:5]

[30]:

array([[3.69523065e-01, 6.30476935e-01],
       [8.29284938e-01, 1.70715062e-01],
       [3.48656758e-01, 6.51343242e-01],
       [9.38509501e-01, 6.14904995e-02],
       [9.99116470e-01, 8.83529803e-04]])

[31]:

pred2 = (pred[:, 1] > 0.5).astype(numpy.int64)

[32]:

x00, y00, x01, y01 = line_cls(-4, 4, softneu.coef[:1, 1:], softneu.bias[0])
x10, y10, x11, y11 = line_cls(-4, 4, softneu.coef[1:, 1:], softneu.bias[1])
xa, ya, xb, yb = line_cls(
    -5,
    5,
    softneu.coef[1:, 1:] - softneu.coef[:1, 1:],
    softneu.bias[1] - softneu.bias[0],
)

[33]:

fig, ax = plt.subplots(1, 2, figsize=(14, 6))
for i in [0, 1]:
    ax[i].scatter(clsX[clsy == 0, 0], clsX[clsy == 0, 1], label="cl0")
    ax[i].scatter(clsX[clsy == 1, 0], clsX[clsy == 1, 1], label="cl1")
    ax[i].scatter(clsX[pred1 == 0, 0] + h, clsX[pred1 == 0, 1] + h, label="LR0")
    ax[i].scatter(clsX[pred1 == 1, 0] + h, clsX[pred1 == 1, 1] + h, label="LR1")
    ax[i].scatter(clsX[pred2 == 0, 0] + h, clsX[pred2 == 0, 1] - h, label="NN0")
    ax[i].scatter(clsX[pred2 == 1, 0] + h, clsX[pred2 == 1, 1] - h, label="NN1")
ax[0].plot([x0, x1], [y0, y1], "y--", lw=4, label="frontière LR")
ax[1].plot([x00, x01], [y00, y01], "r--", lw=4, label="droite neurone 0")
ax[1].plot([x10, x11], [y10, y11], "b--", lw=4, label="droite neurone 1")
ax[0].plot([xa, xb], [ya, yb], "c--", lw=4, label="frontière neurone")
ax[0].set_ylim(
    [max(-6, min([-3, y10, y11, y11, y01])), min(6, max([3, y10, y11, y11, y01]))]
)
ax[1].set_ylim(
    [max(-6, min([-3, y10, y11, y11, y01])), min(6, max([3, y10, y11, y11, y01]))]
)
ax[0].legend()
ax[1].legend()
ax[0].set_title("Frontière de classification")
ax[1].set_title("Neurones");
../../_images/notebooks_ml_neural_tree_49_0.png

Ca marche. On vérifie en calculant le score. Le neurone a deux sorties. La frontière est définie par l’ensemble des points pour lesquels les deux sorties sont égales. Par conséquent, la distance entre les deux droites définies par les coefficients du neurone doivent être égales. Il existe une infinité de solutions menant à la même frontière. On pourrait pénaliser les coefficients pour converger toujours vers la même solution.

[34]:

from sklearn.metrics import roc_auc_score

roc_auc_score(clsy, logr.predict_proba(clsX)[:, 1])

[34]:

np.float64(0.9896)

[35]:

roc_auc_score(clsy, softneu.predict(clsX)[:, 1])

[35]:

np.float64(0.9871999999999999)

La performance est quasiment identique. Que ce soit la régression ou la classification, l’apprentissage d’un neurone fonctionne. En sera-t-il de même pour un assemblage de neurones ?

Apprentissage du réseau de neurones#

Maintenant qu’on a vu les différentes fonctions d’activations et leur application sur des problèmes simples, on revient aux arbres convertis sous la forme d’un réseau de neurones. La prochaine étape est de pouvoir améliorer les performances du modèle issu de la conversion d’un arbre de classification avec un algorithme du gradient. On construit pour cela un nuage de points un peu traficoté.

[36]:

clsX = numpy.empty((150, 2), dtype=numpy.float64)
clsX[:100] = numpy.random.randn(100, 2)
clsX[:20, 0] -= 1
clsX[20:40, 0] -= 0.8
clsX[:100, 1] /= 2
clsX[:100, 1] += clsX[:100, 0] ** 2
clsX[100:] = numpy.random.randn(50, 2)
clsX[100:, 0] /= 2
clsX[100:, 1] += 2.5
clsy = numpy.zeros(X.shape[0], dtype=numpy.int64)
clsy[100:] = 1

logr = LogisticRegression()
logr.fit(clsX, clsy)
pred1 = logr.predict(clsX)
logr.score(clsX, clsy)

[36]:

0.6666666666666666

[37]:

x0, y0, x1, y1 = line_cls(-3, 3, logr.coef_, logr.intercept_)

[38]:

fig, ax = plt.subplots(1, 1, figsize=(4, 4))
plot_grid(clsX, clsy, logr.predict, logr.__class__.__name__, ax=ax)
ax.plot([x0, x1], [y0, y1], "y--", lw=4, label="frontière LR");
../../_images/notebooks_ml_neural_tree_57_0.png

Même chose avec un arbre de décision et le réseau de neurones converti.

[39]:

dec = DecisionTreeClassifier(max_depth=2)
dec.fit(clsX, clsy)
pred2 = dec.predict(clsX)
dec.score(clsX, clsy)

[39]:

0.88

On convertit de réseau de neurones. Le second argument définit la pente dans la fonction d’activation.

[40]:

net = NeuralTreeNet.create_from_tree(dec, 0.5)
net15 = NeuralTreeNet.create_from_tree(dec, 15)

[41]:

from sklearn.metrics import accuracy_score

[42]:

(
    roc_auc_score(clsy, dec.predict_proba(clsX)[:, 1]),
    accuracy_score(clsy, dec.predict(clsX)),
)

[42]:

(np.float64(0.8886), 0.88)

[43]:

(
    roc_auc_score(clsy, net.predict(clsX)[:, -1]),
    accuracy_score(clsy, numpy.argmax(net.predict(clsX)[:, -2:], axis=1)),
)

[43]:

(np.float64(0.8550000000000001), 0.6933333333333334)

[44]:

(
    roc_auc_score(clsy, net15.predict(clsX)[:, -1]),
    accuracy_score(clsy, numpy.argmax(net15.predict(clsX)[:, -2:], axis=1)),
)

[44]:

(np.float64(0.8956000000000001), 0.88)

Le réseau de neurones est plus ou moins performant selon la pente dans la fonction d’activation.

[45]:

fig, ax = plt.subplots(1, 3, figsize=(15, 4))
plot_grid(clsX, clsy, dec.predict, dec.__class__.__name__, ax=ax[0])
plot_grid(
    clsX,
    clsy,
    lambda x: numpy.argmax(net.predict(x)[:, -2:], axis=1),
    net.__class__.__name__,
    ax=ax[1],
)
plot_grid(
    clsX,
    clsy,
    lambda x: numpy.argmax(net15.predict(x)[:, -2:], axis=1),
    net15.__class__.__name__ + " 15",
    ax=ax[2],
)
../../_images/notebooks_ml_neural_tree_67_0.png

Et on apprend le réseau de neurones en partant de l’arbre de départ. On choisit celui qui a la pente d’activation la plus faible.

[46]:

from mlstatpy.ml.neural_tree import label_class_to_softmax_output

clsY = label_class_to_softmax_output(clsy)
clsY[:3]

[46]:

array([[1., 0.],
       [1., 0.],
       [1., 0.]])

[47]:

net2 = net.copy()
net2.fit(clsX, clsY, verbose=True, max_iter=25, lr=3e-6)

0/25: loss: 733.9 lr=3e-06 max(coef): 1 l1=0/15 l2=0/10
1/25: loss: 718.5 lr=2.44e-07 max(coef): 1 l1=1.8e+02/15 l2=3.2e+03/9.8
2/25: loss: 717.2 lr=1.73e-07 max(coef): 1 l1=2.6e+02/15 l2=3.9e+03/9.8
3/25: loss: 716.7 lr=1.41e-07 max(coef): 1 l1=1.9e+02/15 l2=3.3e+03/9.8
4/25: loss: 716.2 lr=1.22e-07 max(coef): 1 l1=2.4e+02/15 l2=4.1e+03/9.8
5/25: loss: 716 lr=1.09e-07 max(coef): 1.1 l1=1.9e+02/15 l2=3.2e+03/9.8
6/25: loss: 715.7 lr=9.99e-08 max(coef): 1.1 l1=2e+02/15 l2=3.1e+03/9.8
7/25: loss: 715.4 lr=9.25e-08 max(coef): 1.1 l1=2.8e+02/15 l2=4.3e+03/9.8
8/25: loss: 715.2 lr=8.66e-08 max(coef): 1.1 l1=3.3e+02/15 l2=9.2e+03/9.8
9/25: loss: 715 lr=8.16e-08 max(coef): 1.1 l1=2.8e+02/15 l2=4.1e+03/9.8
10/25: loss: 714.8 lr=7.74e-08 max(coef): 1.1 l1=1.9e+02/15 l2=3.4e+03/9.8
11/25: loss: 714.7 lr=7.38e-08 max(coef): 1.1 l1=3e+02/15 l2=4.9e+03/9.8
12/25: loss: 714.5 lr=7.07e-08 max(coef): 1.1 l1=2.8e+02/15 l2=4.2e+03/9.8
13/25: loss: 714.3 lr=6.79e-08 max(coef): 1.1 l1=2.2e+02/15 l2=4.2e+03/9.8
14/25: loss: 714.3 lr=6.54e-08 max(coef): 1.1 l1=2.9e+02/15 l2=3.9e+03/9.8
15/25: loss: 714.2 lr=6.32e-08 max(coef): 1.1 l1=3e+02/15 l2=7.1e+03/9.8
16/25: loss: 714 lr=6.12e-08 max(coef): 1.1 l1=2.7e+02/15 l2=4.1e+03/9.8
17/25: loss: 713.9 lr=5.94e-08 max(coef): 1.1 l1=2.9e+02/15 l2=4.5e+03/9.8
18/25: loss: 713.9 lr=5.77e-08 max(coef): 1.1 l1=2.3e+02/15 l2=3.4e+03/9.8
19/25: loss: 713.8 lr=5.62e-08 max(coef): 1.1 l1=2e+02/15 l2=3.2e+03/9.8
20/25: loss: 713.8 lr=5.48e-08 max(coef): 1.1 l1=2.8e+02/15 l2=4.2e+03/9.8
21/25: loss: 713.8 lr=5.34e-08 max(coef): 1.1 l1=2.8e+02/15 l2=4.2e+03/9.8
22/25: loss: 713.8 lr=5.22e-08 max(coef): 1.1 l1=2e+02/15 l2=3.4e+03/9.8
23/25: loss: 713.7 lr=5.11e-08 max(coef): 1.1 l1=2.9e+02/15 l2=4e+03/9.8
24/25: loss: 713.7 lr=5e-08 max(coef): 1.1 l1=2.8e+02/15 l2=3.9e+03/9.8
25/25: loss: 713.6 lr=4.9e-08 max(coef): 1.1 l1=2e+02/15 l2=3.5e+03/9.8

[47]:

NeuralTreeNet(2)

[48]:

fig, ax = plt.subplots(1, 2, figsize=(12, 4))
plot_grid(
    clsX,
    clsy,
    lambda x: numpy.argmax(net.predict(x)[:, -2:], axis=1),
    "Avant apprentissage",
    ax=ax[0],
)
plot_grid(
    clsX,
    clsy,
    lambda x: numpy.argmax(net2.predict(x)[:, -2:], axis=1),
    "Après apprentissage",
    ax=ax[1],
)
../../_images/notebooks_ml_neural_tree_71_0.png

Ca ne marche pas ou pas très bien. Il faudrait vérifier que la configuration actuelle ne se trouve pas dans un minimum local auquel cas l’apprentissage par gradient ne donnera quasiment rien.

[49]:

(
    roc_auc_score(clsy, net2.predict(clsX)[:, -1]),
    accuracy_score(clsy, numpy.argmax(net2.predict(clsX)[:, -2:], axis=1)),
)

[49]:

(np.float64(0.8514), 0.6666666666666666)

[50]:

net2.predict(clsX)[-5:, -2:]

[50]:

array([[0.60888966, 0.39111034],
       [0.64307934, 0.35692066],
       [0.55569147, 0.44430853],
       [0.67979624, 0.32020376],
       [0.93106273, 0.06893727]])

[51]:

net.predict(clsX)[-5:, -2:]

[51]:

array([[0.54506228, 0.45493772],
       [0.58240467, 0.41759533],
       [0.48917761, 0.51082239],
       [0.62370674, 0.37629326],
       [0.88382805, 0.11617195]])

On peut essayer de repartir à zéro. Des fois ça peut marcher mais il faudrait beaucoup plus d’essai.

[52]:

net3 = net.copy()
dim = net3.training_weights.shape
net3.update_training_weights(numpy.random.randn(dim[0]))
net3.fit(clsX, clsY, verbose=True, max_iter=25, lr=3e-6)

0/25: loss: 881.1 lr=3e-06 max(coef): 2.2 l1=0/32 l2=0/44
1/25: loss: 855.3 lr=2.44e-07 max(coef): 2.2 l1=1.3e+03/31 l2=1.8e+05/43
2/25: loss: 852.3 lr=1.73e-07 max(coef): 2.2 l1=4.7e+02/31 l2=2.1e+04/43
3/25: loss: 849.6 lr=1.41e-07 max(coef): 2.2 l1=4.4e+02/31 l2=1.8e+04/43
4/25: loss: 847.5 lr=1.22e-07 max(coef): 2.2 l1=3.4e+02/31 l2=6.4e+03/43
5/25: loss: 845.5 lr=1.09e-07 max(coef): 2.2 l1=6e+02/31 l2=2.8e+04/43
6/25: loss: 843.4 lr=9.99e-08 max(coef): 2.2 l1=2.9e+02/31 l2=5.9e+03/43
7/25: loss: 842 lr=9.25e-08 max(coef): 2.2 l1=7.2e+02/31 l2=5.5e+04/43
8/25: loss: 840.9 lr=8.66e-08 max(coef): 2.2 l1=6.8e+02/31 l2=3.6e+04/43
9/25: loss: 839.7 lr=8.16e-08 max(coef): 2.2 l1=3.7e+02/31 l2=8.7e+03/43
10/25: loss: 838 lr=7.74e-08 max(coef): 2.2 l1=1.3e+03/31 l2=1.8e+05/43
11/25: loss: 836.8 lr=7.38e-08 max(coef): 2.2 l1=4.2e+02/31 l2=1.7e+04/43
12/25: loss: 835.9 lr=7.07e-08 max(coef): 2.2 l1=5.7e+02/31 l2=4.1e+04/43
13/25: loss: 835.2 lr=6.79e-08 max(coef): 2.2 l1=9.1e+02/31 l2=7.7e+04/43
14/25: loss: 834 lr=6.54e-08 max(coef): 2.2 l1=9.2e+02/31 l2=7.5e+04/43
15/25: loss: 833.7 lr=6.32e-08 max(coef): 2.2 l1=3.6e+02/31 l2=8.4e+03/43
16/25: loss: 833.1 lr=6.12e-08 max(coef): 2.2 l1=4.9e+02/31 l2=2.2e+04/43
17/25: loss: 832.2 lr=5.94e-08 max(coef): 2.2 l1=3.3e+02/31 l2=6.4e+03/43
18/25: loss: 831.3 lr=5.77e-08 max(coef): 2.2 l1=4.5e+02/31 l2=1.5e+04/43
19/25: loss: 830.3 lr=5.62e-08 max(coef): 2.2 l1=5.2e+02/31 l2=3.2e+04/43
20/25: loss: 829.4 lr=5.48e-08 max(coef): 2.2 l1=9.5e+02/31 l2=8.5e+04/43
21/25: loss: 828.9 lr=5.34e-08 max(coef): 2.2 l1=7.4e+02/31 l2=4.1e+04/43
22/25: loss: 828.3 lr=5.22e-08 max(coef): 2.2 l1=5.4e+02/31 l2=2.2e+04/43
23/25: loss: 827.5 lr=5.11e-08 max(coef): 2.2 l1=1e+03/31 l2=1e+05/43
24/25: loss: 826.8 lr=5e-08 max(coef): 2.2 l1=5.1e+02/31 l2=2.4e+04/43
25/25: loss: 826 lr=4.9e-08 max(coef): 2.2 l1=3.1e+02/31 l2=6.8e+03/43

[52]:

NeuralTreeNet(2)

[53]:

(
    roc_auc_score(clsy, net3.predict(clsX)[:, -1]),
    accuracy_score(clsy, numpy.argmax(net3.predict(clsX)[:, -2:], axis=1)),
)

[53]:

(np.float64(0.7857999999999999), 0.6666666666666666)

[54]:

fig, ax = plt.subplots(1, 2, figsize=(12, 4))
plot_grid(
    clsX,
    clsy,
    lambda x: numpy.argmax(net.predict(x)[:, -2:], axis=1),
    "Avant apprentissage",
    ax=ax[0],
)
plot_grid(
    clsX,
    clsy,
    lambda x: numpy.argmax(net3.predict(x)[:, -2:], axis=1),
    "Après apprentissage",
    ax=ax[1],
)
../../_images/notebooks_ml_neural_tree_79_0.png

Autre architecture#

Cette fois-ci, on réduit le nombre de neurones. Au lieu d’avoir deux neurones par noeud du graphe, on assemble tous les neurones en deux : un pour les entrées, un autre pour le calcul des sorties. Les deux représentations ne sont pas implémentées de façon rigoureusement identique dans le module mlstatpy. Le code précise les différences.

[55]:

netc = NeuralTreeNet.create_from_tree(dec, 1, arch="compact")
RenderJsDot(netc.to_dot())

[55]:

[56]:

(
    roc_auc_score(clsy, netc.predict(clsX)[:, -1]),
    accuracy_score(clsy, numpy.argmax(netc.predict(clsX)[:, -2:], axis=1)),
)

[56]:

(np.float64(0.8584), 0.32666666666666666)

[57]:

fig, ax = plt.subplots(1, 2, figsize=(12, 4))
plot_grid(
    clsX,
    clsy,
    lambda x: numpy.argmax(dec.predict_proba(x), axis=1),
    "Avant conversion",
    ax=ax[0],
)
plot_grid(
    clsX,
    clsy,
    lambda x: numpy.argmax(netc.predict(x)[:, -2:], axis=1),
    "Après comversion",
    ax=ax[1],
)
../../_images/notebooks_ml_neural_tree_83_0.png

On réapprend.

[58]:

netc4 = netc.copy()
netc4.fit(clsX, clsY, verbose=True, max_iter=25, lr=1e-6)

0/25: loss: 1224 lr=1e-06 max(coef): 1.8 l1=0/27 l2=0/28
1/25: loss: 853.2 lr=8.14e-08 max(coef): 1.8 l1=8.1e+02/27 l2=4.2e+04/28
2/25: loss: 812.8 lr=5.76e-08 max(coef): 1.8 l1=2.8e+02/27 l2=6.1e+03/28
3/25: loss: 794.9 lr=4.71e-08 max(coef): 1.8 l1=6.6e+02/27 l2=2.9e+04/28
4/25: loss: 783.6 lr=4.08e-08 max(coef): 1.8 l1=5.6e+02/27 l2=1.5e+04/28
5/25: loss: 771.7 lr=3.65e-08 max(coef): 1.8 l1=6e+02/27 l2=1.9e+04/28
6/25: loss: 763 lr=3.33e-08 max(coef): 1.8 l1=1.3e+02/27 l2=1.8e+03/28
7/25: loss: 755.1 lr=3.08e-08 max(coef): 1.8 l1=5.3e+02/27 l2=1.5e+04/28
8/25: loss: 748.3 lr=2.89e-08 max(coef): 1.8 l1=6.1e+02/27 l2=1.9e+04/28
9/25: loss: 741.3 lr=2.72e-08 max(coef): 1.8 l1=1.3e+03/27 l2=2.5e+05/28
10/25: loss: 736 lr=2.58e-08 max(coef): 1.8 l1=5.9e+02/27 l2=1.8e+04/28
11/25: loss: 729.2 lr=2.46e-08 max(coef): 1.8 l1=6.1e+02/27 l2=1.9e+04/28
12/25: loss: 723 lr=2.36e-08 max(coef): 1.8 l1=4.8e+02/27 l2=1e+04/28
13/25: loss: 718.7 lr=2.26e-08 max(coef): 1.8 l1=1e+03/27 l2=7.8e+04/28
14/25: loss: 713.8 lr=2.18e-08 max(coef): 1.8 l1=2.5e+02/27 l2=4.8e+03/28
15/25: loss: 709 lr=2.11e-08 max(coef): 1.8 l1=1.3e+03/27 l2=2.4e+05/28
16/25: loss: 705.2 lr=2.04e-08 max(coef): 1.8 l1=6.1e+02/27 l2=1.9e+04/28
17/25: loss: 701 lr=1.98e-08 max(coef): 1.8 l1=6.9e+02/27 l2=2.8e+04/28
18/25: loss: 696.8 lr=1.92e-08 max(coef): 1.8 l1=6.2e+02/27 l2=2e+04/28
19/25: loss: 693.1 lr=1.87e-08 max(coef): 1.8 l1=4.9e+02/27 l2=1.2e+04/28
20/25: loss: 689.5 lr=1.83e-08 max(coef): 1.8 l1=4.6e+02/27 l2=1.3e+04/28
21/25: loss: 686.3 lr=1.78e-08 max(coef): 1.8 l1=1.2e+03/27 l2=2e+05/28
22/25: loss: 683.7 lr=1.74e-08 max(coef): 1.8 l1=1.3e+02/27 l2=2.2e+03/28
23/25: loss: 680.5 lr=1.7e-08 max(coef): 1.8 l1=6e+02/27 l2=1.8e+04/28
24/25: loss: 677.4 lr=1.67e-08 max(coef): 1.8 l1=1.2e+03/27 l2=2e+05/28
25/25: loss: 674.6 lr=1.63e-08 max(coef): 1.8 l1=5.1e+02/27 l2=1.3e+04/28

[58]:

NeuralTreeNet(2)

[59]:

(
    roc_auc_score(clsy, netc4.predict(clsX)[:, -1]),
    accuracy_score(clsy, numpy.argmax(netc4.predict(clsX)[:, -2:], axis=1)),
)

[59]:

(np.float64(0.858), 0.8)

[60]:

fig, ax = plt.subplots(1, 2, figsize=(12, 4))
plot_grid(
    clsX,
    clsy,
    lambda x: numpy.argmax(netc.predict(x)[:, -2:], axis=1),
    "Avant apprentissage",
    ax=ax[0],
)
plot_grid(
    clsX,
    clsy,
    lambda x: numpy.argmax(netc4.predict(x)[:, -2:], axis=1),
    "Après apprentissage",
    ax=ax[1],
)
../../_images/notebooks_ml_neural_tree_87_0.png

C’est mieux…

[62]:
Sur cette page
Montrer le code source