mlstatpy.optim.sgd

class mlstatpy.optim.sgd.BaseOptimizer(coef, learning_rate_init=0.1, min_threshold=None, max_threshold=None, l1=0.0, l2=0.0)

Base stochastic gradient descent optimizer.

Paramètres:

• coef – array, initial coefficient

• learning_rate_init – float The initial learning rate used. It controls the step-size in updating the weights.

• min_threshold – coefficients must be higher than min_thresold

• max_threshold – coefficients must be below than max_thresold

The class holds the following attributes:

• learning_rate: float, the current learning rate

• coef: optimized coefficients

• min_threshold, max_threshold: coefficients thresholds

• l2: L2 regularization

• l1: L1 regularization

iteration_ends(time_step)

Performs update to learning rate and potentially other states at the end of an iteration.

train(X, y, fct_loss, fct_grad, max_iter=100, early_th=None, verbose=False)

Optimizes the coefficients.

Paramètres:

• X – datasets (array)

• y – expected target

• fct_loss – loss function, signature: f(coef, X, y) -> float

• fct_grad – gradient function, signature: g(coef, x, y, i) -> array

• max_iter – number maximum of iteration

• early_th – stops the training if the error goes below this threshold

• verbose – display information

Renvoie:

loss

The method keeps the best coefficients for the minimal loss.

update_coef(grad)

Updates coefficients with given gradient.

Paramètres:

grad – array, gradient

class mlstatpy.optim.sgd.SGDOptimizer(coef, learning_rate_init=0.1, lr_schedule='invscaling', momentum=0.9, power_t=0.5, early_th=None, min_threshold=None, max_threshold=None, l1=0.0, l2=0.0)

Stochastic gradient descent optimizer with momentum.

Paramètres:

• coef – array, initial coefficient

• learning_rate_init – float The initial learning rate used. It controls the step-size in updating the weights,

• lr_schedule – {“constant”, “adaptive”, “invscaling”}, learning rate schedule for weight updates, “constant” for a constant learning rate given by learning_rate_init. “invscaling” gradually decreases the learning rate learning_rate_ at each time step t using an inverse scaling exponent of power_t. learning_rate_ = learning_rate_init / pow(t, power_t), “adaptive”, keeps the learning rate constant to learning_rate_init as long as the training keeps decreasing. Each time 2 consecutive epochs fail to decrease the training loss by tol, or fail to increase validation score by tol if “early_stopping” is on, the current learning rate is divided by 5.

• momentum – float Value of momentum used, must be larger than or equal to 0

• power_t – double The exponent for inverse scaling learning rate.

• early_th – stops if the error goes below that threshold

• min_threshold – lower bound for parameters (can be None)

• max_threshold – upper bound for parameters (can be None)

• l1 – L1 regularization

• l2 – L2 regularization

The class holds the following attributes:

• learning_rate: float, the current learning rate

• velocity*: array, velocity that are used to update params

Stochastic Gradient Descent applied to linear regression

The following example how to optimize a simple linear regression.

<<<

import numpy
from mlstatpy.optim import SGDOptimizer


def fct_loss(c, X, y):
    return numpy.linalg.norm(X @ c - y) ** 2


def fct_grad(c, x, y, i=0):
    return x * (x @ c - y) * 0.1


coef = numpy.array([0.5, 0.6, -0.7])
X = numpy.random.randn(10, 3)
y = X @ coef

sgd = SGDOptimizer(numpy.random.randn(3))
sgd.train(X, y, fct_loss, fct_grad, max_iter=15, verbose=True)
print("optimized coefficients:", sgd.coef)

>>>

    0/15: loss: 13.1 lr=0.1 max(coef): 0.6 l1=0/1.5 l2=0/0.77
    1/15: loss: 3.101 lr=0.0302 max(coef): 0.6 l1=0.058/1.2 l2=0.0013/0.69
    2/15: loss: 0.5244 lr=0.0218 max(coef): 0.71 l1=0.088/1.5 l2=0.0028/0.8
    3/15: loss: 0.5874 lr=0.018 max(coef): 0.72 l1=0.016/1.5 l2=9.7e-05/0.87
    4/15: loss: 0.5043 lr=0.0156 max(coef): 0.71 l1=0.0098/1.6 l2=3.9e-05/0.89
    5/15: loss: 0.4271 lr=0.014 max(coef): 0.71 l1=0.03/1.6 l2=0.00041/0.9
    6/15: loss: 0.3292 lr=0.0128 max(coef): 0.7 l1=0.059/1.6 l2=0.0012/0.92
    7/15: loss: 0.2226 lr=0.0119 max(coef): 0.69 l1=0.0043/1.6 l2=7.5e-06/0.92
    8/15: loss: 0.1844 lr=0.0111 max(coef): 0.69 l1=0.003/1.6 l2=3.6e-06/0.94
    9/15: loss: 0.1643 lr=0.0105 max(coef): 0.68 l1=0.032/1.6 l2=0.00035/0.95
    10/15: loss: 0.1488 lr=0.00995 max(coef): 0.68 l1=0.028/1.6 l2=0.00027/0.95
    11/15: loss: 0.1346 lr=0.00949 max(coef): 0.67 l1=0.034/1.7 l2=0.00042/0.95
    12/15: loss: 0.121 lr=0.00909 max(coef): 0.67 l1=0.035/1.7 l2=0.00045/0.96
    13/15: loss: 0.1124 lr=0.00874 max(coef): 0.67 l1=0.0077/1.7 l2=2.3e-05/0.96
    14/15: loss: 0.1031 lr=0.00842 max(coef): 0.66 l1=0.066/1.7 l2=0.0022/0.97
    15/15: loss: 0.09233 lr=0.00814 max(coef): 0.66 l1=0.026/1.7 l2=0.00028/0.98
    optimized coefficients: [ 0.409  0.664 -0.609]

iteration_ends(time_step)

Performs updates to learning rate and potential other states at the end of an iteration.

Paramètres:

time_step – int number of training samples trained on so far, used to update learning rate for “invscaling”