Mixed Precision Trainings

DynamicLossScalingUpdater

class nnabla.experimental.mixed_precision_training.DynamicLossScalingUpdater(solver, loss, data_feeder=<function DynamicLossScalingUpdater.<lambda>>, scale=8.0, scaling_factor=2.0, N=2000, clear_buffer=True, accum_grad=1, weight_decay=None, comm=None, grads=[])

Dynamic Loss Scaling Updater for the mixed precision training.

Parameters:

• solver (nnabla.solvers.Solver) – Solver object. E.g., Momentum or Adam.

• loss (nnabla.Variable) – Loss variable from which the forward and the backward is called.

• data_feeder (callable object, function, or lambda) – Data feeder

• scale (float) – Loss scale constant. This is dynamically changing during training.

• scaling_factor (float) – Scaling factor for the dynamic loss scaling.

• N (int) – Interval, the number of iterations in training for increasing loss scale by scaling_factor.

• clear_buffer (bool) – Clears the no longer referenced variables during backpropagation to save memory.

• accum_grad (int) – Number of accumulation of gradients. Update method of the Solver is called after the accum_grad number of the forward and backward is called.

• weight_decay (float) – Decay constant. Default is None, not applying the weight decay.

• comm (nnabla.communicators.Communicator) – Communicator when to do distributed training. Default is None.

• grads (list of nnabla.NdArray) – The list of gradients to be exchanged when to do distributed training. Default is the empty list.

solver

Solver object. E.g., Momentum or Adam.

Type:

nnabla.solvers.Solver

loss

Loss variable from which the forward and the backward is called.

Type:

nnabla.Variable

data_feeder

Data feeder

Type:

callable object, function, lambda

scale

Loss scale constant. This is dynamically changing during training.

Type:

float

scaling_factor

Scaling factor for the dynamic loss scaling.

Type:

float

N

Interval, the number of iterations in training for increasing loss scale by scaling_factor.

Type:

int

clear_buffer

Clears the no longer referenced variables during backpropagation to save memory.

Type:

bool

accum_grad

Number of accumulation of gradients. Update method of the Solver is called after the accum_grad number of the forward and backward is called.

Type:

int

weight_decay

Decay constant. Default is None, not applying the weight decay.

Type:

float

comm

Communicator when to do distributed training.

Type:

nnabla.communicators.Communicator

grads

The list of gradients to be exchanged when to do distributed training.

Type:

list of nnabla.NdArray

Example

Reference:

https://docs.nvidia.com/deeplearning/sdk/mixed-precision-training/index.html#scalefactor

update()

Monolithic update method.

This method calls the following methods with the dynamic loss scaling.

1. solver.zerograd

2. feed data

3. loss.forward

4. loss.backward

5. comm.all_reduce (if it is specified)

6. solver.update