extorch.nn.optim.lr_scheduler

CosineWithRestarts

Cosine annealing with restarts (Link).
class extorch.nn.optim.lr_scheduler.CosineWithRestarts(optimizer: torch.optim.optimizer.Optimizer, T_max: int, eta_min: float = 0.0, last_epoch: int = - 1, factor: float = 1.0)[source]

Bases: torch.optim.lr_scheduler._LRScheduler

Cosine annealing with restarts (Link).

Parameters

  • optimizer (Optimizer) – The optimizer.

  • t_max (int) – The maximum number of iterations within the first cycle.

  • eta_min (float) – The minimum learning rate. Default: 0.

  • last_epoch (int) – The index of the last epoch. This is used when restarting. Default: -1.

  • factor (float) – The factor by which the cycle length (T_max) increases after each restart. Default: 1.

get_lr() → List[float][source]

Get updated learning rate.

Returns

A list of current learning rates.

Return type

lrs (List[float])

Note

We need to check if this is the first time self.get_lr() was called, since torch.optim.lr_scheduler._LRScheduler will call self.get_lr() when first initialized, but the learning rate should remain unchanged for the first epoch.