extorch.nn.modules.loss

HintonKDLoss

Knowledge distillation loss proposed by Hinton (`Link`_).

CrossEntropyLabelSmooth

CrossEntropyMixupLoss

CrossEntropyLoss with mixup technique.

DECLoss

Loss used by Deep Embedded Clustering (DEC, `Link`_).
class extorch.nn.modules.loss.CrossEntropyLabelSmooth(epsilon: float)[source]

Bases: torch.nn.modules.module.Module

forward(input: torch.Tensor, target: torch.Tensor) → torch.Tensor[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool
class extorch.nn.modules.loss.CrossEntropyMixupLoss(alpha: float = 1.0, **kwargs)[source]

Bases: torch.nn.modules.module.Module

CrossEntropyLoss with mixup technique.

Parameters

  • alpha (float) – Parameter of the beta distribution. Default: 1.0.

  • kwargs – Other arguments of torch.nn.CrossEntropyLoss (`Link`_).

forward(input: torch.Tensor, target: torch.Tensor, net: torch.nn.modules.module.Module) → torch.Tensor[source]
Parameters

  • input (Tensor) – Input examples.

  • target (Tensor) – Label of the input examples.

  • net (nn.Module) – Network to calculate the loss.

Returns

The loss.

Return type

loss (Tensor)

training: bool
class extorch.nn.modules.loss.DECLoss(alpha: float = 1.0, **kwargs)[source]

Bases: torch.nn.modules.module.Module

Loss used by Deep Embedded Clustering (DEC, `Link`_).

Parameters

alpha (float) – The degrees of freedom of the Student’s tdistribution. Default: 1.0.

Examples::
>>> criterion = DECLoss(alpha = 1.)
>>> embeddings = torch.randn((2, 10))
>>> centers = torch.randn((3, 10))
>>> loss = criterion(embeddings, centers)
forward(input: torch.Tensor, centers: torch.Tensor) → torch.Tensor[source]

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

static target_distribution(input: torch.Tensor) → torch.Tensor[source]
training: bool
class extorch.nn.modules.loss.HintonKDLoss(T: float, alpha: float, reduction: str = 'mean', **kwargs)[source]

Bases: torch.nn.modules.loss.KLDivLoss

Knowledge distillation loss proposed by Hinton (`Link`_).

$L = (1 - alpha) * L_{CE}(P_s, Y_{gt}) + alpha * T^2 * L_{CE}(P_s, P_t)$

Parameters

  • T (float) – Temperature parameter (>= 1.) used to smooth the softmax output.

  • alpha (float) – Trade-off coefficient between distillation and origin loss.

  • reduction (str) – Specifies the reduction to apply to the output. Default: “mean”.

  • kwargs – Other configurations for nn.CrossEntropyLoss.

Examples::
>>> criterion = HintonKDLoss(T = 4., alpha = 0.9)
>>> s_output = torch.randn((5, 10))
>>> t_output = torch.randn((5, 10))
>>> target = torch.ones(5, dtype = torch.long)
>>> loss = criterion(s_output, t_output, target)
forward(s_output: torch.Tensor, t_output: torch.Tensor, target: torch.Tensor) → torch.Tensor[source]
Parameters

  • s_output (Tensor) – Student network output.

  • t_output (Tensor) – Teacher network output.

  • target (Tensor) – Hard label of the input.

Returns

The calculated loss.

Return type

Tensor

reduction: str