ReLU Variants

ReLU and Leaky ReLU variants for PyTorch.

This module provides several simple ReLU-based activation functions.

activations_plus.simple.relu_variants.dual_line(x: Tensor, a: float = 0.5, b: float = 0.5) → Tensor[source]

Apply the Dual Line activation function.

\[\begin{split}\text{DualLine}(x) = \begin{cases} a \cdot x, & \text{if } x < 0 \\ b \cdot x, & \text{if } x \geq 0 \end{cases}\end{split}\]

Parameters:

x (torch.Tensor) – Input tensor.
a (float, optional) – Negative slope coefficient (default 0.5).
b (float, optional) – Positive slope coefficient (default 0.5).

Returns:

The element-wise Dual Line of the input.

Return type:

torch.Tensor

Source

See also

A generalized linear activation function discussed in “Survey of Activation Functions for Deep Neural Networks” by Nwankpa et al. (2018).

arxiv

Example

import matplotlib.pyplot as plt
import torch

from activations_plus.simple import dual_line

x = torch.linspace(-3, 3, 200)
y = dual_line(x, a=1.0, b=0.01)
fig, ax = plt.subplots()
ax.plot(x.numpy(), y.numpy())
ax.set_title("Dual Line (a=1.0, b=0.01)")
ax.set_xlabel("Input")
ax.set_ylabel("Output")
ax.grid(alpha=0.3)
fig.show()  # This will be mocked in tests

(Source code, png, hires.png, pdf)