ELU Variants

ELU-based activation functions and their variants for neural networks.

activations_plus.simple.elu_variants.isrlu(x: Tensor, alpha: float = 1.0) → Tensor[source]

Apply the Inverse Square Root Linear Unit activation function.

\[\begin{split}\text{ISRLU}(x) = \begin{cases} x, & x \geq 0 \\ \frac{x}{\sqrt{1 + \alpha x^2}}, & x < 0 \end{cases}\end{split}\]
Parameters:

  • x (torch.Tensor) – Input tensor.

  • alpha (float, optional) – Scaling parameter (default 1.0).

Returns:

The element-wise ISRLU of the input.

Return type:

torch.Tensor

Source

See also

Proposed in “Improving Deep Neural Networks with New Activation Functions” by Carlile et al. (2017).

arxiv

Example

import matplotlib.pyplot as plt
import torch

from activations_plus.simple import isrlu

x = torch.linspace(-3, 3, 200)
y = isrlu(x)
fig, ax = plt.subplots()
ax.plot(x.numpy(), y.numpy())
ax.set_title("ISRLU (Inverse Square Root Linear Unit)")
ax.set_xlabel("Input")
ax.set_ylabel("Output")
ax.grid(alpha=0.3)
fig.show()  # This will be mocked in tests

(Source code, png, hires.png, pdf)

../_images/isrlu_example.png
activations_plus.simple.elu_variants.pelu(x: Tensor, alpha: float = 1.0, beta: float = 1.0) → Tensor[source]

Parametric Exponential Linear Unit (PELU) activation function.

\[\begin{split}\text{PELU}(x) = \begin{cases} \frac{\alpha}{\beta} \times x, & \text{if } x \geq 0 \\ \alpha \times (e^{x/\beta} - 1), & \text{if } x < 0 \end{cases}\end{split}\]
Parameters:

  • x (torch.Tensor) – Input tensor.

  • alpha (float, optional) – The alpha parameter for scaling (default 1.0).

  • beta (float, optional) – The beta parameter for controlling the negative part (default 1.0).

Returns:

Output tensor with the PELU activation applied.

Return type:

torch.Tensor

Source

See also

Proposed in “Parametric exponential linear unit for deep convolutional neural networks” by Trottier et al. (2016).

arxiv

Example

"""Example demonstrating the PELU activation function."""

import matplotlib.pyplot as plt
import torch

from activations_plus.simple import pelu


def main() -> None:
    """Plot the PELU activation function with different parameter values."""
    x = torch.linspace(-5, 5, 1000)

    # Different parameter combinations
    params = [
        {"alpha": 1.0, "beta": 1.0, "label": "α=1.0, β=1.0 (default)"},
        {"alpha": 1.5, "beta": 1.0, "label": "α=1.5, β=1.0"},
        {"alpha": 1.0, "beta": 1.5, "label": "α=1.0, β=1.5"},
        {"alpha": 1.5, "beta": 1.5, "label": "α=1.5, β=1.5"},
    ]

    # Create the plot
    plt.figure(figsize=(10, 6))

    # Plot PELU with different parameter combinations
    for param in params:
        y_pelu = pelu(x, alpha=param["alpha"], beta=param["beta"])
        plt.plot(x.numpy(), y_pelu.numpy(), label=param["label"], linewidth=2)

    # Add vertical and horizontal lines at origin
    plt.axhline(y=0, color="k", linestyle="-", alpha=0.3)
    plt.axvline(x=0, color="k", linestyle="-", alpha=0.3)

    # Configure the plot
    plt.grid(True, alpha=0.3)
    plt.xlabel("x")
    plt.ylabel("f(x)")
    plt.title("PELU Activation Function with Different Parameters")
    plt.legend()
    plt.tight_layout()

    plt.show()


if __name__ == "__main__":
    main()

(Source code, png, hires.png, pdf)

../_images/pelu_example.png

References