Tanh Variants

Tanh-based activation functions and their variants for neural networks.

activations_plus.simple.tanh_variants.penalized_tanh(x: Tensor, a: float = 0.25) → Tensor[source]

Apply the Penalized Tanh activation function.

\[\text{PenalizedTanh}(x) = \tanh(x) - a \cdot \tanh^2(x)\]

Parameters:

x (torch.Tensor) – Input tensor.
a (float, optional) – Penalty coefficient (default 0.25).

Returns:

The element-wise Penalized Tanh of the input.

Return type:

torch.Tensor

Source

See also

A variant of tanh activation function proposed in “Activation Functions: Comparison in Neural Network Architecture” by Sharma et al. (2021).

arxiv

Example

import matplotlib.pyplot as plt
import torch

from activations_plus.simple import penalized_tanh

x = torch.linspace(-3, 3, 200)
y = penalized_tanh(x)
fig, ax = plt.subplots()
ax.plot(x.numpy(), y.numpy())
ax.set_title("Penalized Tanh")
ax.set_xlabel("Input")
ax.set_ylabel("Output")
ax.grid(alpha=0.3)
fig.show()  # This will be mocked in tests

(Source code, png, hires.png, pdf)