SReLU

SReLU (S-shaped Rectified Linear Unit) activation function.

activations_plus.SReLU.__init__(self, lower_threshold: float = -1.0, upper_threshold: float = 1.0) → None

Initialize the SReLU activation function with user-defined thresholds.

This activation function applies constraints on the input values, where inputs below a specified lower threshold or above an upper threshold are clipped. The SReLU is a piecewise linear activation function primarily used in neural networks.

Parameters:

lower_threshold – The minimum value an input can take after being passed through the activation function.
upper_threshold – The maximum value an input can take after being passed through the activation function.

Raises:

ValueError – If lower_threshold is greater than upper_threshold.

activations_plus.SReLU.forward(self, x: Tensor) → Tensor

Apply a thresholding operation to the input tensor, clipping values that are below or above.

the specified thresholds.

If a value in the input tensor is less than the lower_threshold, it is replaced with lower_threshold. If a value exceeds the upper_threshold, it is replaced with upper_threshold. Values in between these thresholds remain unchanged.

Parameters:: x (Tensor) – Input tensor to which the thresholding operation is applied.
Returns:: A tensor with values clipped according to the thresholding criteria.
Return type:: Tensor

Reference Paper: SReLU Activation Function

Mathematical Explanation:

The SReLU activation function is defined as:

\[\begin{split}f(x) = \begin{cases} t_1 + a_1(x - t_1), & x < t_1 \\ x, & t_1 \leq x \leq t_2 \\ t_2 + a_2(x - t_2), & x > t_2 \end{cases}\end{split}\]

where \(t_1\), \(t_2\), \(a_1\), and \(a_2\) are learnable parameters.

Example Usage:

import torch
from activations_plus.srelu import SReLU

srelu = SReLU()
input_tensor = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
output_tensor = srelu(input_tensor)
print(output_tensor)  # Example output: tensor([-1.5, -0.5,  0.0,  1.0,  2.0])