Soft Clipping

Soft Clipping activation function.

This activation function smoothly limits the range of activations, preventing extreme values without hard truncation. It is particularly useful for stabilizing neural network training.

activations_plus.SoftClipping.__init__(self, x_min: float = -1.0, x_max: float = 1.0, clip_func: ~typing.Callable = <built-in method sigmoid of type object>)

Initialize the SoftClipping module.

SoftClipping is a configurable module for applying soft clipping to values within a specified range. The module uses a differentiable non-linear clipping function to constrain input values between a defined minimum and maximum range, facilitating gradient-based optimization and control over numerical saturation.

The class allows the customization of the clipping function as well as the range for clipping, which makes it applicable in a variety of machine learning tasks requiring such transformations.

Parameters:

x_min (float) – The minimum value to which the input will be clipped. Defaults to -1.0.
x_max (float) – The maximum value to which the input will be clipped. Defaults to 1.0.
clip_func (Callable) – A differentiable function used to softly clip the input values. This function determines the softness and shape of the transition between unclipped and clipped regions. Defaults to torch.sigmoid.

activations_plus.SoftClipping.forward(self, x: Tensor) → Tensor

Apply a transformation to the input tensor by scaling and clipping its values.

Scale and clip the input tensor based on predefined min_val and max_val, using a clipping function.

Parameters:: x (torch.Tensor) – Input tensor to be transformed.
Returns:: Transformed tensor after applying the scaling and clipping operation.
Return type:: torch.Tensor

Reference Paper: Soft Clipping Activation Function

Mathematical Explanation:

The Soft Clipping activation function is defined as:

\[f(x) = x_{\text{min}} + (x_{\text{max}} - x_{\text{min}}) \cdot g(x)\]

where \(g(x)\) is the clipping function, which can be chosen as sigmoid, tanh, or any other differentiable function. For example:

Sigmoid: \(g(x) = \frac{1}{1 + e^{-x}}\)
Hyperbolic Tangent (tanh): \(g(x) = \tanh(x)\)

This ensures smooth clipping of input values within the range \([x_{\text{min}}, x_{\text{max}}]\), with the behavior determined by the choice of \(g(x)\).

—

Example Usage:

import torch
from activations_plus.soft_clipping import SoftClipping

# Initialize the Soft Clipping activation function
activation = SoftClipping(x_min=-2.0, x_max=2.0)

# Input tensor
input_tensor = torch.tensor([-3.0, -1.0, 0.0, 1.0, 3.0])

# Apply the activation function
output_tensor = activation(input_tensor)

# Print the output
print(output_tensor)  # Example output: tensor([-1.9998, -1.2689,  0.0000,  1.2689,  1.9998])

—

Custom Clipping Function:

The Soft Clipping activation function can also use a custom clipping function instead of the default sigmoid. For example:

import torch
from activations_plus.soft_clipping import SoftClipping

# Custom clipping function (e.g., hyperbolic tangent)
custom_clip_func = torch.tanh

# Initialize the Soft Clipping activation function with a custom function
activation = SoftClipping(x_min=-2.0, x_max=2.0, clip_func=custom_clip_func)

# Input tensor
input_tensor = torch.tensor([-3.0, -1.0, 0.0, 1.0, 3.0])

# Apply the activation function
output_tensor = activation(input_tensor)

# Print the output
print(output_tensor)  # Example output: tensor([-1.9993, -1.7616,  0.0000,  1.7616,  1.9993])

—

This activation function is particularly useful for stabilizing neural network training by smoothly limiting the range of activations.