Elish

ELiSH (Exponential Linear Sigmoid Squash) activation function.

Combines properties of exponential and sigmoid functions,

aiming to retain small negative values while maintaining smoothness.

activations_plus.ELiSH.__init__(self, *args, **kwargs) → None

Initialize internal Module state, shared by both nn.Module and ScriptModule.

activations_plus.ELiSH.forward(self, x: Tensor) → Tensor

Apply the Swish activation function element-wise.

When the input value is greater than zero, the Swish function scales it by a sigmoid factor. Otherwise, an exponential transformation is applied. This allows for a smooth non-linear activation that aids deep learning models in learning complex data patterns more effectively.

Parameters:

x – A PyTorch tensor input representing the data to apply the Swish activation function.

Returns:

A PyTorch tensor containing the element-wise output after applying the Swish activation function.

Reference Paper: ELiSH Activation Function

Mathematical Explanation:

The ELiSH activation function is defined as:

\[\begin{split}f(x) = \begin{cases} x \cdot \frac{e^x - 1}{e^x + 1}, & x \geq 0 \\ \frac{e^x - 1}{e^x + 1}, & x < 0 \end{cases}\end{split}\]

Example Usage:

import torch
from activations_plus.elish import ELiSH

activation = ELiSH()
x = torch.tensor([-3.0, 0.0, 3.0])
y = activation(x)
print("ELiSH Output:", y)