The Mish Activation Function
Neural Units & Activations DS practice problem on Onlearn.
Difficulty: medium.
Topics: Understanding Non-Linear Activation Functions in Deep Learning, Softplus Function, Hyperbolic Tangent, Log-Sum-Exp Trick, Self-Gating Mechanism, Computational Graph Construction, Calculus, Deep Learning, Optimization, Numerical Analysis, Neural Network Architecture, Activation Functions, Gradient Flow, Vanishing Gradient Problem, Non-linear Mapping, Function Approximation.
The Mish activation function is defined as f(x) = x tanh(ln(1 + e^x)). It is a self regularized, non monotonic activation function that often outperforms ReLU in deep networks. Implement a function 'mish(x)' that computes this value for a given input. Ensure your implementation is numerically stable for large positive values of x.