IBM TechXchange Community Group Library

Large Language Models (LLMs) can be very large, from billions to trillions of parameters, and will require significant new cloud computing resources and power consumption. At the core, LLMs are linear algebra and (nonlinear) activation functions. We propose a new class of efficiently computable activation functions, capable of standing in for Sigmoid, GELU, tanh, SiLU, ELU, and SoftMax. Each function is a piecewise combination of variations of the exponential function. Architectures with efficient exponential function computation can easily add efficient activation function evaluation, saving both time and power. We explain why they will be significantly faster and have lower power consumption and present numerical simulations showing that they perform as well or better than the originals.