The Liquid Neural Networks Story

[almost technical] back in 2016, Mathias and I were looking into how neurons in animal brains exchange information with each other, when prompted by an input signal. this is not new! in fact it dates back to Lord Kelvin’s discovery of cable equation in 1855, then to Lapicque’s integrate & fire ODE model of neural dynamics in 1907, and finally to the full dynamical system representation of axonal neurons with gated dynamics by Hodgkin & Huxley in 1952 for which they won a Nobel Prize in 1963.

What we saw was that the communication dynamics had mathematical operators that allowed the input signal not only stimulate the system to generate an output response instantaneously, but do so by impacting the inner behavior of the cell to refine the generated output. This impact was in the form a multiplicative complex gating mechanisms that were input & state dependent. That behavior sounded to me like a Liquid behavior — a flexible inner dynamics allowing for a more robust output (NOTE: this does not mean the system is a continual learning system, it only means the neural dynamics gets into flexible cycles of computations before they output a response).

This observation alongside the continuous-nature of neural dynamics, combined with our backgrounds in control theory, dynamical systems, and machine learning got us into thinking how to unify these concepts into a new powerful class of deep learning algorithms for robust decision-making. This became Liquid Time-constant Networks (LTCs) [Hasani & Lechner at al. 2018, 2020].

LTCs extend the familiar input- and state-dependent gating found in LSTMs to a continuous-time setting, letting the neuron’s time-constant and synaptic strengths themselves become learnable functions of the current input and hidden state.

In essence, LTCs can be viewed as a deep-learning unification of the following threads: they marry Grossberg-style multiplicative “gain control” [Grossberg, 1973], bilinear input modulation from control theory [R. R. Mohler & R. E. Rink, 1966] and Hodgkin–Huxley-style state-dependent time scales [Hodgkin & Huxley, 1952], while remaining end-to-end trainable with modern gradient-based approaches.

Today, many modern deep learning architectures fall under the LTC’s umbrella, including Liquid foundation models (LFMs) https://lnkd.in/dBn-r2Pk, our latest work at Liquid AI, as well as every input-dependent recurrent, state-space and convolutional neural networks.

LTCs are not the end-game. mother nature has gifted us knowledge-priors like LTCs that are vital ingredients in our humanity’s journey to understanding intelligence. We are just scratching the surface.