TY - JOUR

T1 - Closed-form continuous-time neural networks

AU - Hasani, Ramin

AU - Lechner, Mathias

AU - Amini, Alexander

AU - Liebenwein, Lucas

AU - Ray, Aaron

AU - Tschaikowski, Max

AU - Teschl, Gerald

AU - Rus , Daniela

N1 - The article has a publisher correction: https://doi.org/10.1038/s42256-022-00597-y
In the version of this article initially published, Lemma 1 appeared at the end of the article rather than within the “Analytical LTC approximation for general inputs” subsection. In that subsection, the text now reading, in part, “Inspired by equations (7) and (8),” mention of equation (8) was missing in the original version. Also, equation (9) in Lemma 1 was incorrectly labeled as “(10)” and has been renumbered. Further, the Acknowledgements section was an incorrect version. All changes have been made to HTML and PDF versions of the article.

PY - 2022/11

Y1 - 2022/11

N2 - Continuous-time neural networks are a class of machine learning systems that can tackle representation learning on spatiotemporal decision-making tasks. These models are typically represented by continuous differential equations. However, their expressive power when they are deployed on computers is bottlenecked by numerical differential equation solvers. This limitation has notably slowed down the scaling and understanding of numerous natural physical phenomena such as the dynamics of nervous systems. Ideally, we would circumvent this bottleneck by solving the given dynamical system in closed form. This is known to be intractable in general. Here, we show that it is possible to closely approximate the interaction between neurons and synapses—the building blocks of natural and artificial neural networks—constructed by liquid time-constant networks efficiently in closed form. To this end, we compute a tightly bounded approximation of the solution of an integral appearing in liquid time-constant dynamics that has had no known closed-form solution so far. This closed-form solution impacts the design of continuous-time and continuous-depth neural models. For instance, since time appears explicitly in closed form, the formulation relaxes the need for complex numerical solvers. Consequently, we obtain models that are between one and five orders of magnitude faster in training and inference compared with differential equation-based counterparts. More importantly, in contrast to ordinary differential equation-based continuous networks, closed-form networks can scale remarkably well compared with other deep learning instances. Lastly, as these models are derived from liquid networks, they show good performance in time-series modelling compared with advanced recurrent neural network models.

AB - Continuous-time neural networks are a class of machine learning systems that can tackle representation learning on spatiotemporal decision-making tasks. These models are typically represented by continuous differential equations. However, their expressive power when they are deployed on computers is bottlenecked by numerical differential equation solvers. This limitation has notably slowed down the scaling and understanding of numerous natural physical phenomena such as the dynamics of nervous systems. Ideally, we would circumvent this bottleneck by solving the given dynamical system in closed form. This is known to be intractable in general. Here, we show that it is possible to closely approximate the interaction between neurons and synapses—the building blocks of natural and artificial neural networks—constructed by liquid time-constant networks efficiently in closed form. To this end, we compute a tightly bounded approximation of the solution of an integral appearing in liquid time-constant dynamics that has had no known closed-form solution so far. This closed-form solution impacts the design of continuous-time and continuous-depth neural models. For instance, since time appears explicitly in closed form, the formulation relaxes the need for complex numerical solvers. Consequently, we obtain models that are between one and five orders of magnitude faster in training and inference compared with differential equation-based counterparts. More importantly, in contrast to ordinary differential equation-based continuous networks, closed-form networks can scale remarkably well compared with other deep learning instances. Lastly, as these models are derived from liquid networks, they show good performance in time-series modelling compared with advanced recurrent neural network models.

UR - http://www.scopus.com/inward/record.url?scp=85141901775&partnerID=8YFLogxK

U2 - 10.1038/s42256-022-00556-7

DO - 10.1038/s42256-022-00556-7

M3 - Journal article

SN - 2522-5839

VL - 4

SP - 992

EP - 1003

JO - Nature Machine Intelligence

JF - Nature Machine Intelligence

IS - 11

ER -