Technical Report MSC-2021-07

Title: Charting and Navigating the Space of Solutions for Recurrent Neural Networks
Authors: Elia Turner
Supervisors: Omri Barak
PDFCurrently accessibly only within the Technion network
Abstract: In recent years, trained recurrent neural networks (RNNs) were used as models for the neural activity of behaving animals. In this framework, networks are trained to perform a task that is similar to the experimental one. Features of the network's behavior and neural activity are then compared to neural recordings. Despite this increasing use, it is still unknown in which cases this approach will work. In particular, the match between model and data seems to suggest a unique solution found by both biology and the artificial network -- a puzzling conjecture.

Recent work (Maheswaranathan et al., 2019) addressed the uniqueness problem and proposed that the solutions to various canonical tasks are, from a topological perspective, widely universal. Here, we study a slightly more complex task -- the Ready-Set-Go timing task. We find that universality no longer holds for this task and that even identical settings can lead to qualitatively different solutions - both from behavioral and neuronal perspectives. We discover these differences by testing the trained networks' ability to extrapolate, as a perturbation to a system often reveals hidden structure. Intending to understand the solution space, we cluster the solutions into discrete sets and characterize each - showing that the neural and behavioral clusters are highly consistent. We draw a low-dimensional map of the solution space and sketch the training process as a trajectory in it. We suggest that the effects of updating parameters during training often take form as a bifurcation in the governing discrete ODE, which results in a topological change in the dynamical mechanism and behavior.

Moreover, we explore the question of nature vs. nurture - the effect of the initial weights vs. training set over the final solution and show that, in our setting, only the former has a meaningful impact on the learned solution.

CopyrightThe above paper is copyright by the Technion, Author(s), or others. Please contact the author(s) for more information

Remark: Any link to this technical report should be to this page (, rather than to the URL of the PDF files directly. The latter URLs may change without notice.

To the list of the MSC technical reports of 2021
To the main CS technical reports page

Computer science department, Technion