Mathematical Analysis of Neural Networks Dynamics

In this project we intended to investigate and analyze neural networks signal propagation with mathematical and experimental examination.

Abstract

This project is theoretical research and consists of two parts:
The first one is intended to investigate and analyze neural networks signal propagation with mathematical and experimental examination of neural networks pathways as a function of the environment and network architecture. Neural networks considered to be powerful parallel processing machine, where neural network pathways are signals and each neuron’s spike or its absence is an information unit. This part of the project contains formulas for computation of the number of pathways, definition of the liquid current intensity (LCI) and practical algorithms for their demonstration using Matlab tools.
The second part of the project is concentrated on building the approximated linear model of neural network firing rate, where firing rate (FR) of a neuron is number of its spikes per second in defined time window, centered at the measured moment. As it concluded in the project, such model can be built although the neural network itself is definitely non linear system. This model allows calculating FR of neurons at any time as function of neural network architecture and the stimulus using the simplicity of linear systems computation.

The aim of the project

1.1 Creating an algorithm for extracting neural network pathways from input to any spike in the readout
1.2 Creating formulas to calculate the amount of pathways as a function of network architecture and the readout
1.3 Defining a parameter in order to signify the strength or intensity of a pathways to any spike as a function of primary and secondary input currents

2.1 Defining firing rate of the neural network response
2.2 Making linear model of neural network firing rate

Tools

  • MATLAB
    MATLAB was used as an environment for executing and analyzing experiments data
  • CSIM
    CSIM framework (http://www.lsm.tugraz.at/) have been used for simulating neural networks in experiments

The project overview

1.1 Number of pathways in fully connected neural microcircuit (NM):
1

1.2 Extracting pathways from the source to any specific spike:

Each spike has at least one source spike and can be a source to other spikes.

  • Each spike has at least one source spike and can be a source to other spikes.
  • A spike of neuron i in time t1 called a source of a spike of neuron j in time t2 if it obeys the following conditions:
    1) Neuron i connected to neuron j
    2) t1 took place in: 2

In the typical example below the actual amount of pathways from input source (#91) to the spike of neuron #95 in temporal layer 33 ms is 1900 (!). Five of those pathways are illustrated.

3

The number of pathways can be approximately calculated by the following formula:

4
1.3 Liquid-current intensity (LCI)

LCI signifies the probability of spiking-not spiking under 5 and is defined by the following recursive algorithm:
6
An example of LCI computation for two simple patterns:
7
The colors of spikes indicate LCI values on log scale.
8
Each spike can be considered as an information unit. Thus, a group of spikes, called cliques can be considered as more powerful data, where either firing or non-firing of members of the clique can indicate the relationship between 5and 9.
10
As we can see some three spikes have significant LCI value relatively to their temporal layer. These spikes can be considered as candidates to be members of a clique.
This clique keep under 5.
11

2.1 NM activity and stability

A typical NM system consists of two subsystems of excitatory and inhibitory neurons:
12
This relation can be presented by:
13
The time step can be chosen as 14
Model parameters indicate the conditions of NM dynamics and stability.

2.2 Firing rate (FR)

FR of a neuron in a moment t is defined as number of spikes per second in time window M, centered at t.
Dragging this window through the spike train we can calculate FR for any time.
15

2.3 Analyzing firing rate readout of small neural microcircuit

In the following example there are only two neurons in the NM and one input neuron (FRstim is firing rate of the input neuron):

16
17
Some linear dependence of neuron’s FR on FR of the input neuron (FRstim) can be definitely seen from these examples.

2.4 Static linear model of firing rate in neural microcircuit

Claim: firing rate of a neuron depends linearly on the injection current to this neuron.
Suppose, FR of all neurons in the NM including input neuron are constants (this can be achieved by setting FRstim to be constant).Then
18
Where W is a matrix of synaptic weights and a is a constant
19
20
In order to calculate the constant a we define the error by:
21
and look for a, which minimizes this error:
22
The result is 23
24
Examples (N=8. FR empirical (red) and theoretical (blue)):
25
26

2.5 Dynamic linear model of firing rate in neural microcircuit

The method, described above, can be used in dynamic firing rate prediction too, i.e. the firing rate of input neuron is not constant. In this case:
27
Where, the time steps can be chosen as 14

Z-transform:
28
Thus, the transfer function from input FRstim to FR of each neuron is given by:
29
Several features deserve comments:
First, the noise, which presents in biological NM, is not taken into account in this model.
Second, the low activity of an input to each neuron may not to lead to the activity of this neuron. This is due to existing relaxation time constant of neuron’s membrane.
Third, at high input firing rate to the neuron, this neuron can reach saturation regime of maximal firing rate, which defined by:
30

Acknowledgment

I am very thankful to my supervisors Karina Odinaev and Igal Raichelgauz for their patients, guiding me throughout the project and many interesting and useful discussions. I am also grateful to the Lab engineer Johanan Erez for his help and support.
I am also grateful to the Ollendorf Minerva Center for their contribution to the Vision and Imaging Sciences Laboratory.