A major problem in computational neuro-biology is to measure the computational power of neural networks (natural or simulated).
Abstract
A major problem in computational neuro-biology is to measure the computational power of neural networks (natural or simulated). Here we suggest two approaches for quantifying this property. One is based on the chaos theory, and the other on the Liquid State Machine (LSM) theory. We present two implementations (one for each approach), that will be the basis for further studies of biological realistic neural networks.
The problem (or the background):
Given a MEA (Multi Electrode Array, an i/o interface to real neural culture) or a neural network model, we have 3 requirements:
- Apply it to a specific task
- Qualify its computational power
- Set structural parameters for optimization
For the first task, the LSM model is applied. Our project’s goal was to develop tools for analyzing the computational power of the model by computer simulations.
The basic approach:
The following scheme defines the basic approach of our project:
Figure 1 – Block diagram of the basic approach: we change the network structural parameters in order to influence it’s dynamics, which in turn, according to the chaos theory, will lead to performance optimization
To Analyze the network dynamics, we used the chaos analysis theory presented in Real-Time Computation at the Edge of Chaos in Recurrent Neural Networks by Nils Bertschinger and Thomas Natschl¨ager.
According to the theory, a network can move between to types of dynamics. A network can either be in an “ordered” phase, which means that the current network state is determined to a large extent by the current input, or it can be in a “chaotic” phase, which means that arbitrary small differences in a (initial) network state are highly amplified and do not vanish. It can be shown that the highest computational capability is achieved if a network is near the “critical line”, which is the turn-over point between ordered and chaotic dynamics. In this project we developed two tools to analyze the network dynamics according to chosen parameters, and to show whether we are getting closer to or further from the critical line. To test the network’s computational power we looked at four definitions:
: distance between states of two structurally identical networks
: distance between states of the networks after an identical input at time t
: distance between states of the networks after two different inputs at time t
: distance between states of the networks resulting only from the input
The value is computed by the following equation:
The 
and 
values of the equation are calculated at infinity. In our simulation the stable value which the system converges to after a short period of time. The last part of the equation is suppose to compensate for neurons who just “copy” the input to the output, so networks which just pass the input to the output won’t get a high value. b is the distance between the inputs, and q is the fraction of neurons that simply copy the input. this last part was not calculated in our simulation.
We have developed the CAT (chaos analysis tool) to measure the NM-separation of a given network parameters with two degrees of freedom (two of the parameters can be changed). The output of the tool is a 3D plot describing the change in the NM-separation value according to the chosen parameters. We tested the tool on several networks. The free parameters we chose were network size and network connectivity (a statistical parameter which determines the number of connections to each neuron). Here is an example plot:
Figure 2 – Changes to NM-separation by the network size (written here in cube root) and connectivity
Another way to approximate the computational power of the liquid is to measure the amount of separation it has (the ability of the liquid to distinguish between two different inputs). For this task, We have developed the SPA (separation property analyzer) tool. On a given network, this tool measures how distant are its outputs for two different inputs. The distance between two inputs (outputs) is measured by applying a Gaussian convolution on each one of the inputs, and then subtracting one from the other. This method was also used by CAT to measure the state distances. Here is an example of the SPA output:
Figure 3 – The result of 6 consecutive identical experiments (in green), the mean (in red) and the standard deviation (in blue)
Tools
We have used Matlab 6.5 for implementation of the CAT and the SPA tools and the CSIM simulator (can be downloaded here).
Conclusions
We have developed two tools for analyzing the network computational power. The first one in the SPA tool that measures the amount of separation a given network had and the second one is the CAT that measures the dynamics of the given network according to the chaos theorem. From the outputs we can see that as expected, the bigger the network is the more chaotic it tends to be but, there is a problem in measuring low connectivity networks. We can see that a network gets high NM-separation values at low connectivity (where the input is simply “copied” to the output). This implies that the third parameter [b (2q (u,b) – 1)2] is probably needed, or, due to difficulties in calculating this parameters, inputs should be sophistically planned, to avoid such behaviors (for example, inputs that require memory capability in order to separate between them).
Acknowledgment
We are grateful to our project supervisors Igal and Karina for their help and guidance throughout this work. We are also grateful to Johanan Erez and the VISL staff for their patience dealing with the long simulation time requests.