Tag: Neural networks

PCA with Rubner-Tavan Networks

Deep Learning, Machine Learning, Machine Learning Algorithms Addenda, Neural networks, PythonNo Comments

One of the most interesting effects of PCA (Principal Component Analysis) is to decorrelate the input covariance matrix C, by computing the eigenvectors and operating a base change using a matrix V:   The eigenvectors are sorted in descending order considering the corresponding eigenvalue, therefore Cpca is a diagonal matrix where the non-null elements are λ1 >= λ2 >= λ3 >= … >= λn. By selecting the top p eigenvalues, it’s possible to operate a dimensionality reduction by projecting the samples in the new sub-space determined by the p top eigenvectors (it’s possible to use Gram-Schmidt orthonormalization if they don’t have a unitary length). The standard PCA procedure works with a bottom-up approach, obtaining the decorrelation of C as a final effect, however, it’s possible to employ neural networks, imposing this condition as an optimization step. One the most effective model has been proposed by Rubner and Tavan (and it’s named after them). Its […]

Hopfield Networks addendum: Brain-State-in-a-Box model

Artificial Intelligence, Complex Systems, Deep Learning, Machine Learning, Machine Learning Algorithms Addenda, Neural networks, PythonNo Comments

The Brain-State-in-a-Box is neural model proposed by Anderson, Silverstein, Ritz and Jones in 1977, that presents very strong analogies with Hopfield networks (read the previous post about them). The structure of the network is similar: recurrent, fully-connected with symmetric weights and non-null auto-recurrent connections. All neurons are bipolar (-1 and 1). If there are N neurons, it’s possible to imagine an N-dimensional hypercube: The main differences with a Hopfield network are the activation function: and the dynamics that, in this case, is synchronous. Therefore, all neurons are updated at the same time. The activation function is linear when the weighted input a(i) is bounded between -1 and 1 and saturates to -1 and 1 outside the boundaries. A stable state of the network is one of the hypercube vertices (that’s why it’s called in this way). The training rule is always an extended Hebbian based on the pre-synaptic and post-synaptic […]

ML Algorithms Addendum: Hopfield Networks

Artificial Intelligence, Computational Neuroscience, Deep Learning, Generic, Machine Learning, Machine Learning Algorithms Addenda, Neural networks, Python2 Comments

Hopfield networks (named after the scientist John Hopfield) are a family of recurrent neural networks with bipolar thresholded neurons. Even if they are have replaced by more efficient models, they represent an excellent example of associative memory, based on the shaping of an energy surface. In the following picture, there’s the generic schema of a Hopfield network with 3 neurons: Conventionally the synaptic weights obey the following conditions: If we have N neurons, also the generic input vector must be N-dimension and bipolar (-1 and 1 values).The activation function for each neuron is hence defined as: In the previous formula the threshold for each neuron is represented by θ (a common value is 0, that implies a strong symmetry). Contrary to MLP, in this kind of networks, there’s no separation between input and output layers. Each unit can receive its input value, processes it and outputs the result. According to the […]

Quickprop: an almost forgotten neural training algorithm

Artificial Intelligence, Computational Neuroscience, Deep Learning, Generic, Machine Learning, Machine Learning Algorithms Addenda, Neural networks, Python3 Comments

Standard Back-propagation is probably the best neural training algorithm for shallow and deep networks, however, it is based on the chain rule of derivatives and an update in the first layers requires a knowledge back-propagated from the last layer. This non-locality, especially in deep neural networks, reduces the biological plausibility of the model because, even if there’s enough evidence of negative feedback in real neurons, it’s unlikely that, for example, synapsis in LGN (Lateral Geniculate Nucleus) could change their dynamics (weights) considering a chain of changes starting from the primary visual cortex. Moreover, classical back-propagation doesn’t scale very well in large networks. For these reasons, in 1988 Fahlman proprosed an alternative local and quicker update rule, where the total loss function L is approximated with a quadratic polynomial function (using Taylor expansion) for each weight independently (assuming that each update has a limited influence on the neighbors). The resulting weight update […]

ML Algorithms Addendum: Hebbian Learning

Artificial Intelligence, Computational Neuroscience, Machine Learning, Machine Learning Algorithms Addenda, Neural networks, Python2 Comments

Hebbian Learning is one the most famous learning theories, proposed by the Canadian psychologist Donald Hebb in 1949, many years before his results were confirmed through neuroscientific experiments. Artificial Intelligence researchers immediately understood the importance of his theory when applied to artificial neural networks and, even if more efficient algorithms have been adopted in order to solve complex problems, neuroscience continues finding more and more evidence of natural neurons whose learning process is almost perfectly modeled by Hebb’s equations. Hebb’s rule is very simple and can be discussed starting from a high-level structure of a neuron with a single output: We are considering a linear neuron, therefore the output y is a linear combination of its input values x: According to the Hebbian theory, if both pre- and post-synaptic units behave in the same way (firing or remaining in the steady state), the corresponding synaptic weight will be reinforced. Vice […]

Hodgkin-Huxley spiking neuron model in Python

Artificial Intelligence, Complex Systems, Computational Neuroscience, Neural networks, PythonNo Comments

The Hodgkin-Huxley model (published on 1952 in The Journal of Physiology [1]) is the most famous spiking neuron model (also if there are simpler alternatives like the “Integrate-and-fire” model which performs quite well). It’s made up of a system of four ordinary differential equations that can be easily integrated using several different tools. The main idea is based on an electrical representation of the neuron, considering only Potassium (K) and Sodium (Na) voltage-gated ion channels (even if it can be extended to include more channels). A schematic representation is shown in the following figure: The elements are: Cm: a capacitance per unit area representing the membrane lipid-bilayer (adopted value: 1 µF/cm²) gNa: voltage-controlled conductance per unit area associated with the Sodium (Na) ion-channel (adopted value: 120 µS/cm²) gK: voltage-controlled conductance per unit area associated with the Potassium (K) ion-channel (adopted value: 36 µS/cm²) gl: conductance per unit area associated with the leak channels (adopted […]

Archimedes: a simple exercise with Keras and Scikit-Fuzzy

Deep Learning, Machine Learning, Neural networks, Python, Scikit-Fuzzy, Scikit-LearnNo Comments

This is a simple exercise, not a real, complete implementation. However I think it’s a good starting point if you want to use Keras in order to learn time sequences and Scikit-Fuzzy, to extract probabilistic rules (which descrive the evolution) from them. First of all, you need both packages: Keras (http://keras.io/) Theano (http://deeplearning.net/software/theano/) Scikit-Fuzzy (https://github.com/scikit-fuzzy/scikit-fuzzy) Moreover you need Numpy (I do suggest to use Anaconda as Python distribution, because it contains more or less everything you need) I start creating a sample sequence (in this case, very simple and reasonably predictable): import numpy as np class SampleGenerator(object): def __init__(self, num_samples, num_tests): self.num_samples = num_samples self.num_tests = num_tests def generate_discrete_sequence(self): symbols = (0, 1, 2, 3, 4, 5) data = [] data.append(np.random.randint(0, high=len(symbols))) for i in range(1, self.num_samples + self.num_tests): # Apply “probabilistic-logic” rules if data[i-1] == symbols[0]: if self.uniform_threshold(): data.append(symbols[2]) continue else: data.append(symbols[3]) continue if data[i-1] == symbols[1]: if self.uniform_threshold(): […]