Masters Thesis Part 3: Algorithm Overview

I represent each RF using a multivariate Gaussian distribution with a mean vector (position), covariance matrix (alignment and size), and gain factor (response magnitude). I determine the value of each of these parameters for each RF using a competitive algorithm where the result of a custom greedy algorithm competes against the result of a genetic algorithm. Gaussians are trained one at a time in a staged approach so that the addition of each new Gaussian to the model takes into consideration the effects of all of the previously trained Gaussians.

Actually, the genetic algorithm applies a little more intelligence by dividing a population of Gaussians into species based on similar characteristics and mating (i.e. crossing-over) the best performing Gaussian members within the same species during each generation of evolutionary training. The addition of each new Gaussian to the model attempts to address some residual error in the model resulting from recorded neuron firing rate response values that have not yet been fully explained by the addition of previous Gaussian distributions. The theory is that the best performing Gaussian of each species is associated with a different local minima in the residual error search space, thereby increasing the chances that one of those Gaussians is associated with the global minimum. The Gaussian which results in the greatest drop in residual error among all of the species is selected and added to the model.

The Gaussian model (also known as a Scaled Gaussian Mixture Model) is then used to extract features that are fed into a linear regression (LR) technique. Why is LR necessary on top of the Gaussian model? The reason is because not all RFs are the same. A great deal of research indicates the existence of hotspot segments in the visual field which have a major impact on the response values generated by RFs in non-hotspot segments. Specifically, the directional preference of RFs associated with hotspot segments plays a major role. Thus, the response value of RFs associated with hotspot segment needs to be weighted differently than those for non-hotspot segments and the response values each segment needs to be divided into directional contribution vectors that are each weighted differently.

Overall my algorithmic approach trains models that are capable of predicting neuron firing rate response values with moderate to high accuracy. As such my research offers support for a number of existing concepts in the field of visual neuroscience, such as the gradient hypothesis and opponent organization.

Read the full report (PDF) for a complete explanation.

Masters Thesis Part 2: Receptive Field Theory

So how did I go about creating a model capable of predicting individual neuron responses of primates to various patterns of moving white dots? The key was to use receptive field (RF) theory. In general, a RF is a physiological construct associated with an area of the visual field. Let's say you're staring straight ahead and perceive some motion across the upper right of your visual field. You may have a RF associated with that region such that as you perceive motion across it your nervous system reacts by speeding up (excitation) or slowing down (inhibition) the number of electric impulses between neurons in your visual cortex within a certain period of time.

Each RF is very simple in structure. Imagine a doughnut. The outer ring and missing center form 2 separate areas across which the RF responds to motion. There are 2 flavors of RFs. The first is on-center off-surround where motion across the center causes an excitatory response and motion across the ring causes an inhibitory response. The second is off-center on-surround, which behaves in the opposite way. Multiple RFs can be aligned in various ways to respond to various directions of motion. If you consider your visual field to be 2-D with an x-axis (left/right) and y-axis (up/down), you can place multiple RFs within that plane to respond to all kinds of motion.

An extension of the RF theory is that each RF has a directional preference. This makes sense if you think about how multiple simple on-center off-surround RFs and off-center on-surround RFs can be combined to form more complex RFs. With the addition of a directional preference, the 2-D visual field space becomes a 3-D RF space. The task then becomes a matter of placing RFs within the RF space, aligning them properly, sizing them properly, and understanding the relationships between them.

Read the full report (PDF) for a complete explanation.

Masters Thesis Part 1: Primate Experiments

My computer science thesis work (PDF) at the Rochester Institute of Technology involved analyzing the firing rate response patterns of neurons in the Medial Superior Temporal (MST) region of Rhesus monkeys, which resides above the temple on both sides of the skull. The primate experiments were performed by Dr. Charles Duffy and Dr. William Page at the University of Rochester Neuroscience Department.

Each primate was trained to sit in a special chair and stare directly at a red dot in the center of a rear projection screen covering 90 deg. x 90 deg. of his visual field. The rest of the screen was black. The primate was rewarded with sweet juice if he remained still and stared at the dot for 5 minutes. After successfully completing this task various patterns of moving white dots were shown in the screen, like snow in the wind (except with uniform density and constant velocity). The screen was divided into 9 segments (like the side of a traditional Rubik's cube). Each dot would appear at one edge of the screen and travel in a certain pattern before exiting at another edge of the screen.

The simplest patterns consisted of linear motion where all of the dots moved in the same vertical or horizontal direction across one of the 9 screen segments. More complex patterns consisted of linear motion in 2 of the 9 screen segments where the dots could be moving in different directions within each segment. The most complex patterns were designed to simulation self-motion and occupied all 9 of the screen segments. For example, imagine the old Windows starfield screensaver where the dots travel from the center of the screen outwards, which makes you feel like you're moving into the screen. A cylindrical piece of the primate's skull cap was removed and an electrode was positioned to record individual neuron firing rate responses as the primate perceived each moving pattern on screen.

My task was to design a mathematical model capable of predicting how the primate responded to self-motion patterns given his response to simpler patterns in regards to individual MST neurons. Why? To further our understanding the mammalian brain. Such a biologically-inspired model can potentially be used to program the AI for more realistic robots, or potentially be used to program the firmware for a cybernetic implant to help treat the sensory degradation symptoms of Alzheimer's disease patients.

Note that firing rate signals were captured in a region of the brain before any higher level cognition was performed. In other words, the response values were not affected by what the primate was thinking about when viewing the patterns on screen (i.e. bananas). Well, in theory that's what we'd like to believe.

Object Tracking using Particle Filters

During my first year as a computer science grad I took a course in advanced computer vision. The class was divided into teams and each team could choose one of two projects to work on. The first project involved identifying students as they walked into a classroom based on their facial features. The second project involved tracking students as they walked into the classroom in-between other students. My team decided to work on the second project because my partner and I were both interested in motion tracking at the time.

We developed 3 systems with various degrees of motion-tracking success:

1. Kalman filter. Actually the particle filter developed by Cuevas, Zaldivar, and Rojas which is based on the Extended Kalman Filter (EKF). Some success.
2. Kalman filter with spring forces. Attempt to use multiple particle systems linked by spring forces to track multiple parts of a subject's body. Limited success.
3. Hierarchical particle filter. Based on the work of Viola and Jones and the work of Yang, et al. Uses rectangular windows for feature extraction. Quite different from the Kalman filter and the 2 systems described above. Most successful.

A lot of time was spent adjusting the particle set properties used by the first 2 systems for each student test video. The third system is more robust and doesn't require as much custom-tailoring to the video to which it is applied.

The particle filter presented by Cuevas, Zaldivar, and Rojas attempts to track a small color distribution within a circular window centered around a target pixel.

Alternatively, the approach presented by Viola and Jones employs a rectangular target window to determine the relative position of average color intensities within an area of the video.

Viola and Jones had great success detecting faces using the target window to capture the average color intensity around a person’s eyes and the average color intensity of that person’s upper cheeks. It's a very simple concept but it works well in practice because of the intensity difference between those two regions of a person's face. In general a person's upper cheeks are much brighter than the inset region around their eyes.

Resources:

• Technical report (PDF)
• Presentation 1 (PPT)
• Presentation 2 (PPT)
• Presentation 3 (PPT)

Handwritten Digit Recognizer

During my first quarter as a computer science grad I took an incredibly enlightening course in pattern recognition. Teams of students were assigned the task of implementing a handwritten digit recognizer. Like humans, computers need to determine the content of handwritten information before it can be used in a meaningful way. This is accomplished through a form of optical character recognition (OCR).

The postal service accepts packages and envelopes with handwritten addresses which must be read and interpreted in order to sort mail and send each item to its intended destination. It's both impractical and expensive to have humans sort large volumes of mail so automated computer systems are often used instead. The systems often consist of cameras which take pictures of the addresses and feed the images into program for processing.

My team decided to implement a convolutional neural network similar to Lecun's LeNet-5.

• The first layer is the input layer and consists of one neuron per pixel in a 29x29 padded version of the sample image.
• The second layer applies 6 feature maps to the input layer. Each feature map is a randomly distributed 5x5 convolutional kernel.
• The third layer applies 50 feature maps to all 6 of the previous feature maps after sub-sampling. Again, each feature map is a randomly distributed 5x5 convolutional kernel. These 2 layers are referred to as a trainable feature extractor.
• The fourth and fifth layers are referred to as a trainable feature classifier. These 2 layers are fully connected and compose a universal classifier.

A convolutional neural network exploits the spatial structure of digits and attempts to train weights to identify spatial differences between digits. We calculated classification error rates using standard backpropagation, which played a major role in training the system.

After running the network with 100 hidden nodes for 5 epochs over 60000 MNIST training samples and 10000 test samples, our network misclassified 851 samples and had a 91.4% success rate. Not bad.

While this project focused on recognizing handwritten digits, the concepts and algorithms covered can be easily extended to apply to all alphanumeric characters.

Resources:

• Technical paper (PDF)