Introduction to Neural Networks

Summary

We then introduce one kind of network in detail: the feedforward network trained by backpropagation of error. We discuss model architectures, training methods and data representation issues. We hope to cover everything you need to know to get backpropagation working for you. A range of applications and extensions to the basic model will be presented in the final section of the module.

Course content

• Computation in the brain
• Artificial neuron models
• Linear regression
• Linear neural networks
• Multi-layer networks
• Error Backpropagation

• Revision: the backprop algorithm
• Backprop: an example
• Overfitting and regularization
• Growing and pruning networks
• Preconditioning the network
• Momentum and learning rate adaptation

Section 3: Advanced Topics

• Learning rate adaptation
• Classification
• Non-supervised learning
• Recurrent neural networks
• Real-Time Recurrent Learning
• Dynamics of RNNs
• Long Short-Term Memory

List of English terms for mathematical expressions that we are using.

A few suggestions for possible project topics.

Section 3 as a ZIP file.

Tutorials:

• The Nervous System - a very nice introduction, many pictures

• Neural Java - a neural network tutorial with Java applets

• a book chapter describing the Backpropagation Algorithm (Postscript)

• A short set of pages showing how a simple backprop net learns to recognize the digits 0-9, with C code

• Reinforcement Learning - A Tutorial

	Brainwave: a Java based simulator
	tlearn: W*ndows, M*cintosh and Un*x implentation of backprop and variants. Written in C.
	PDP++: C++ software with every conceivable bell and whistle. Un*x only. The manual also makes a good tutorial.

Related stuff of interest:

• A page of neural network links