edu.wlu.cs.levy.SNARLI
Class BPLayer

java.lang.Object
  |
  +--edu.wlu.cs.levy.SNARLI.BPLayer

public class BPLayer

extends java.lang.Object

BPLayer is a class supporting creation and training of layered neural networks through back-propagation. Sigma-Pi connections and back-prop-through-time are also supported. Networks are built implicitly by connecting layers. Public methods use double-precision floating point arrays; internal computation uses the package JLinAlg.

Since:

JDK1.2

Constructor Summary

BPLayer(int siz)
Creates a neural network layer.

Method Summary

double	actdev(double xact) Applies first derivative of layer's activation function to activation on unit (not net input).
double	actfun(double xnet) Applies layer's activation function to net input.
double[]	activate(edu.wlu.cs.levy.SNARLI.BPLayer src, double[] clmp) Activates this layer by clamping activations on another and running a forward pass.
void	attach(double[][] pattern) Attaches pattern to layer as input or target.
void	batch(double eta, double mu) Runs one step of back-propagation in batch mode.
void	batch(int nep, double eta, double mu) Same as batch(nep, eta, mu, report), but with error reporting every second.
void	batch(int nep, double eta, double mu, int report) Trains all layer's in this layer's network, using back-propagation in batch mode.
void	bpttPattern() Steps through one pattern for Back-Prop Thru Time.
void	bpttPattern(int n) Steps through one pattern for Back-Prop Thru Time.
void	bpttResetEta() Resets current weight- and bias-changes for Back-Prop-Through-Time.
void	bpttUpdate(double eta, double mu, int npat) Updates weights and bias on layer using the Delta Rule, for Back-Prop-Through-Time.
void	connect(edu.wlu.cs.levy.SNARLI.BPLayer from) Makes a normal (full, Sigma) connection to this layer from another layer.
void	connect(edu.wlu.cs.levy.SNARLI.BPLayer from1, edu.wlu.cs.levy.SNARLI.BPLayer from2) Makes a Sigma-Pi connection to this layer from two other layers.
void	delay(edu.wlu.cs.levy.SNARLI.BPLayer from) Same as delay(from, weight), with weight = 1.0.
void	delay(edu.wlu.cs.levy.SNARLI.BPLayer from, double weight) Makes one-to-one time-delay connection to this layer from the specified layer, using the specified connection strength.
double	dontCare() Returns out-of-bounds value for don't-care condition.
double[]	getBias() Returns the bias on this layer.
double	getMaxError() Returns error of maximum magnitude on layer over all training patterns.
double	getRMSError() Returns Root-Mean-Squared error on layer over all training patterns.
double[][]	getSquaredErrors() Returns squared errors on layer over all training patterns.
double[][]	getWeights(edu.wlu.cs.levy.SNARLI.BPLayer from) Returns the weights on this layer from another layer (Sigma connection), as a 2D array.
double[][][]	getWeights(edu.wlu.cs.levy.SNARLI.BPLayer from1, edu.wlu.cs.levy.SNARLI.BPLayer from2) Returns the weights on this layer from two other layers (Sigma-Pi connection), as an array of 2D arrays.
void	online(int nep, double eta, double mu) Same as online(nep, eta, mu, report), but with error report every second.
void	online(int nep, double eta, double mu, int report) Trains all layers in this layer's network, using back-propagation in on-line mode.
void	randomize() Same as randomize(seed), with arbitrary seed.
void	randomize(long seed) Randomizes biases and weights on this layer.
void	randomize(java.util.Random rand) Randomizes biases and weights on this layer to values normally distributed around zero.
static void	reportValue(int iter, int maxit, int report, double value, java.io.PrintStream stream) Reports a value in a friendly way.
void	resetMu() Resets previous weight- and bias- changes.
void	setBias(double[] v) Sets bias on this layer to the values in vector.
void	setWeights(edu.wlu.cs.levy.SNARLI.BPLayer from1, edu.wlu.cs.levy.SNARLI.BPLayer from2, double[][][] w) Sets weights on this layer from two others (Sigma-Pi connection) to values in array of 2D arrays.
void	setWeights(edu.wlu.cs.levy.SNARLI.BPLayer from, double[][] w) Sets weights on this layer from another (Sigma connection) to values in 2D array.
double[][]	test() Tests a (trained) network layer.
double[][]	test(int n, double a) Tests a (recurrent) network layer without any input.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BPLayer

public BPLayer(int siz)

Creates a neural network layer. Activation function for non-input layers is the logistic sigmoid f(x) = 1/(1+exp(-x)), whose first derivative is f(x)*(1-f(x)). These defaults can be overridden by subclassing with a class that defines the actfun and actdev methods.

Parameters:

siz - number of units in the layer

Method Detail

actfun

public double actfun(double xnet)

Applies layer's activation function to net input. Override this method to get different activation functions.

Parameters:

xnet - net input to activation function

Returns:

activations

actdev

public double actdev(double xact)

Applies first derivative of layer's activation function to activation on unit (not net input). Override this method to get different activation functions.

Parameters:

xact - activation on unit

Returns:

first deriviative of activation

dontCare

public double dontCare()

Returns out-of-bounds value for don't-care condition.

Returns:

don't-care value

randomize

public void randomize(long seed)

Randomizes biases and weights on this layer. Creates random weights and biases using specified random seed, supporting reproducible experiments.

Parameters:

seed - seed for random number generator

randomize

public void randomize()

Same as randomize(seed), with arbitrary seed.

randomize

public void randomize(java.util.Random rand)

Randomizes biases and weights on this layer to values normally distributed around zero. Creates random weights and biases using specified random number generator, supporting reproducible experiments.

setWeights

public void setWeights(edu.wlu.cs.levy.SNARLI.BPLayer from,
                       double[][] w)
                throws edu.wlu.cs.levy.SNARLI.BPConnectException,
                       edu.wlu.cs.levy.SNARLI.BPSizeException

Sets weights on this layer from another (Sigma connection) to values in 2D array. Array should have M rows and N columns, where M is the number of units in this layer, and N is the number in the connected-from layer.

Parameters:

from - the layer connected from

w - the matrix of weights

Throws:

BPConnectException - if there is no such conenction

BPSizeException - if array size mismatches layer sizes

setWeights

public void setWeights(edu.wlu.cs.levy.SNARLI.BPLayer from1,
                       edu.wlu.cs.levy.SNARLI.BPLayer from2,
                       double[][][] w)
                throws edu.wlu.cs.levy.SNARLI.BPConnectException,
                       edu.wlu.cs.levy.SNARLI.BPSizeException

Sets weights on this layer from two others (Sigma-Pi connection) to values in array of 2D arrays. The array should have K elements of M rows and N columns, where K is the number of units in this layer, M is the number of units in the first connected-from layer, and N is the number in the second connected-from layer.

Parameters:

from1 - layer connected from

from2 - layer connected from

w - array of weights

Throws:

BPConnectException - if there is no such conenction

BPSizeException - if array size mismatches layer sizes

setBias

public void setBias(double[] v)
             throws edu.wlu.cs.levy.SNARLI.BPSizeException

Sets bias on this layer to the values in vector.

Parameters:

v - the vector of bias values

Throws:

BPSizeException - if vector size doesn't equal layer size

getWeights

public double[][] getWeights(edu.wlu.cs.levy.SNARLI.BPLayer from)
                      throws edu.wlu.cs.levy.SNARLI.BPConnectException,
                             edu.wlu.cs.levy.SNARLI.BPInitException

Returns the weights on this layer from another layer (Sigma connection), as a 2D array. The array will have M rows and N columns, where M is the number of units in this layer, and N is the number in the connected-from layer.

Parameters:

from - the layer connected from

Returns:

connection weights from that layer to this

Throws:

BPConnectException - if there is no such conenction

BPInitException - some network weights are uninitialized

getWeights

public double[][][] getWeights(edu.wlu.cs.levy.SNARLI.BPLayer from1,
                               edu.wlu.cs.levy.SNARLI.BPLayer from2)
                        throws edu.wlu.cs.levy.SNARLI.BPConnectException,
                               edu.wlu.cs.levy.SNARLI.BPInitException

Returns the weights on this layer from two other layers (Sigma-Pi connection), as an array of 2D arrays. The array will have K elements of M rows and N columns, where K is the number of units in this layer, M is the number of units in the first connected-from layer, and N is the number in the second connected-from layer.

Parameters:

from1 - layer connected from

from2 - layer connected from

Returns:

connection weights from those layers to this

Throws:

BPConnectException - if there is no such conenction

BPInitException - some network weights are uninitialized

getBias

public double[] getBias()
                 throws edu.wlu.cs.levy.SNARLI.BPInitException

Returns the bias on this layer.

Returns:

vector of bias values

Throws:

BPInitException - if the bias is uninitialized

getRMSError

public double getRMSError()
                   throws edu.wlu.cs.levy.SNARLI.BPInitException,
                          edu.wlu.cs.levy.SNARLI.BPSizeException

Returns Root-Mean-Squared error on layer over all training patterns. Network weights must be set before this method is invoked, or an exception will be raised. The sizes of the input and output patterns must be be compatible, or an exception will be raised.

Returns:

RMS error

Throws:

BPInitException - if the weights are uninitialized

BPSizeException - if number of input and output patterns differs

getMaxError

public double getMaxError()
                   throws edu.wlu.cs.levy.SNARLI.BPInitException,
                          edu.wlu.cs.levy.SNARLI.BPSizeException

Returns error of maximum magnitude on layer over all training patterns. Network.init must be called between network creation and this method, or an exception will be raised. The number of input and output patterns must be be equal, or an exception will be raised.

Returns:

maximum error

Throws:

BPInitException - if some network weights are uninitialized

BPSizeException - if number of input and output patterns differs

getSquaredErrors

public double[][] getSquaredErrors()
                            throws edu.wlu.cs.levy.SNARLI.BPInitException,
                                   edu.wlu.cs.levy.SNARLI.BPSizeException

Returns squared errors on layer over all training patterns. Network.init must be called between network creation and this method, or an exception will be raised. The number of input and output patterns must be be equal, or an exception will be raised.

Returns:

array of squared errors

Throws:

BPInitException - if some network weights are uninitialized

BPSizeException - if number of input and output patterns differs

connect

public void connect(edu.wlu.cs.levy.SNARLI.BPLayer from)

Makes a normal (full, Sigma) connection to this layer from another layer. The activations a_j on this layer is computed as a_j = f ( &sum_ia_iw_ji + b_j ), where a_i are the activations on the other layer, w are the weights between the layers, b are the biases on this layer, and f is the layer's activation function. Weights and biases can be initialized via the randomize or setWeights and setBias methods.

Parameters:

from - layer to connect from

connect

public void connect(edu.wlu.cs.levy.SNARLI.BPLayer from1,
                    edu.wlu.cs.levy.SNARLI.BPLayer from2)

Makes a Sigma-Pi connection to this layer from two other layers. The activation on this layer is computed as a_k = f ( &sum_i&sum_j a_ia_j w_kji + b_k ), where a_i and a_j are the activations on the other two layers w are the weights between the layers, b are the biases on this layer, and f is the layer's activation function. Weights and biases can be initialized via the randomize or setWeights and setBias methods.

Parameters:

from1 - layer to connect from

from2 - layer to connect from

attach

public void attach(double[][] pattern)
            throws edu.wlu.cs.levy.SNARLI.BPSizeException

Attaches pattern to layer as input or target. Width (number of columns) of pattern must match layer size. Pattern matrices should contain one pattern per row. For example, A = {{0,0}, {0,1}, {1,0}, {1,1}}, B = {{0}, {1}, {1}, {0}} are the patterns for learning B = XOR(A). Don't-care conditions are encoded as out-of-bound values. For example, B = {{0}, {1}, {1}, {2}} would encode XOR where you don't care about the output B when A = {1,1}.

Parameters:

pattern - pattern to attach

Throws:

BPSizeException - on width/size mismatch

batch

public void batch(int nep,
                  double eta,
                  double mu,
                  int report)
           throws edu.wlu.cs.levy.SNARLI.BPInitException,
                  edu.wlu.cs.levy.SNARLI.BPSizeException

Trains all layer's in this layer's network, using back-propagation in batch mode. This means that the errors are computed over all patterns, and then averaged to compute the weight- and bias- changes. The lengths of any patterns attached to layers and output patterns must be be compatible, or an exception will be raised.

Parameters:

nep - number of epochs for training

eta - learning rate

mu - momentum

report - number of generations between error reports

Throws:

BPInitException - if some network weights are uninitialized

BPSizeException - if lengths of patterns differ

batch

public void batch(int nep,
                  double eta,
                  double mu)
           throws edu.wlu.cs.levy.SNARLI.BPInitException,
                  edu.wlu.cs.levy.SNARLI.BPSizeException

Same as batch(nep, eta, mu, report), but with error reporting every second.

Parameters:

nep - number of epochs for training

eta - learning rate

mu - momentum

Throws:

BPInitException - if weights hav not been initialized

BPSizeException - if number of input and output patterns differs

batch

public void batch(double eta,
                  double mu)
           throws edu.wlu.cs.levy.SNARLI.BPInitException,
                  edu.wlu.cs.levy.SNARLI.BPSizeException

Runs one step of back-propagation in batch mode.

Parameters:

eta - learning rate

mu - momentum

Throws:

BPInitException - if some network weights haven't been initialized

BPSizeException - if number of input and output patterns differs

online

public void online(int nep,
                   double eta,
                   double mu,
                   int report)
            throws edu.wlu.cs.levy.SNARLI.BPInitException,
                   edu.wlu.cs.levy.SNARLI.BPSizeException

Trains all layers in this layer's network, using back-propagation in on-line mode. This means that the weights and biases are updated after the presentation of each pattern. The lengths of any patterns attached to layers and output patterns must be be compatible, or an exception will be raised.

Parameters:

nep - number of passes through data

eta - learning rate

mu - momentum

report - number of generations between error reports

Throws:

BPInitException - if some network weights are uninitialized

BPSizeException - if lengths of patterns differ

online

public void online(int nep,
                   double eta,
                   double mu)
            throws edu.wlu.cs.levy.SNARLI.BPInitException,
                   edu.wlu.cs.levy.SNARLI.BPSizeException

Same as online(nep, eta, mu, report), but with error report every second.

Parameters:

nep - number of epochs for training

eta - learning rate

mu - momentum

Throws:

BPInitException - if weights hav not been initialized

BPSizeException - if number of input and output patterns differs

test

public double[][] test()
                throws edu.wlu.cs.levy.SNARLI.BPInitException,
                       edu.wlu.cs.levy.SNARLI.BPSizeException

Tests a (trained) network layer.

Returns:

output patterns

Throws:

BPInitException - if some network weights are uninitialized

BPSizeException - if number of input and output patterns differs

test

public double[][] test(int n,
                       double a)
                throws edu.wlu.cs.levy.SNARLI.BPInitException,
                       edu.wlu.cs.levy.SNARLI.BPSizeException

Tests a (recurrent) network layer without any input.

Parameters:

n - number of steps to run

a - intial activation value

Returns:

output pattern on layer

Throws:

BPInitException - if some network weights are uninitialized

BPSizeException - if number of input and output patterns differs

delay

public void delay(edu.wlu.cs.levy.SNARLI.BPLayer from,
                  double weight)
           throws edu.wlu.cs.levy.SNARLI.BPSizeException

Makes one-to-one time-delay connection to this layer from the specified layer, using the specified connection strength. This means that this layer's activations will come from a scalar multiple of the activations of the other layer on the previous time step. On the first time step these activations are are defined be the average value of this layer's activation function (e.g., 0.5 for the logistic simgoid squashing function). If the layers have different sizes, and exception will be raised.

Parameters:

from - layer to delay from

weight - connection strength

Throws:

BPSizeException - if layers have different sizes

delay

public void delay(edu.wlu.cs.levy.SNARLI.BPLayer from)

Same as delay(from, weight), with weight = 1.0.

Parameters:

from - layer to delay from

resetMu

public void resetMu()

Resets previous weight- and bias- changes. This is only necessary when running one epoch at a time.

bpttResetEta

public void bpttResetEta()

Resets current weight- and bias-changes for Back-Prop-Through-Time.

bpttUpdate

public void bpttUpdate(double eta,
                       double mu,
                       int npat)
                throws edu.wlu.cs.levy.SNARLI.BPMomentumException

Updates weights and bias on layer using the Delta Rule, for Back-Prop-Through-Time.

Parameters:

eta - learning rate

mu - momentum

npat - total number of patterns

Throws:

BPMomentumException - if no momentum has been set on the layer

activate

public double[] activate(edu.wlu.cs.levy.SNARLI.BPLayer src,
                         double[] clmp)

Activates this layer by clamping activations on another and running a forward pass. Clamped activations are temporary and do not affect patterns set by attach.

Parameters:

src - "source" layer to clamp

clmp - vector of clamping values

Returns:

activation of this layer

Throws:

BPConnectException - if there is no path between the layers

BPInitException - if some network weights are uninitialized

bpttPattern

public void bpttPattern(int n)
                 throws edu.wlu.cs.levy.SNARLI.BPInitException,
                        edu.wlu.cs.levy.SNARLI.BPSizeException,
                        java.lang.IllegalArgumentException

Steps through one pattern for Back-Prop Thru Time. Implements a restricted version of BPTT, specified by the number of ticks in the argument.

Parameters:

n - number of ticks from end of pattern

Throws:

BPInitException - if some network weights are uninitialized

BPSizeException - if lengths of patterns differ

java.lang.IllegalArgumentException - if n < 0 or n >= current pattern length

bpttPattern

public void bpttPattern()
                 throws edu.wlu.cs.levy.SNARLI.BPInitException,
                        edu.wlu.cs.levy.SNARLI.BPSizeException

Steps through one pattern for Back-Prop Thru Time. Implements the full BPTT algorithm on the pattern. Same as bptt(n), with n = current pattern length - 1.

Throws:

BPInitException - if some network weights are uninitialized

BPSizeException - if lengths of patterns differ

reportValue

public static void reportValue(int iter,
                               int maxit,
                               int report,
                               double value,
                               java.io.PrintStream stream)

Reports a value in a friendly way. This is useful for reporting RMS and maximum error when training one epoch at a time.

Parameters:

iter - iteration number

maxit - maximum number of iterations

report - reporting interval

value - value to report

stream - print stream that reports value