edu.wlu.cs.levy.SNARLI
Class BPLayer

java.lang.Object
  |
  +--edu.wlu.cs.levy.SNARLI.BPLayer

public class BPLayer
extends java.lang.Object

BPLayer is a class supporting creation and training of layered neural networks through back-propagation. Sigma-Pi connections and back-prop-through-time are also supported. Networks are built implicitly by connecting layers. Public methods use double-precision floating point arrays; internal computation uses the package JLinAlg.

Since:
JDK1.2

Constructor Summary
BPLayer(int siz)
          Creates a neural network layer.
 
Method Summary
 double actdev(double xact)
          Applies first derivative of layer's activation function to activation on unit (not net input).
 double actfun(double xnet)
          Applies layer's activation function to net input.
 double[] activate(edu.wlu.cs.levy.SNARLI.BPLayer src, double[] clmp)
          Activates this layer by clamping activations on another and running a forward pass.
 void attach(double[][] pattern)
          Attaches pattern to layer as input or target.
 void batch(double eta, double mu)
          Runs one step of back-propagation in batch mode.
 void batch(int nep, double eta, double mu)
          Same as batch(nep, eta, mu, report), but with error reporting every second.
 void batch(int nep, double eta, double mu, int report)
          Trains all layer's in this layer's network, using back-propagation in batch mode.
 void bpttPattern()
          Steps through one pattern for Back-Prop Thru Time.
 void bpttPattern(int n)
          Steps through one pattern for Back-Prop Thru Time.
 void bpttResetEta()
          Resets current weight- and bias-changes for Back-Prop-Through-Time.
 void bpttUpdate(double eta, double mu, int npat)
          Updates weights and bias on layer using the Delta Rule, for Back-Prop-Through-Time.
 void connect(edu.wlu.cs.levy.SNARLI.BPLayer from)
          Makes a normal (full, Sigma) connection to this layer from another layer.
 void connect(edu.wlu.cs.levy.SNARLI.BPLayer from1, edu.wlu.cs.levy.SNARLI.BPLayer from2)
          Makes a Sigma-Pi connection to this layer from two other layers.
 void delay(edu.wlu.cs.levy.SNARLI.BPLayer from)
          Same as delay(from, weight), with weight = 1.0.
 void delay(edu.wlu.cs.levy.SNARLI.BPLayer from, double weight)
          Makes one-to-one time-delay connection to this layer from the specified layer, using the specified connection strength.
 double dontCare()
          Returns out-of-bounds value for don't-care condition.
 double[] getBias()
          Returns the bias on this layer.
 double getMaxError()
          Returns error of maximum magnitude on layer over all training patterns.
 double getRMSError()
          Returns Root-Mean-Squared error on layer over all training patterns.
 double[][] getSquaredErrors()
          Returns squared errors on layer over all training patterns.
 double[][] getWeights(edu.wlu.cs.levy.SNARLI.BPLayer from)
          Returns the weights on this layer from another layer (Sigma connection), as a 2D array.
 double[][][] getWeights(edu.wlu.cs.levy.SNARLI.BPLayer from1, edu.wlu.cs.levy.SNARLI.BPLayer from2)
          Returns the weights on this layer from two other layers (Sigma-Pi connection), as an array of 2D arrays.
 void online(int nep, double eta, double mu)
          Same as online(nep, eta, mu, report), but with error report every second.
 void online(int nep, double eta, double mu, int report)
          Trains all layers in this layer's network, using back-propagation in on-line mode.
 void randomize()
          Same as randomize(seed), with arbitrary seed.
 void randomize(long seed)
          Randomizes biases and weights on this layer.
 void randomize(java.util.Random rand)
          Randomizes biases and weights on this layer to values normally distributed around zero.
static void reportValue(int iter, int maxit, int report, double value, java.io.PrintStream stream)
          Reports a value in a friendly way.
 void resetMu()
          Resets previous weight- and bias- changes.
 void setBias(double[] v)
          Sets bias on this layer to the values in vector.
 void setWeights(edu.wlu.cs.levy.SNARLI.BPLayer from1, edu.wlu.cs.levy.SNARLI.BPLayer from2, double[][][] w)
          Sets weights on this layer from two others (Sigma-Pi connection) to values in array of 2D arrays.
 void setWeights(edu.wlu.cs.levy.SNARLI.BPLayer from, double[][] w)
          Sets weights on this layer from another (Sigma connection) to values in 2D array.
 double[][] test()
          Tests a (trained) network layer.
 double[][] test(int n, double a)
          Tests a (recurrent) network layer without any input.
 
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

BPLayer

public BPLayer(int siz)
Creates a neural network layer. Activation function for non-input layers is the logistic sigmoid f(x) = 1/(1+exp(-x)), whose first derivative is f(x)*(1-f(x)). These defaults can be overridden by subclassing with a class that defines the actfun and actdev methods.

Parameters:
siz - number of units in the layer
Method Detail

actfun

public double actfun(double xnet)
Applies layer's activation function to net input. Override this method to get different activation functions.

Parameters:
xnet - net input to activation function
Returns:
activations

actdev

public double actdev(double xact)
Applies first derivative of layer's activation function to activation on unit (not net input). Override this method to get different activation functions.

Parameters:
xact - activation on unit
Returns:
first deriviative of activation

dontCare

public double dontCare()
Returns out-of-bounds value for don't-care condition.

Returns:
don't-care value

randomize

public void randomize(long seed)
Randomizes biases and weights on this layer. Creates random weights and biases using specified random seed, supporting reproducible experiments.

Parameters:
seed - seed for random number generator

randomize

public void randomize()
Same as randomize(seed), with arbitrary seed.


randomize

public void randomize(java.util.Random rand)
Randomizes biases and weights on this layer to values normally distributed around zero. Creates random weights and biases using specified random number generator, supporting reproducible experiments.


setWeights

public void setWeights(edu.wlu.cs.levy.SNARLI.BPLayer from,
                       double[][] w)
                throws edu.wlu.cs.levy.SNARLI.BPConnectException,
                       edu.wlu.cs.levy.SNARLI.BPSizeException
Sets weights on this layer from another (Sigma connection) to values in 2D array. Array should have M rows and N columns, where M is the number of units in this layer, and N is the number in the connected-from layer.

Parameters:
from - the layer connected from
w - the matrix of weights
Throws:
BPConnectException - if there is no such conenction
BPSizeException - if array size mismatches layer sizes

setWeights

public void setWeights(edu.wlu.cs.levy.SNARLI.BPLayer from1,
                       edu.wlu.cs.levy.SNARLI.BPLayer from2,
                       double[][][] w)
                throws edu.wlu.cs.levy.SNARLI.BPConnectException,
                       edu.wlu.cs.levy.SNARLI.BPSizeException
Sets weights on this layer from two others (Sigma-Pi connection) to values in array of 2D arrays. The array should have K elements of M rows and N columns, where K is the number of units in this layer, M is the number of units in the first connected-from layer, and N is the number in the second connected-from layer.

Parameters:
from1 - layer connected from
from2 - layer connected from
w - array of weights
Throws:
BPConnectException - if there is no such conenction
BPSizeException - if array size mismatches layer sizes

setBias

public void setBias(double[] v)
             throws edu.wlu.cs.levy.SNARLI.BPSizeException
Sets bias on this layer to the values in vector.

Parameters:
v - the vector of bias values
Throws:
BPSizeException - if vector size doesn't equal layer size

getWeights

public double[][] getWeights(edu.wlu.cs.levy.SNARLI.BPLayer from)
                      throws edu.wlu.cs.levy.SNARLI.BPConnectException,
                             edu.wlu.cs.levy.SNARLI.BPInitException
Returns the weights on this layer from another layer (Sigma connection), as a 2D array. The array will have M rows and N columns, where M is the number of units in this layer, and N is the number in the connected-from layer.

Parameters:
from - the layer connected from
Returns:
connection weights from that layer to this
Throws:
BPConnectException - if there is no such conenction
BPInitException - some network weights are uninitialized

getWeights

public double[][][] getWeights(edu.wlu.cs.levy.SNARLI.BPLayer from1,
                               edu.wlu.cs.levy.SNARLI.BPLayer from2)
                        throws edu.wlu.cs.levy.SNARLI.BPConnectException,
                               edu.wlu.cs.levy.SNARLI.BPInitException
Returns the weights on this layer from two other layers (Sigma-Pi connection), as an array of 2D arrays. The array will have K elements of M rows and N columns, where K is the number of units in this layer, M is the number of units in the first connected-from layer, and N is the number in the second connected-from layer.

Parameters:
from1 - layer connected from
from2 - layer connected from
Returns:
connection weights from those layers to this
Throws:
BPConnectException - if there is no such conenction
BPInitException - some network weights are uninitialized

getBias

public double[] getBias()
                 throws edu.wlu.cs.levy.SNARLI.BPInitException
Returns the bias on this layer.

Returns:
vector of bias values
Throws:
BPInitException - if the bias is uninitialized

getRMSError

public double getRMSError()
                   throws edu.wlu.cs.levy.SNARLI.BPInitException,
                          edu.wlu.cs.levy.SNARLI.BPSizeException
Returns Root-Mean-Squared error on layer over all training patterns. Network weights must be set before this method is invoked, or an exception will be raised. The sizes of the input and output patterns must be be compatible, or an exception will be raised.

Returns:
RMS error
Throws:
BPInitException - if the weights are uninitialized
BPSizeException - if number of input and output patterns differs

getMaxError

public double getMaxError()
                   throws edu.wlu.cs.levy.SNARLI.BPInitException,
                          edu.wlu.cs.levy.SNARLI.BPSizeException
Returns error of maximum magnitude on layer over all training patterns. Network.init must be called between network creation and this method, or an exception will be raised. The number of input and output patterns must be be equal, or an exception will be raised.

Returns:
maximum error
Throws:
BPInitException - if some network weights are uninitialized
BPSizeException - if number of input and output patterns differs

getSquaredErrors

public double[][] getSquaredErrors()
                            throws edu.wlu.cs.levy.SNARLI.BPInitException,
                                   edu.wlu.cs.levy.SNARLI.BPSizeException
Returns squared errors on layer over all training patterns. Network.init must be called between network creation and this method, or an exception will be raised. The number of input and output patterns must be be equal, or an exception will be raised.

Returns:
array of squared errors
Throws:
BPInitException - if some network weights are uninitialized
BPSizeException - if number of input and output patterns differs

connect

public void connect(edu.wlu.cs.levy.SNARLI.BPLayer from)
Makes a normal (full, Sigma) connection to this layer from another layer. The activations aj on this layer is computed as aj = f ( &sumiaiwji + bj ), where ai are the activations on the other layer, w are the weights between the layers, b are the biases on this layer, and f is the layer's activation function. Weights and biases can be initialized via the randomize or setWeights and setBias methods.

Parameters:
from - layer to connect from

connect

public void connect(edu.wlu.cs.levy.SNARLI.BPLayer from1,
                    edu.wlu.cs.levy.SNARLI.BPLayer from2)
Makes a Sigma-Pi connection to this layer from two other layers. The activation on this layer is computed as ak = f ( &sumi&sumj aiaj wkji + bk ), where ai and aj are the activations on the other two layers w are the weights between the layers, b are the biases on this layer, and f is the layer's activation function. Weights and biases can be initialized via the randomize or setWeights and setBias methods.

Parameters:
from1 - layer to connect from
from2 - layer to connect from

attach

public void attach(double[][] pattern)
            throws edu.wlu.cs.levy.SNARLI.BPSizeException
Attaches pattern to layer as input or target. Width (number of columns) of pattern must match layer size. Pattern matrices should contain one pattern per row. For example, A = {{0,0}, {0,1}, {1,0}, {1,1}}, B = {{0}, {1}, {1}, {0}} are the patterns for learning B = XOR(A). Don't-care conditions are encoded as out-of-bound values. For example, B = {{0}, {1}, {1}, {2}} would encode XOR where you don't care about the output B when A = {1,1}.

Parameters:
pattern - pattern to attach
Throws:
BPSizeException - on width/size mismatch

batch

public void batch(int nep,
                  double eta,
                  double mu,
                  int report)
           throws edu.wlu.cs.levy.SNARLI.BPInitException,
                  edu.wlu.cs.levy.SNARLI.BPSizeException
Trains all layer's in this layer's network, using back-propagation in batch mode. This means that the errors are computed over all patterns, and then averaged to compute the weight- and bias- changes. The lengths of any patterns attached to layers and output patterns must be be compatible, or an exception will be raised.

Parameters:
nep - number of epochs for training
eta - learning rate
mu - momentum
report - number of generations between error reports
Throws:
BPInitException - if some network weights are uninitialized
BPSizeException - if lengths of patterns differ

batch

public void batch(int nep,
                  double eta,
                  double mu)
           throws edu.wlu.cs.levy.SNARLI.BPInitException,
                  edu.wlu.cs.levy.SNARLI.BPSizeException
Same as batch(nep, eta, mu, report), but with error reporting every second.

Parameters:
nep - number of epochs for training
eta - learning rate
mu - momentum
Throws:
BPInitException - if weights hav not been initialized
BPSizeException - if number of input and output patterns differs

batch

public void batch(double eta,
                  double mu)
           throws edu.wlu.cs.levy.SNARLI.BPInitException,
                  edu.wlu.cs.levy.SNARLI.BPSizeException
Runs one step of back-propagation in batch mode.

Parameters:
eta - learning rate
mu - momentum
Throws:
BPInitException - if some network weights haven't been initialized
BPSizeException - if number of input and output patterns differs

online

public void online(int nep,
                   double eta,
                   double mu,
                   int report)
            throws edu.wlu.cs.levy.SNARLI.BPInitException,
                   edu.wlu.cs.levy.SNARLI.BPSizeException
Trains all layers in this layer's network, using back-propagation in on-line mode. This means that the weights and biases are updated after the presentation of each pattern. The lengths of any patterns attached to layers and output patterns must be be compatible, or an exception will be raised.

Parameters:
nep - number of passes through data
eta - learning rate
mu - momentum
report - number of generations between error reports
Throws:
BPInitException - if some network weights are uninitialized
BPSizeException - if lengths of patterns differ

online

public void online(int nep,
                   double eta,
                   double mu)
            throws edu.wlu.cs.levy.SNARLI.BPInitException,
                   edu.wlu.cs.levy.SNARLI.BPSizeException
Same as online(nep, eta, mu, report), but with error report every second.

Parameters:
nep - number of epochs for training
eta - learning rate
mu - momentum
Throws:
BPInitException - if weights hav not been initialized
BPSizeException - if number of input and output patterns differs

test

public double[][] test()
                throws edu.wlu.cs.levy.SNARLI.BPInitException,
                       edu.wlu.cs.levy.SNARLI.BPSizeException
Tests a (trained) network layer.

Returns:
output patterns
Throws:
BPInitException - if some network weights are uninitialized
BPSizeException - if number of input and output patterns differs

test

public double[][] test(int n,
                       double a)
                throws edu.wlu.cs.levy.SNARLI.BPInitException,
                       edu.wlu.cs.levy.SNARLI.BPSizeException
Tests a (recurrent) network layer without any input.

Parameters:
n - number of steps to run
a - intial activation value
Returns:
output pattern on layer
Throws:
BPInitException - if some network weights are uninitialized
BPSizeException - if number of input and output patterns differs

delay

public void delay(edu.wlu.cs.levy.SNARLI.BPLayer from,
                  double weight)
           throws edu.wlu.cs.levy.SNARLI.BPSizeException
Makes one-to-one time-delay connection to this layer from the specified layer, using the specified connection strength. This means that this layer's activations will come from a scalar multiple of the activations of the other layer on the previous time step. On the first time step these activations are are defined be the average value of this layer's activation function (e.g., 0.5 for the logistic simgoid squashing function). If the layers have different sizes, and exception will be raised.

Parameters:
from - layer to delay from
weight - connection strength
Throws:
BPSizeException - if layers have different sizes

delay

public void delay(edu.wlu.cs.levy.SNARLI.BPLayer from)
Same as delay(from, weight), with weight = 1.0.

Parameters:
from - layer to delay from

resetMu

public void resetMu()
Resets previous weight- and bias- changes. This is only necessary when running one epoch at a time.


bpttResetEta

public void bpttResetEta()
Resets current weight- and bias-changes for Back-Prop-Through-Time.


bpttUpdate

public void bpttUpdate(double eta,
                       double mu,
                       int npat)
                throws edu.wlu.cs.levy.SNARLI.BPMomentumException
Updates weights and bias on layer using the Delta Rule, for Back-Prop-Through-Time.

Parameters:
eta - learning rate
mu - momentum
npat - total number of patterns
Throws:
BPMomentumException - if no momentum has been set on the layer

activate

public double[] activate(edu.wlu.cs.levy.SNARLI.BPLayer src,
                         double[] clmp)
Activates this layer by clamping activations on another and running a forward pass. Clamped activations are temporary and do not affect patterns set by attach.

Parameters:
src - "source" layer to clamp
clmp - vector of clamping values
Returns:
activation of this layer
Throws:
BPConnectException - if there is no path between the layers
BPInitException - if some network weights are uninitialized

bpttPattern

public void bpttPattern(int n)
                 throws edu.wlu.cs.levy.SNARLI.BPInitException,
                        edu.wlu.cs.levy.SNARLI.BPSizeException,
                        java.lang.IllegalArgumentException
Steps through one pattern for Back-Prop Thru Time. Implements a restricted version of BPTT, specified by the number of ticks in the argument.

Parameters:
n - number of ticks from end of pattern
Throws:
BPInitException - if some network weights are uninitialized
BPSizeException - if lengths of patterns differ
java.lang.IllegalArgumentException - if n < 0 or n >= current pattern length

bpttPattern

public void bpttPattern()
                 throws edu.wlu.cs.levy.SNARLI.BPInitException,
                        edu.wlu.cs.levy.SNARLI.BPSizeException
Steps through one pattern for Back-Prop Thru Time. Implements the full BPTT algorithm on the pattern. Same as bptt(n), with n = current pattern length - 1.

Throws:
BPInitException - if some network weights are uninitialized
BPSizeException - if lengths of patterns differ

reportValue

public static void reportValue(int iter,
                               int maxit,
                               int report,
                               double value,
                               java.io.PrintStream stream)
Reports a value in a friendly way. This is useful for reporting RMS and maximum error when training one epoch at a time.

Parameters:
iter - iteration number
maxit - maximum number of iterations
report - reporting interval
value - value to report
stream - print stream that reports value