clarion.system
Class SimplifiedQBPNet

java.lang.Object
  extended by clarion.system.AbstractImplicitModule
      extended by clarion.system.AbstractTrainableImplicitModule
          extended by clarion.system.AbstractNeuralNet
              extended by clarion.system.BPNet
                  extended by clarion.system.AbstractRuntimeTrainableBPNet
                      extended by clarion.system.SimplifiedQBPNet
All Implemented Interfaces:
InterfaceHandlesFeedback, InterfaceRuntimeTrainable, InterfaceTracksMatchStatistics, InterfaceTrainable

public class SimplifiedQBPNet
extends AbstractRuntimeTrainableBPNet

This class implements a simplified Q-learning backpropagating neural network within CLARION. It extends the AbstractRuntimeTrainableBPNet class.

Usage:

A simplified Q-learning backpropagating neural network can be taken advantage of when reinforcement to train the network is recieved at every time step. This class is capable of performing learning during runtime.

Note that the weight vectors and thresholds for the network can be hardcoded if they were recorded from a previous training session (just like a standard backpropagating neural network).

The Input for the neural network is a collection of dimension-values where each value represents one node.

If the network is being used as an action network in the ACS and you are using goals or specialized working memory chunks, remember that the input space must also contain all dimension-value pairs within those chunks that differ from the sensory information space.

Once the input space has been defined it cannot be changed. At any given forward pass through the network, the network can accept an arbitrary collection of dimension-values as input, but it will only adjust the activations of the inputs that were specified during initialization.

The nodes in the output layer are represented as output chunks.

The general procedure when using this class is:

  1. setInput
  2. forwardPass
  3. setChosenAction
  4. setFeedback
  5. backwardPass
  6. Goto Step 1

Note: The current implementation of CLARION does not allow for multiple action dimensions on the bottom level as stipulated in the CLARION tutorial. However, actions that contain multiple action dimensions are still possible by simply specifying action chunks on the top level that contain multiple activated action dimensions. This is the case because the output of the bottom level only specifies actions and not action dimension-values.

Version:
6.0.4
Author:
Nick Wilson

Field Summary
 
Fields inherited from class clarion.system.AbstractRuntimeTrainableBPNet
Feedback, GLOBAL_POSITIVE_MATCH_THRESHOLD, NM, PM, POSITIVE_MATCH_THRESHOLD
 
Fields inherited from class clarion.system.BPNet
GLOBAL_LEARNING_RATE, GLOBAL_MOMENTUM, GLOBAL_RZERO, LEARNING_RATE, MOMENTUM, RZERO
 
Fields inherited from class clarion.system.AbstractNeuralNet
GLOBAL_LOWER_INIT_THRESHOLD, GLOBAL_LOWER_INIT_WEIGHT, GLOBAL_UPPER_INIT_THRESHOLD, GLOBAL_UPPER_INIT_WEIGHT, Hidden, HiddenThresholds, HiddenToOutputWeights, InputToHiddenWeights, LOWER_INIT_THRESHOLD, LOWER_INIT_WEIGHT, OutputThresholds, UPPER_INIT_THRESHOLD, UPPER_INIT_WEIGHT
 
Fields inherited from class clarion.system.AbstractTrainableImplicitModule
DesiredOutput
 
Fields inherited from class clarion.system.AbstractImplicitModule
ACTUATION_TIME, ChosenOutput, DECISION_TIME, GLOBAL_ACTUATION_TIME, GLOBAL_DECISION_TIME, GLOBAL_PERCEPTION_TIME, InputAsCollection, Output, PERCEPTION_TIME
 
Constructor Summary
SimplifiedQBPNet(java.util.Collection<Dimension> InputSpace, int NumHidden, AbstractOutputChunkCollection<? extends AbstractOutputChunk> Outputs)
          Initializes a backpropagating neural network that uses simplified Q-Learning for training the network.
 
Method Summary
 void backwardPass()
          Updates the neural network using simplified Q-Learning.
 boolean checkMatchCriterion()
          Checks to see if the positive match criterion is satisfied given the immediate feedback received.
 
Methods inherited from class clarion.system.AbstractRuntimeTrainableBPNet
getFeedback, getNM, getPM, incrementNM, incrementPM, resetMatchStatistics, setFeedback, setNM, setPM, updateMatchStatistics
 
Methods inherited from class clarion.system.BPNet
computeHiddenActivation, computeOutputActivation, modifyHiddenToOutput, modifyInputToHidden
 
Methods inherited from class clarion.system.AbstractNeuralNet
forwardPass, getHiddenThresholds, getHtoOWeightMatrix, getItoHWeightMatrix, getNumHidden, getOutputThresholds, hardcodeWeights
 
Methods inherited from class clarion.system.AbstractTrainableImplicitModule
getSumSqErrors, setDesiredOutput, setDesiredOutput
 
Methods inherited from class clarion.system.AbstractImplicitModule
getChosenOutput, getInput, getNumInput, getNumOutput, getOutput, getOutput, getResponseTime, setChosenOutput, setInput, setInput, setInput
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 
Methods inherited from interface clarion.system.InterfaceTrainable
getSumSqErrors, setDesiredOutput, setDesiredOutput
 

Constructor Detail

SimplifiedQBPNet

public SimplifiedQBPNet(java.util.Collection<Dimension> InputSpace,
                        int NumHidden,
                        AbstractOutputChunkCollection<? extends AbstractOutputChunk> Outputs)
Initializes a backpropagating neural network that uses simplified Q-Learning for training the network.

The Input for the neural network is a collection of dimension-values where each value represents one node.

If the network is being used as an action network in the ACS and you are using goals or specialized working memory chunks, remember that the input space must also contain all dimension-value pairs within those chunks that differ from the sensory information space.

Once input has been set it cannot be changed. At any given forward pass through the network, the network can accept an arbitrary collection of dimension-values as input, but it will only adjust the activations of the inputs that were specified during initialization.

The nodes in the output layer are represented as output chunks.

Parameters:
InputSpace - A collection of dimension-value pairs to set as the input nodes.
NumHidden - The number of hidden nodes.
Outputs - The chunks to associate with the output layer.
Method Detail

backwardPass

public void backwardPass()
Updates the neural network using simplified Q-Learning. This method should not be called before the setFeedback and setChosenAction methods have been called.

Specified by:
backwardPass in interface InterfaceTrainable
Overrides:
backwardPass in class BPNet

checkMatchCriterion

public boolean checkMatchCriterion()
Checks to see if the positive match criterion is satisfied given the immediate feedback received.

This check is usually performed before the backwardPass method is called but after the setFeedback and setChosenAction methods have been called.

Returns:
True if the positive match criterion has been satisfied, otherwise false.