Normalization 101

New: we recommend using neural-data-normalizer to make you life easier

So I'm creating this post after this issue, where a user asked about how to feed data to a neural network, and it seemed to be useful for other folks as well as a nice example of data normalization.

The example training set was the following:

[
  {name:'workout', duration:'120', enjoy: true, time:1455063275, tags:['gym', 'weights']},
  {name:'lunch', duration:'45', enjoy: false, time:1455063275, tags:['salad', 'wine']},
  {name:'sleep', duration:'420', enjoy: true, time:1455063275, tags:['bed', 'romance']}
]

For neural nets to work you need to feed a training set, consisting of inputs and their desired outputs. In this case that would be like:

("workout",  120, ['gym', 'weights']) => 'is enjoyable'
("lunch",  45, ['salad', 'wine']) => 'is not enjoyable'
("sleep",  420, ['bed', 'romance']) => 'is enjoyable'

But neural networks don't know what "workout" or 45 or ['salad', 'wine'] are. They only understand a single input, containing only values between 0 and 1, and it has to have a fixed size, so all the inputs have the same length. So you need to normalize your input/output data.

The 'name' input can be normalized into categories. Let's say you have 3 categories: "workout", "lunch" and "sleep", each can be represented with a flag bit (note, this is commonly called binarization). So we can use 3 bits:

"workout" => 0, 0, 1
"lunch" => 0, 1, 0
"sleep" => 1, 0, 0

Then, the 'duration' can be normalized to a value between 0 and 1 setting up a maximum value, and dividing by it. So let's say your maximum duration is 1000, then your inputs would look like this:

120 => 0.12
45 => 0.045
420 = > 0.42

For the 'tags' categories you can use categories again, but combined. So let's say that you have 6 categories:

gym => 0,0,0,0,0,1
weights => 0,0,0,0,1,0
salad => 0,0,0,1,0,0
wine => 0,0,1,0,0,0
bed => 0,1,0,0,0,0
romance => 1,0,0,0,0,0

Then your inputs would look like this:

['gym', 'weights'] => 0,0,0,0,1,1
['salad', 'wine'] => 0,0,1,1,0,0
['bed', 'romance'] => 1,1,0,0,0,0

And finaly, 'enjoyable' is the easiest one since it is a boolean, it can be written as:

true => 1
false => 0

Putting all this together your training set would be like:

("workout",  120, ['gym', 'weights']) => 0,0,1 + 0.12, +  0,0,0,0,1,1 => [0,0,1,0.12,0,0,0,0,1,1]
("lunch",  45, ['salad', 'wine']) => 0,1,0 + 0.045, +  0,0,1,1,0,0 => [0,1,0,0.045,0,0,1,1,0,0] 
("sleep",  420, ['bed', 'romance']) => 1,0,0 + 0,42 + 1,1,0,0,0,0 => [1,0,0,0.42,1,1,0,0,0,0]

And your outputs:

true => 1 => [1]
false => 0 => [0]
true => 1 => [1]

Now, you need to translate this to synaptic. You need a network with 10 neurons in the input layer and 1 in the output layer (since that's the size of your inputs and outputs). You can choose from different Architecture. If the sequence of the set matters for the training, you need to use a network with context memory, like LSTM. If the sequence is not important, then you shoud use a Perceptron, which is context unaware.

What I mean is like, if this:

("workout",  120, ['gym', 'weights']) => true 
("lunch",  45, ['salad', 'wine']) => false 
("sleep",  420, ['bed', 'romance']) => true 

Should or should not be the same as this:

("lunch",  45, ['salad', 'wine']) => false 
("workout",  120, ['gym', 'weights']) => true 
("sleep",  420, ['bed', 'romance']) => true 

If, let's say, having lunch before working out would change the ouput of the sleep set, then you need to use a network that remembers it's previous activations (LSTM).

But to keep it simple let's say that the order of the set doesn't matter, and use a Perceptron.

var myNet = new Architect.Perceptron(10, 7, 1);

I created it with 10 inputs, 7 hidden neurons, and 1 output neuron. The number of hidden neurons can't be guessed straightforwardly usualy you use a number in between the number of inputs and outputs.

Now, you feed your training set to the network's trainer:

var trainingSet = [
  {
    input: [0,0,1,0.12,0,0,0,0,1,1],
    output: [1]
  },
  {
    input:  [0,1,0,0.045,0,0,1,1,0,0],
    output: [0]
  },
  {
    input:  [1,0,0,0.42,1,1,0,0,0,0],
    output: [1]
  }
]

var trainingOptions = {
  rate: .1,
  iterations: 20000,
  error: .005,
}

myNet.trainer.train(trainingSet, trainingOptions);

If you receive NaN in the result confirm your input/output values and that number of input nodes is no larger then the length of your input array.

The training options should be tweaked according to each special case, you can read more about the trainer options in the Trainer Documentation Page