RNN Outputs and Loss: How a Recurrent Network Predicts the Next Word

A recurrent neural network keeps a memory, but how does it make a prediction and how do we know how wrong it is? This video adds an output layer to an RNN and follows the sentence "the cat sat" with real numbers. See the output formula y_t = W_y h_t + b_y turn the hidden state into one score per word, softmax turn scores into probabilities, and categorical cross-entropy measure the mistake at each time step, then add up into a loss for the whole sentence. Also learn which output and loss to use for other sequence tasks: one output with sigmoid and binary cross-entropy for sentiment, or mean squared error for forecasting numbers.