1. [Publications](/index.php/publications)
2. Feedforward and Recurrent Neural Networks Backward Propagation and Hessian in Matrix Form
 
 # Feedforward and Recurrent Neural Networks Backward Propagation and Hessian in Matrix Form

  ![](/sites/default/files/styles/wide/public/publications/fnn_rnn_gradient_hessian_logo_small.png?itok=3JvGr7fd)

 In this paper we focus on the linear algebra theory behind feedforward (FNN) and recurrent (RNN) neural networks. We review backward propagation, including backward propagation through time (BPTT). Also, we obtain a new exact expression for Hessian, which represents second order effects. We show that for *t* time steps the weight gradient can be expressed as a rank-*t* matrix, while the weight Hessian is as a sum of *t*2 Kronecker products of rank-*1* and WTAW matrices, for some matrix A and weight matrix W. Also, we show that for a mini-batch of size *r*, the weight update can be expressed as a rank-*rt* matrix. Finally, we briefly comment on the eigenvalues of the Hessian matrix.



 ## Authors



Maxim Naumov (NVIDIA)

 

 

 ## Publication Date



Saturday, September 16, 2017

 

 ## Published in



[arXiv:1709.06080 \[cs.LG\]](https://arxiv.org/abs/1709.06080)

 

 ## Research Area



[High Performance Computing](/index.php/research-area/high-performance-computing)

[Artificial Intelligence and Machine Learning ](/index.php/research-area/machine-learning-artificial-intelligence)