1. [Publications](/publications)
2. Temporal Ensembling for Semi-Supervised Learning
 
 # Temporal Ensembling for Semi-Supervised Learning

  ![](/sites/default/files/styles/wide/public/publications/laine2017iclr_thumb.png?itok=Gt7tMmmX)

 In this paper, we present a simple and efficient method for training deep neural networks in a semi-supervised setting where only a small portion of training data is labeled. We introduce self-ensembling, where we form a consensus prediction of the unknown labels using the outputs of the network-in-training on different epochs, and most importantly, under different regularization and input augmentation conditions. This ensemble prediction can be expected to be a better predictor for the unknown labels than the output of the network at the most recent training epoch, and can thus be used as a target for training. Using our method, we set new records for two standard semi-supervised learning benchmarks, reducing the (non-augmented) classification error rate from 18.44% to 7.05% in SVHN with 500 labels and from 18.63% to 16.55% in CIFAR-10 with 4000 labels, and further to 5.12% and 12.16% by enabling the standard augmentations. We additionally obtain a clear improvement in CIFAR-100 classification accuracy by using random images from the Tiny Images dataset as unlabeled extra inputs during training. Finally, we demonstrate good tolerance to incorrect labels.



 ## Authors



[Samuli Laine](/person/samuli-laine)

[Timo Aila](/person/timo-aila)

 

 

 ## Publication Date



Friday, October 7, 2016

 

 ## Published in



[ICLR 2017](http://www.iclr.cc/doku.php?id=iclr2017:main)

 

 ## Research Area



[Artificial Intelligence and Machine Learning ](/research-area/machine-learning-artificial-intelligence)

 

 

 ## External Links



[ArXiv](https://arxiv.org/abs/1610.02242)

 

 

 ## Uploaded Files



[Paper (PDF)](https://research.nvidia.com/sites/default/files/publications/laine2017iclr_paper.pdf "Open file in new window")307.32 KB