Learning with Marginalized Corruption
A long stream of strong empirical evidences have documented the drastic improvement to learning by exploiting “artificial” data, generated by corrupting training samples. In this vein, we have recently proposed and studied a new learning framework, learning with marginalized corruption (MC). Our approaches are based on the observation that the corrupting process is asymptotically equivalent to learning with a modified loss function. The hallmark of those approaches is to derive by marginalizing out the corruption in closed-form. In words, we can effectively learn with infinite amounts of (corrupted) data from without actually increasing neither the data set size nor the computational cost. Our results so far have been very strong, achieving orders of magnitude speed-ups over explicit corruption and further improvements in learning. Our work also has interesting connections to very recent work in deep learning, where MC is a form of marginalized dropout heuristic for shallow classifiers.
[Talk at John’s Hopkins University, Spring 2013]
Relevant publications:
[PDF][CODE][BIBTEX] Minmin Chen, Kilian Q. Weinberger, Fei Sha, Yoshua Bengio, Marginalized Denoising Auto-encoders for Nonlinear Representations. International Conference on Machine Learning (ICML), Beijing China (in press ...)
[PDF][CODE][BIBTEX] Minmin Chen, Alice Zheng, Kilian Q. Weinberger. Fast Image Tagging. Proceedings of 30th International Conference on Machine Learning (ICML), Atlanta, GA, pages 1274-1282, 2013.
[PDF][CODE][BIBTEX] Laurens van der Maaten, Minmin Chen, Stephen Tyree, and Kilian Q. Weinberger. Learning with Marginalized Corrupted Features. Proceedings of 30th International Conference on Machine Learning (ICML), Atlanta, GA, pages 410-418, 2013.
[PDF][CODE][BIBTEX] Zhixiang (Eddie) Xu, Minmin Chen, Kilian Q. Weinberger, Fei Sha. From sBoW to dCoT: Marginalized Encoders for Text Representation. Proc. of 21st ACM Conf. of Information and Knowledge Management (CIKM), Hawaii, 2012 (In press.)
[PDF][CODE][BIBTEX] Minmin Chen, Zhixiang (Eddie) Xu, Kilian Q. Weinberger, Fei Sha. Marginalized Stacked Denoising Autoencoders for Domain Adaptation. Proceedings of 29th International Conference on Machine Learning (ICML), Edingburgh Scotland, Omnipress, pages 767-774, 2012.
[PDF][BIBTEX] Zhixiang Eddie Xu, Kilian Q. Weinberger, Fei Sha. Rapid Feature Learning with Stacked Linear Denoisers. arXiv:1105.0972, 2011 (Technical report, not peer-reviewed.) Presented at ICML 2011 Workshop on Unsupervised and Transfer Learning.