Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Deep Learning at Scale and at Ease

Deep Learning at Scale and at Ease Recently, deep learning techniques have enjoyed success in various multimedia applications, such as image classification and multimodal data analysis. Large deep learning models are developed for learning rich representations of complex data. There are two challenges to overcome before deep learning can be widely adopted in multimedia and other applications. One is usability, namely the implementation of different models and training algorithms must be done by nonexperts without much effort, especially when the model is large and complex. The other is scalability, namely the deep learning system must be able to provision for a huge demand of computing resources for training large models with massive datasets. To address these two challenges, in this article we design a distributed deep learning platform called SINGA, which has an intuitive programming model based on the common layer abstraction of deep learning models. Good scalability is achieved through flexible distributed training architecture and specific optimization techniques. SINGA runs on both GPUs and CPUs, and we show that it outperforms many other state-of-the-art deep learning systems. Our experience with developing and training deep learning models for real-life multimedia applications in SINGA shows that the platform is both usable and scalable. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) Association for Computing Machinery

Deep Learning at Scale and at Ease

Download PDF
25 pages

Loading next page...
 
/lp/association-for-computing-machinery/deep-learning-at-scale-and-at-ease-y8MoYbVwYw
Publisher
Association for Computing Machinery
Copyright
Copyright © 2016 Owner/Author
ISSN
1551-6857
eISSN
1551-6865
DOI
10.1145/2996464
Publisher site
See Article on Publisher Site

Abstract

Recently, deep learning techniques have enjoyed success in various multimedia applications, such as image classification and multimodal data analysis. Large deep learning models are developed for learning rich representations of complex data. There are two challenges to overcome before deep learning can be widely adopted in multimedia and other applications. One is usability, namely the implementation of different models and training algorithms must be done by nonexperts without much effort, especially when the model is large and complex. The other is scalability, namely the deep learning system must be able to provision for a huge demand of computing resources for training large models with massive datasets. To address these two challenges, in this article we design a distributed deep learning platform called SINGA, which has an intuitive programming model based on the common layer abstraction of deep learning models. Good scalability is achieved through flexible distributed training architecture and specific optimization techniques. SINGA runs on both GPUs and CPUs, and we show that it outperforms many other state-of-the-art deep learning systems. Our experience with developing and training deep learning models for real-life multimedia applications in SINGA shows that the platform is both usable and scalable.

Journal

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) – Association for Computing Machinery

Published: Nov 2, 2016

Keywords: Multimedia

References

  • TensorFlow: Large-scale machine learning on heterogeneous systems
    Abadi, Martín; Agarwal, Ashish; Barham, Paul; Brevdo, Eugene; Chen, Zhifeng; Citro, Craig; Corrado, Greg S.
  • Theano: New features and speed improvements
    Bastien, Frédéric; Lamblin, Pascal; Pascanu, Razvan; Bergstra, James; Goodfellow, Ian J.; Bergeron, Arnaud; Bouchard, Nicolas; Bengio, Yoshua
  • Training deep nets with sublinear memory cost
    Chen, Tianqi; Xu, Bing; Zhang, Chiyuan; Guestrin, Carlos
  • MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems
    Chen, Tianqi; Li, Mu; Li, Yutian; Lin, Min; Wang, Naiyan; Wang, Minjie; Xiao, Tianjun; Xu, Bing; Zhang, Chiyuan; Zhang, Zheng
  • Revisiting distributed synchronous SGD
    Chen, Jianmin; Monga, Rajat; Bengio, Samy; Józefowicz, Rafal
  • Project Adam: Building an efficient and scalable deep learning training system
    Chilimbi, Trishul; Suzue, Yutaka; Apacible, Johnson; Kalyanaraman, Karthik
  • Zheng
    Chua, Tat-Seng; Tang, Jinhui; Hong, Richang; Li, Haojie; Luo, Zhiping; Yan-Tao,
  • Deep big simple neural nets excel on handwritten digit recognition
    Ciresan, Dan Claudiu; Meier, Ueli; Gambardella, Luca Maria; Schmidhuber, Jürgen
  • Deep learning with COTS HPC systems
    Coates, Adam; Huval, Brody; Wang, Tao; Wu, David J.; Catanzaro, Bryan C.; Ng, Andrew Y.
  • Torch7: A Matlab-like environment for machine learning
    Collobert, R.; Kavukcuoglu, K.; Farabet, C.
  • Petuum: A framework for iterative-convergent distributed ML
    Dai, Wei; Wei, Jinliang; Zheng, Xun; Kim, Jin Kyu; Lee, Seunghak; Yin, Junming; Ho, Qirong; Xing, Eric P.
  • Large scale distributed deep networks
    Dean, Jeffrey; Corrado, Greg; Monga, Rajat; Chen, Kai; Devin, Matthieu; Le, Quoc V.; Mao, Mark Z.; Ranzato, Marc’Aurelio; Senior, Andrew W.; Tucker, Paul A.; Yang, Ke; Ng, Andrew Y.
  • Adaptive subgradient methods for online learning and stochastic optimization
    Duchi, John C.; Hazan, Elad; Singer, Yoram
  • Cross-modal retrieval with correspondence autoencoder
    Feng, Fangxiang; Wang, Xiaojie; Li, Ruifan
  • Deep residual learning for image recognition
    He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian
  • Reducing the dimensionality of data with neural networks
    Hinton, Geoffrey; Salakhutdinov, Ruslan
  • Caffe: Convolutional architecture for fast feature embedding
    Jia, Yangqing; Shelhamer, Evan; Donahue, Jeff; Karayev, Sergey; Long, Jonathan; Girshick, Ross; Guadarrama, Sergio; Darrell, Trevor
  • epiC: An extensible and scalable system for processing big data
    Jiang, Dawei; Chen, Gang; Ooi, Beng Chin; Tan, Kian-Lee; Wu, Sai
  • One weird trick for parallelizing convolutional neural networks
    Krizhevsky, Alex
  • ImageNet classification with deep convolutional neural networks
    Krizhevsky, Alex; Sutskever, Ilya; Hinton, Geoffrey E.
  • Building high-level features using large scale unsupervised learning
    Le, Quoc V.; Ranzato, Marc’Aurelio; Monga, Rajat; Devin, Matthieu; Corrado, Greg; Chen, Kai; Dean, Jeffrey; Ng, Andrew Y.
  • Efficient BackProp
    LeCun, Yann; Bottou, Léon; Orr, Genevieve B.; Müller, Klaus-Robert
  • Scaling distributed machine learning with the parameter server
    Li, Mu; Andersen, David G.; Park, Jun Woo; Smola, Alexander J.; Ahmed, Amr; Josifovski, Vanja; Long, James; Shekita, Eugene J.; Su, Bor-Yiing
  • Distributed representations of words and phrases and their compositionality
    Mikolov, Tomas; Sutskever, Ilya; Chen, Kai; Corrado, Gregory S.; Dean, Jeffrey
  • Extensions of recurrent neural network language model
    Mikolov, Tomas; Kombrink, Stefan; Burget, Lukás; Cernocký, Jan; Khudanpur, Sanjeev
  • SINGA: A distributed deep learning platform
    Ooi, Beng Chin; Tan, Kian-Lee; Wang, Sheng; Wang, Wei; Cai, Qingchao; Chen, Gang; Gao, Jinyang
  • GPU asynchronous stochastic gradient descent to speed up neural network training
    Paine, Thomas; Jin, Hailin; Yang, Jianchao; Lin, Zhe; Huang, Thomas S.
  • Hogwild: A lock-free approach to parallelizing stochastic gradient descent
    Recht, Benjamin; Re, Christopher; Wright, Stephen J.; Niu, Feng
  • 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs
    Seide, Frank; Fu, Hao; Droppo, Jasha; Li, Gang; Yu, Dong
  • Very deep convolutional networks for large-scale image recognition
    Simonyan, Karen; Zisserman, Andrew
  • Going deeper with convolutions
    Szegedy, Christian; Liu, Wei; Jia, Yangqing; Sermanet, Pierre; Reed, Scott; Anguelov, Dragomir; Erhan, Dumitru; Vanhoucke, Vincent; Rabinovich, Andrew
  • Giving meanings to WWW images
    Shen, Heng Tao; Ooi, Beng Chin; Tan, Kian-Lee
  • In-memory databases: Challenges and opportunities from software and hardware perspectives
    Tan, Kian-Lee; Cai, Qingchao; Ooi, Beng Chin; Wong, Weng-Fai; Yao, Chang; Zhang, Hao
  • Deep learning for content-based image retrieval: A comprehensive study
    Wan, Ji; Wang, Dayong; Hoi, Steven Chu Hong; Wu, Pengcheng; Zhu, Jianke; Zhang, Yongdong; Li, Jintao
  • Improving content-based and hybrid music recommendation using deep learning
    Wang, Xinxi; Wang, Ye
  • Effective multi-modal retrieval based on stacked auto-encoders
    Wang, Wei; Ooi, Beng Chin; Yang, Xiaoyan; Zhang, Dongxiang; Zhuang, Yueting
  • SINGA: Putting deep learning in the hands of multimedia users
    Wang, Wei; Chen, Gang; Dinh, Tien Tuan Anh; Gao, Jinyang; Ooi, Beng Chin; Tan, Kian-Lee; Wang, Sheng
  • Effective deep learning-based multi-modal retrieval
    Wang, Wei; Yang, Xiaoyan; Ooi, Beng Chin; Zhang, Dongxiang; Zhuang, Yueting
  • Deep Image: Scaling up image recognition
    Wu, Ren; Yan, Shengen; Shan, Yi; Dang, Qingqing; Sun, Gang
  • Exploring inter-feature and inter-class relationships with deep neural networks for video classification
    Wu, Zuxuan; Jiang, Yu-Gang; Wang, Jun; Pu, Jian; Xue, Xiangyang
  • Multi-GPU training of ConvNets
    Yadan, Omry; Adams, Keith; Taigman, Yaniv; Ranzato, Marc’Aurelio
  • Joint visual-textual sentiment analysis with deep neural networks
    You, Quanzeng; Luo, Jiebo; Jin, Hailin; Yang, Jianchao
  • An Introduction to Computational Networks and the Computational Network Toolkit
    Yu, Dong; Eversole, Adam; Seltzer, Mike; Yao, Kaisheng; Kuchaiev, Oleksii; Zhang, Yu; Seide, Frank
  • DimmWitted: A study of main-memory statistical analytics
    Zhang, Ce; Re, Christopher
  • Start from scratch: Towards automatically identifying, modeling, and naming visual attributes
    Zhang, Hanwang; Yang, Yang; Luan, Huan-Bo; Yang, Shuicheng; Chua, Tat-Seng