Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Dynamic connection pruning for densely connected convolutional neural networks

Dynamic connection pruning for densely connected convolutional neural networks Densely connected convolutional neural networks dominate in a variety of downstream tasks due to their extraordinary performance. However, such networks typically require excessive computing resources, which hinders their deployment on mobile devices. In this paper, we propose a dynamic connection pruning algorithm, which is a cost-effective method to eliminate a large amount of redundancy in densely connected networks. First, we propose a Sample-Evaluation process to assess the contributions of connections. Specifically, sub-networks are sampled from the unpruned network in each epoch, while the parameters of the unpruned network are subsequently updated and the contributions of the connections are evaluated based on the performance of the sub-networks. Connections with low contribution will be pruned first. Then, we search for the distribution of pruning ratios by the Markov process. Finally, we prune the network based on the connection contribution and pruning ratios learned in the above two stages and obtain a lightweight network. The effectiveness of our method is verified on both high-level and low-level tasks. On the CIFAR-10 dataset, the top-1 accuracy barely drops (-0.03%) when FLOPs are reduced by 46.8%. In the super-resolution task, our model remarkably outperforms other lightweight networks in both visual and quantitative experiments. These results verify the effectiveness and generality of our proposed method. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Applied Intelligence Springer Journals

Dynamic connection pruning for densely connected convolutional neural networks

Download PDF
17 pages

Loading next page...
 
/lp/springer-journals/dynamic-connection-pruning-for-densely-connected-convolutional-neural-cWQHCPki1n
Publisher
Springer Journals
Copyright
Copyright © The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
ISSN
0924-669X
eISSN
1573-7497
DOI
10.1007/s10489-023-04513-8
Publisher site
See Article on Publisher Site

Abstract

Densely connected convolutional neural networks dominate in a variety of downstream tasks due to their extraordinary performance. However, such networks typically require excessive computing resources, which hinders their deployment on mobile devices. In this paper, we propose a dynamic connection pruning algorithm, which is a cost-effective method to eliminate a large amount of redundancy in densely connected networks. First, we propose a Sample-Evaluation process to assess the contributions of connections. Specifically, sub-networks are sampled from the unpruned network in each epoch, while the parameters of the unpruned network are subsequently updated and the contributions of the connections are evaluated based on the performance of the sub-networks. Connections with low contribution will be pruned first. Then, we search for the distribution of pruning ratios by the Markov process. Finally, we prune the network based on the connection contribution and pruning ratios learned in the above two stages and obtain a lightweight network. The effectiveness of our method is verified on both high-level and low-level tasks. On the CIFAR-10 dataset, the top-1 accuracy barely drops (-0.03%) when FLOPs are reduced by 46.8%. In the super-resolution task, our model remarkably outperforms other lightweight networks in both visual and quantitative experiments. These results verify the effectiveness and generality of our proposed method.

Journal

Applied Intelligence – Springer Journals

Published: Mar 8, 2023

Keywords: Model compression; Image classification; Super-resolution; Connection pruning; Architecture search; DenseNet

References

  • Structured pruning for efficient convolutional neural networks via incremental regularization
    Wang, H; Hu, X; Zhang, Q; Wang, Y; Yu, L; Hu, H
  • Residual dense network for image restoration
    Zhang, Y; Tian, Y; Kong, Y; Zhong, B; Fu, Y
  • Fine-grained neural architecture search for image super-resolution
    Kim, H; Hong, S; Han, B; Myeong, H; Lee, KM
  • Automatic densenet sparsification
    Li, T; Jiao, W; Wang, L-N; Zhong, G
  • Evolutionary neural architecture search for high-dimensional skip-connection structures on densenet style networks
    O’Neill, D; Xue, B; Zhang, M
  • An introduction to deep reinforcement learning
    François-Lavet, V; Henderson, P; Islam, R; Bellemare, MG; Pineau, J
  • Evolutionary algorithms and their applications to engineering problems
    Slowik, A; Kwasnicka, H
  • Multipath feedforward network for single image super-resolution
    Shen, M; Yu, P; Wang, R; Yang, J; Xue, L; Hu, M
  • Image super-resolution using deep convolutional networks
    Dong, C; Loy, CC; He, K; Tang, X
  • Efficient processing of deep neural networks: a tutorial and survey
    Sze, V; Chen, Y-H; Yang, T-J; Emer, JS
  • Pruning and quantization for deep neural network acceleration: a survey
    Liang, T; Glossner, J; Wang, L; Shi, S; Zhang, X
  • Dnn pruning with principal component analysis and connection importance estimation
    Riera, M; Arnau, JM; González, A