Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Learning Click-Based Deep Structure-Preserving Embeddings with Visual Attention

Learning Click-Based Deep Structure-Preserving Embeddings with Visual Attention One fundamental problem in image search is to learn the ranking functions (i.e., the similarity between query and image). Recent progress on this topic has evolved through two paradigms: the text-based model and image ranker learning. The former relies on image surrounding texts, making the similarity sensitive to the quality of textual descriptions. The latter may suffer from the robustness problem when human-labeled query-image pairs cannot represent user search intent precisely. We demonstrate in this article that the preceding two limitations can be well mitigated by learning a cross-view embedding that leverages click data. Specifically, a novel click-based Deep Structure-Preserving Embeddings with visual Attention (DSPEA) model is presented, which consists of two components: deep convolutional neural networks followed by image embedding layers for learning visual embedding, and a deep neural networks for generating query semantic embedding. Meanwhile, visual attention is incorporated at the top of the convolutional neural network to reflect the relevant regions of the image to the query. Furthermore, considering the high dimension of the query space, a new click-based representation on a query set is proposed for alleviating this sparsity problem. The whole network is end-to-end trained by optimizing a large margin objective that combines cross-view ranking constraints with in-view neighborhood structure preservation constraints. On a large-scale click-based image dataset with 11.7 million queries and 1 million images, our model is shown to be powerful for keyword-based image search with superior performance over several state-of-the-art methods and achieves, to date, the best reported NDCG25 of 52.21%. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) Association for Computing Machinery

Learning Click-Based Deep Structure-Preserving Embeddings with Visual Attention

Download PDF
19 pages

Loading next page...
 
/lp/association-for-computing-machinery/learning-click-based-deep-structure-preserving-embeddings-with-visual-4GI6Rtg6dG

References

References for this paper are not available at this time. We will be adding them shortly, thank you for your patience.

Publisher
Association for Computing Machinery
Copyright
Copyright © 2019 ACM
ISSN
1551-6857
eISSN
1551-6865
DOI
10.1145/3328994
Publisher site
See Article on Publisher Site

Abstract

One fundamental problem in image search is to learn the ranking functions (i.e., the similarity between query and image). Recent progress on this topic has evolved through two paradigms: the text-based model and image ranker learning. The former relies on image surrounding texts, making the similarity sensitive to the quality of textual descriptions. The latter may suffer from the robustness problem when human-labeled query-image pairs cannot represent user search intent precisely. We demonstrate in this article that the preceding two limitations can be well mitigated by learning a cross-view embedding that leverages click data. Specifically, a novel click-based Deep Structure-Preserving Embeddings with visual Attention (DSPEA) model is presented, which consists of two components: deep convolutional neural networks followed by image embedding layers for learning visual embedding, and a deep neural networks for generating query semantic embedding. Meanwhile, visual attention is incorporated at the top of the convolutional neural network to reflect the relevant regions of the image to the query. Furthermore, considering the high dimension of the query space, a new click-based representation on a query set is proposed for alleviating this sparsity problem. The whole network is end-to-end trained by optimizing a large margin objective that combines cross-view ranking constraints with in-view neighborhood structure preservation constraints. On a large-scale click-based image dataset with 11.7 million queries and 1 million images, our model is shown to be powerful for keyword-based image search with superior performance over several state-of-the-art methods and achieves, to date, the best reported NDCG25 of 52.21%.

Journal

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) – Association for Computing Machinery

Published: Aug 8, 2019

Keywords: CNN

References

  • Neural machine translation by jointly learning to align and translate
    Bahdanau, Dzmitry; Cho, Kyunghyun; Bengio, Yoshua
  • Polynomial semantic indexing
    Bai, Bing; Weston, Jason; Grangier, David; Collobert, Ronan; Sadamasa, Kunihiko; Qi, Yanjun; Cortes, Corinna; Mohri, Mehryar
  • Bag-of-words based deep neural network for image retrieval
    Bai, Yalong; Yu, Wei; Xiao, Tianjun; Xu, Chang; Yang, Kuiyuan; Ma, Wei-Ying; Zhao, Tiejun
  • Large scale online learning of image similarity through ranking
    Chechik, Gal; Sharma, Varun; Shalit, Uri; Bengio, Samy
  • Discriminative feature selection for multi-view cross-domain learning
    Fang, Zheng; Zhang, Zhongfei Mark
  • Devise: A deep visual-semantic embedding model
    Frome, Andrea; Corrado, Greg S.; Shlens, Jonathon; Bengio, Samy; Dean, Jeffrey; Ranzato, Marc’Aurelio; Mikolov, Tomas
  • Statistical consistency of kernel canonical correlation analysis
    Fukumizu, Kenji; Bach, Francis R.; Gretton, Arthur
  • A multi-view embedding space for modeling Internet images, tags, and their semantics
    Gong, Yunchao; Ke, Qifa; Isard, Michael; Lazebnik, Svetlana
  • Canonical correlation analysis: An overview with application to learning methods
    Hardoon, David R.; Szedmak, Sandor; Shawe-Taylor, John
  • Deep residual learning for image recognition
    He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian
  • Cross-modal retrieval via deep and bidirectional representation learning
    He, Yonghao; Xiang, Shiming; Kang, Cuicui; Wang, Jian; Pan, Chunhong
  • Large margin rank boundaries for ordinal regression
    Herbrich, Ralf; Graepel, Thore; Obermayer, Klaus
  • Clickage: Towards bridging semantic and intent gaps via mining click logs of search engines
    Hua, Xian-Sheng; Yang, Linjun; Wang, Jingdong; Wang, Jing; Ye, Ming; Wang, Kuansan; Rui, Yong; Li, Jin
  • Learning to re-rank: Query-dependent image re-ranking using click data
    Jain, Vidit; Varma, Manik
  • Caffe: Convolutional architecture for fast feature embedding
    Jia, Yangqing; Shelhamer, Evan; Donahue, Jeff; Karayev, Sergey; Long, Jonathan; Girshick, Ross; Guadarrama, Sergio; Darrell, Trevor
  • Multi-view discriminant analysis
    Kan, Meina; Shan, Shiguang; Zhang, Haihong; Lao, Shihong; Chen, Xilin
  • Learning consistent feature representation for cross-modal multimedia retrieval
    Kang, Cuicui; Xiang, Shiming; Liao, Shengcai; Xu, Changsheng; Pan, Chunhong
  • ImageNet classification with deep convolutional neural networks
    Krizhevsky, Alex; Sutskever, Ilya; Hinton, Geoffrey E.
  • Share-and-chat: Achieving human-level video commenting by search and multi-view embedding
    Li, Yehao; Yao, Ting; Mei, Tao; Chao, Hongyang; Rui, Yong
  • Click-through-based deep visual-semantic embedding for image search
    Liu, Yuan; Shi, Zhongchao; Li, Xue; Wang, Gang
  • Laplacian support vector machines trained in the primal
    Melacci, Stefano; Belkin, Mikhail
  • Click-through-based cross-view learning for image search
    Pan, Yingwei; Yao, Ting; Mei, Tao; Li, Houqiang; Ngo, Chong-Wah; Rui, Yong
  • Click-through-based subspace learning for image search
    Pan, Yingwei; Yao, Ting; Tian, Xinmei; Li, Houqiang; Ngo, Chong-Wah
  • Image search by graph-based label propagation with image representation from DNN
    Pan, Yingwei; Yao, Ting; Yang, Kuiyuan; Li, Houqiang; Ngo, Chong-Wah; Wang, Jingdong; Mei, Tao
  • Okapi at TREC-3
    Robertson, S. E.; Walker, S.; Jones, S.; Hancock-Beaulieu, M.; Gatford, M.
  • On the behavior of randomization tests without a group invariance assumption
    Romano, Joseph P.
  • Overview and recent advances in partial least squares
    Rosipal, Roman; Krämer, Nicole
  • Introduction to Modern Information Retrieval
    Salton, Gerard; McGill, Michael J.
  • Attention is all you need
    Vaswani, Ashish; Shazeer, Noam; Parmar, Niki; Uszkoreit, Jakob; Jones, Llion; Gomez, Aidan N.; Kaiser, Łukasz; Polosukhin, Illia
  • Graph attention networks
    Veličković, Petar; Cucurull, Guillem; Casanova, Arantxa; Romero, Adriana; Liò, Pietro; Bengio, Yoshua
  • Show and tell: A neural image caption generator
    Vinyals, Oriol; Toshev, Alexander; Bengio, Samy; Erhan, Dumitru
  • What value do explicit high level concepts have in vision to language problems? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
    Wu, Qi; Shen, Chunhua; Liu, Lingqiao; Dick, Anthony; Van Den Hengel, Anton
  • Learning query and document similarities from click-through bipartite graph with metadata
    Wu, Wei; Li, Hang; Xu, Jun
  • Double-bit quantization and index hashing for nearest neighbor search
    Xie, Hongtao; Mao, Zhendong; Zhang, Yongdong; Deng, Han; Yan, Chenggang; Chen, Zhineng
  • Show, attend and tell: Neural image caption generation with visual attention
    Xu, Kelvin; Ba, Jimmy; Kiros, Ryan; Cho, Kyunghyun; Courville, Aaron; Salakhudinov, Ruslan; Zemel, Rich; Bengio, Yoshua
  • Deep correlation for matching images and text
    Yan, Fei; Mikolajczyk, Krystian
  • Effective transfer tagging from image to video
    Yang, Yang; Yang, Yi; Shen, Heng Tao
  • Stacked attention networks for image question answering
    Yang, Zichao; He, Xiaodong; Gao, Jianfeng; Deng, Li; Smola, Alex
  • Co-reranking by mutual reinforcement for image search
    Yao, Ting; Mei, Tao; Ngo, Chong-Wah
  • Learning query and image similarities with ranking canonical correlation analysis
    Yao, Ting; Mei, Tao; Ngo, Chong-Wah
  • Annotation for free: Video tagging by mining user search behavior
    Yao, Ting; Mei, Tao; Ngo, Chong-Wah; Li, Shipeng
  • Circular reranking for visual search
    Yao, Ting; Ngo, Chong-Wah; Mei, Tao
  • Semi-supervised domain adaptation with subspace learning for visual recognition
    Yao, Ting; Pan, Yingwei; Ngo, Chong-Wah; Li, Houqiang; Mei, Tao
  • A study of smoothing methods for language models applied to information retrieval
    Zhai, Chengxiang; Lafferty, John
  • Learning from collective intelligence: Feature learning using social images and tags
    Zhang, Hanwang; Shang, Xindi; Luan, Huanbo; Wang, Meng; Chua, Tat-Seng
  • Scalable similarity search with topology preserving hashing
    Zhang, Lei; Zhang, Yongdong; Gu, Xiaoguang; Tang, Jinhui; Tian, Qi