Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Adaptive Text Denoising Network for Image Caption Editing

Adaptive Text Denoising Network for Image Caption Editing Image caption editing, which aims at editing the inaccurate descriptions of the images, is an interdisciplinary task of computer vision and natural language processing. As the task requires encoding the image and its corresponding inaccurate caption simultaneously and decoding to generate an accurate image caption, the encoder-decoder framework is widely adopted for image caption editing. However, existing methods mostly focus on the decoder, yet ignore a big challenge on the encoder: the semantic inconsistency between image and caption. To this end, we propose a novel Adaptive Text Denoising Network (ATD-Net) to filter out noises at the word level and improve the model’s robustness at sentence level. Specifically, at the word level, we design a cross-attention mechanism called Textual Attention Mechanism (TAM), to differentiate the misdescriptive words. The TAM is designed to encode the inaccurate caption word by word based on the content of both image and caption. At the sentence level, in order to minimize the influence of misdescriptive words on the semantic of an entire caption, we introduce a Bidirectional Encoder to extract the correct semantic representation from the raw caption. The Bidirectional Encoder is able to model the global semantics of the raw caption, which enhances the robustness of the framework. We extensively evaluate our proposals on the MS-COCO image captioning dataset and prove the effectiveness of our method when compared with the state-of-the-arts. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png ACM Transactions on Multimedia Computing Communications and Applications (TOMCCAP) Association for Computing Machinery

Adaptive Text Denoising Network for Image Caption Editing

Download PDF
18 pages

Loading next page...
 
/lp/association-for-computing-machinery/adaptive-text-denoising-network-for-image-caption-editing-vAvWvTN01h
Publisher
Association for Computing Machinery
Copyright
Copyright © 2023 Association for Computing Machinery.
ISSN
1551-6857
eISSN
1551-6865
DOI
10.1145/3532627
Publisher site
See Article on Publisher Site

Abstract

Image caption editing, which aims at editing the inaccurate descriptions of the images, is an interdisciplinary task of computer vision and natural language processing. As the task requires encoding the image and its corresponding inaccurate caption simultaneously and decoding to generate an accurate image caption, the encoder-decoder framework is widely adopted for image caption editing. However, existing methods mostly focus on the decoder, yet ignore a big challenge on the encoder: the semantic inconsistency between image and caption. To this end, we propose a novel Adaptive Text Denoising Network (ATD-Net) to filter out noises at the word level and improve the model’s robustness at sentence level. Specifically, at the word level, we design a cross-attention mechanism called Textual Attention Mechanism (TAM), to differentiate the misdescriptive words. The TAM is designed to encode the inaccurate caption word by word based on the content of both image and caption. At the sentence level, in order to minimize the influence of misdescriptive words on the semantic of an entire caption, we introduce a Bidirectional Encoder to extract the correct semantic representation from the raw caption. The Bidirectional Encoder is able to model the global semantics of the raw caption, which enhances the robustness of the framework. We extensively evaluate our proposals on the MS-COCO image captioning dataset and prove the effectiveness of our method when compared with the state-of-the-arts.

Journal

ACM Transactions on Multimedia Computing Communications and Applications (TOMCCAP) – Association for Computing Machinery

Published: Feb 3, 2023

Keywords: Image caption editing

References

  • Show and tell: A neural image caption generator
    [,
  • Deep visual-semantic alignments for generating image descriptions
    [,
  • Show, attend and tell: Neural image caption generation with visual attention
    [,
  • Self-critical sequence training for image captioning
    [,
  • Bottom-up and top-down attention for image captioning and visual question answering
    [,
  • Look and modify: Modification networks for image captioning
    [,
  • Show, edit and tell: A framework for editing image captions
    [,
  • Recall what you see continually using GridLSTM in image captioning
    [,
  • Show, tell and polish: Ruminant decoding for image captioning
    [,
  • Normalized and geometry-aware self-attention network for image captioning
    [,
  • Attention is all you need
    [,
  • Exploring visual relationship for image captioning
    [,
  • X-linear attention networks for image captioning
    [,
  • Attention on attention for image captioning
    [,
  • SCA-CNN: Spatial and channel-wise attention in convolutional networks for image captioning
    [,
  • When to look: Adaptive attention via a visual sentinel for image captioning
    [,
  • Sequence to sequence learning with neural networks
    [,
  • Neural machine translation by jointly learning to align and translate
    [,
  • Spatio-temporal memory attention for image captioning
    [,
  • Image captioning with end-to-end attribute detection and subsequent attributes prediction
    [,
  • Multi-level policy and reward-based deep reinforcement learning framework for image captioning
    [,
  • Say as you wish: Fine-grained control of image caption generation with bstract scene graphs
    [,
  • Re-caption: Saliency-enhanced image captioning through two-phase learning
    [,
  • Meshed-memory transformer for image captioning
    [,
  • Auto-encoding scene graphs for image captioning
    [,
  • Context-aware visual policy network for fine-grained image captioning
    [,
  • MSCap - multi-style image captioning with unpaired stylized text
    [,
  • BERT: Pre-training of deep bidirectional transformers for language understanding
    [,
  • Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning
    [,
  • Deep residual learning for image recognition
    [,
  • ImageNet large scale visual recognition challenge
    [,
  • Deep visual-semantic alignments for generating image descriptions
    [,
  • Bleu: A method for automatic evaluation of machine translation
    [,
  • Rouge: A package for automatic evaluation of summaries
    [,
  • Cider: Consensus-based image description evaluation
    [,
  • Visual Genome: Connecting language and vision using crowdsourced dense image annotations
    [,
  • Adam: A method for stochastic optimization
    [,
  • Recurrent fusion network for image captioning
    [,
  • Auto-encoding scene graphs for image captioning
    [,
  • Learning phrase representations using RNN encoder-decoder for statistical machine translation
    [,
  • Neural machine translation by jointly learning to align and translate
    [,
  • On using very large target vocabulary for neural machine translation
    [,
  • Effective approaches to attention-based neural machine translation
    [,
  • Encode, tag, realize: high-precision text editing
    [,
  • Recurrent inference in text editing
    [,
  • QuaSE: Sequence editing under quantifiable guidance
    [,
  • Denoising based sequence-to-sequence pre-training for text generation
    [,
  • Microsoft COCO: Common objects in context
    [,
  • Adaptively aligned image captioning via adaptive attention time
    [,
  • Look back and predict forward in image captioning
    [,
  • Show, recall, and tell: Image captioning with recall mechanism
    [,
  • Dual-level collaborative transformer for image captioning
    [,
  • Adaptively clustering-driven learning for visual relationship detection
    [,
  • DF-GAN: A simple and effective baseline for text-to-image synthesis
    [,
  • DualVGR: A dual-visual graph reasoning unit for video question answering
    [,
  • Facial chirality: Using self-face reflection to learn discriminative features for facial expression recognition
    [,
  • AU-assisted graph attention convolutional network for micro-expression recognition
    [,
  • FashionMirror: Co-attention feature-remapping virtual try-on with sequential template poses
    [,