Access the full text.
Sign up today, get DeepDyve free for 14 days.
Loading next page...
/lp/association-for-computing-machinery/extrinsic-summarization-evaluation-a-decision-audit-task-G00brvCAck
- Publisher
- Association for Computing Machinery
- Copyright
- The ACM Portal is published by the Association for Computing Machinery. Copyright © 2010 ACM, Inc.
- Subject
- Evaluation/methodology
- ISSN
- 1550-4875
- DOI
- 10.1145/1596517.1596518
- Publisher site
- See Article on Publisher Site
Abstract
In this work we describe a large-scale extrinsic evaluation of automatic speech summarization technologies for meeting speech. The particular task is a decision audit, wherein a user must satisfy a complex information need, navigating several meetings in order to gain an understanding of how and why a given decision was made. We compare the usefulness of extractive and abstractive technologies in satisfying this information need, and assess the impact of automatic speech recognition (ASR) errors on user performance. We employ several evaluation methods for participant performance, including post-questionnaire data, human subjective and objective judgments, and a detailed analysis of participant browsing behavior. We find that while ASR errors affect user satisfaction on an information retrieval task, users can adapt their browsing behavior to complete the task satisfactorily. Results also indicate that users consider extractive summaries to be intuitive and useful tools for browsing multimodal meeting data. We discuss areas in which automatic summarization techniques can be improved in comparison with gold-standard meeting abstracts.
Journal
ACM Transactions on Speech and Language Processing (TSLP) – Association for Computing Machinery
Published: Oct 1, 2009