Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Multimodal Fusion of Smart Home and Text-based Behavior Markers for Clinical Assessment Prediction

Multimodal Fusion of Smart Home and Text-based Behavior Markers for Clinical Assessment Prediction New modes of technology are offering unprecedented opportunities to unobtrusively collect data about people's behavior. While there are many use cases for such information, we explore its utility for predicting multiple clinical assessment scores. Because clinical assessments are typically used as screening tools for impairment and disease, such as mild cognitive impairment (MCI), automatically mapping behavioral data to assessment scores can help detect changes in health and behavior across time. In this article, we aim to extract behavior markers from two modalities, a smart home environment and a custom digital memory notebook app, for mapping to 10 clinical assessments that are relevant for monitoring MCI onset and changes in cognitive health. Smart-home-based behavior markers reflect hourly, daily, and weekly activity patterns, while app-based behavior markers reflect app usage and writing content/style derived from free-form journal entries. We describe machine learning techniques for fusing these multimodal behavior markers and utilizing joint prediction. We evaluate our approach using three regression algorithms and data from 14 participants with MCI living in a smart-home environment. We observed moderate to large correlations between predicted and ground-truth assessment scores, ranging from r = 0.601 to r = 0.871 for each clinical assessment. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png ACM Transactions on Computing for Healthcare (HEALTH) Association for Computing Machinery

Multimodal Fusion of Smart Home and Text-based Behavior Markers for Clinical Assessment Prediction

Download PDF
25 pages

Loading next page...
 
/lp/association-for-computing-machinery/multimodal-fusion-of-smart-home-and-text-based-behavior-markers-for-UOvY7JY5wV
Publisher
Association for Computing Machinery
Copyright
Copyright © 2022 Association for Computing Machinery.
ISSN
2691-1957
eISSN
2637-8051
DOI
10.1145/3531231
Publisher site
See Article on Publisher Site

Abstract

New modes of technology are offering unprecedented opportunities to unobtrusively collect data about people's behavior. While there are many use cases for such information, we explore its utility for predicting multiple clinical assessment scores. Because clinical assessments are typically used as screening tools for impairment and disease, such as mild cognitive impairment (MCI), automatically mapping behavioral data to assessment scores can help detect changes in health and behavior across time. In this article, we aim to extract behavior markers from two modalities, a smart home environment and a custom digital memory notebook app, for mapping to 10 clinical assessments that are relevant for monitoring MCI onset and changes in cognitive health. Smart-home-based behavior markers reflect hourly, daily, and weekly activity patterns, while app-based behavior markers reflect app usage and writing content/style derived from free-form journal entries. We describe machine learning techniques for fusing these multimodal behavior markers and utilizing joint prediction. We evaluate our approach using three regression algorithms and data from 14 participants with MCI living in a smart-home environment. We observed moderate to large correlations between predicted and ground-truth assessment scores, ranging from r = 0.601 to r = 0.871 for each clinical assessment.

Journal

ACM Transactions on Computing for Healthcare (HEALTH) – Association for Computing Machinery

Published: Nov 3, 2022

Keywords: Behavior markers

References

  • MCI is associated with deficits in everyday functioning
    [,
  • Creating a digital memory notebook application for individuals with mild cognitive impairment to support everyday functioning
    [,
  • Smart home-driven digital memory notebook support of activity self-management for older adults
    [,
  • Using smart homes to detect and analyze health events
    [,
  • Fusing ambient and mobile sensor features into a behaviorome for predicting clinical health scores
    [,
  • Automated cognitive health assessment from smart home-based behavior data
    [,
  • Unobtrusive detection of mild cognitive impairment in older adults through home monitoring
    [,
  • Smart home-based prediction of multidomain symptoms related to Alzheimer's disease
    [,
  • Predicting MCI status from multimodal language data using cascaded classifiers
    [,
  • Partnering a compensatory application with activity-aware prompting to improve use in individuals with amnestic mild cognitive impairment: A randomized controlled pilot clinical trial
    [,
  • Enhancing activity recognition using CPD-based activity segmentation
    [,
  • Data-driven activity prediction: algorithms, evaluation methodology, and applications
    [,
  • Modeling patterns of activities using activity curves
    [,
  • Behavioral differences between subject groups identified using smart homes and change point detection
    [,
  • Automated cognitive health assessment in smart homes using machine learning
    [,
  • A smart-home system to unobtrusively and continuously assess loneliness in older adults
    [,
  • Is cognitive impairment associated with reduced syntactic complexity in writing? Evidence from automated text analysis
    [,
  • Detection of depression-related posts in Reddit social media forum
    [,
  • Predicting Alzheimer's disease from spoken and written language using fusion-based stacked generalization
    [,
  • Language impairment in adults with end-stage liver disease: Application of natural language processing towards patient-generated health records
    [,
  • Towards early dementia detection: Fusing linguistic and non-linguistic clinical data
    [,
  • Combining text mining of long constructed responses and item-based measures: A hybrid test design to screen for posttraumatic stress disorder (PTSD)
    [,
  • Sönmez and Y
    [,
  • Linguistic features and automatic classifiers for identifying mild cognitive impairment and dementia
    [,
  • Automatic speech classifier for mild cognitive impairment and early dementia
    [,
  • Multimodal fusion of BERT-CNN and gated CNN representations for depression detection
    [,
  • Using automatic assessment of speech production to predict current and future cognitive function in older adults
    [,
  • Tackling the adress challenge: A multimodal approach to the automated recognition of Alzheimer's dementia
    [,
  • An automatic system for dementia detection using acoustic and linguistic features
    [,
  • Classifying Alzheimer's disease using audio and text-based representations of speech
    [,
  • A review on recognizing depression in social networks: Challenges and opportunities
    [,
  • Alzheimer's dementia recognition through spontaneous speech: The ADReSS challenge
    [,
  • A systematic review of natural language processing and text mining of symptoms from electronic patient-authored text data
    [,
  • Identifying mild cognitive impairment and mild Alzheimer's disease based on spontaneous speech using ASR and linguistic features
    [,
  • Multimodal machine learning: a survey and taxonomy
    [,
  • Patient similarity and joint features for rehabilitation outcome prediction
    [,
  • The development of a manual-based digital memory notebook intervention with case study illustrations
    [,
  • CASAS: A smart home in a box
    [,
  • Criteria for mild cognitive impairment due to Alzheimer's disease in the community
    [,
  • The repeatable battery for the assessment of neuropsychological status (RBANS): Preliminary clinical validity
    [,
  • Wechsler Test of Adult Reading
    [,
  • Delis-Kaplan executive function system: Examiner's manual
    [,
  • Prospective and retrospective memory in normal ageing and dementia: A questionnaire study
    [,
  • Development and psychometric properties of the instrumental activities of daily living: Compensation scale
    [,
  • Geriatric depression scale (GDS): Recent evidence and development of a shorter version
    [,
  • Quality of life in Alzheimer's disease: Patient and caregiver reports
    [,
  • A validity and reliability study of the coping self-efficacy scale
    [,
  • The satisfaction with life scale
    [,
  • The development and psychometric properties of LIWC2015
    [,
  • Automatic detection and classification of cognitive distortions in mental health text
    [,
  • Tracking depression dynamics in college students using mobile phone and wearable sensing
    [,
  • Multimodal integration-a statistical view
    [,
  • Wrappers for feature subset selection
    [,
  • Auto-encoder based dimensionality reduction
    [,
  • Gradient boosting for high-dimensional prediction of rare events
    [,