Automated estimation of food type and amount consumed from body-worn audio and motion sensors

Mark Mirtchouk, Christopher Merck, Samantha Kleinberg

Research output: Chapter in Book/Report/Conference proceedingConference contribution

23 Citations (Scopus)

Abstract

Determining when an individual is eating can be useful for tracking behavior and identifying patterns, but to create nutrition logs automatically or provide real-time feedback to people with chronic disease, we need to identify both what they are consuming and in what quantity. However, food type and amount have mainly been estimated using image data (requiring user involvement) or acoustic sensors (tested with a restricted set of foods rather than representative meals). As a result, there is not yet a highly accurate automated nutrition monitoring method that can be used with a variety of foods. We propose that multi-modal sensing (in-ear audio plus head and wrist motion) can be used to more accurately classify food type, as audio and motion features provide complementary information. Further, we propose that knowing food type is critical for estimating amount consumed in combination with sensor data. To test this we use data from people wearing audio and motion sensors, with ground truth annotated from video and continuous scale data. With data from 40 unique foods we achieve a classification accuracy of 82.7% with a combination of sensors (versus 67.8% for audio alone and 76.2% for head and wrist motion). Weight estimation error was reduced from a baseline of 127.3% to 35.4% absolute relative error. Ultimately, our estimates of food type and amount can be linked to food databases to provide automated calorie estimates from continuously-collected data.

Original languageEnglish (US)
Title of host publicationUbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing
PublisherAssociation for Computing Machinery, Inc
Pages451-462
Number of pages12
ISBN (Electronic)9781450344616
DOIs
StatePublished - Sep 12 2016
Event2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp 2016 - Heidelberg, Germany
Duration: Sep 12 2016Sep 16 2016

Publication series

NameUbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing

Other

Other2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp 2016
CountryGermany
CityHeidelberg
Period9/12/169/16/16

Fingerprint

Sensors
Nutrition
Error analysis
Acoustics
Feedback
Monitoring

All Science Journal Classification (ASJC) codes

  • Software
  • Information Systems
  • Human-Computer Interaction
  • Hardware and Architecture
  • Computer Networks and Communications

Cite this

Mirtchouk, M., Merck, C., & Kleinberg, S. (2016). Automated estimation of food type and amount consumed from body-worn audio and motion sensors. In UbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (pp. 451-462). (UbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing). Association for Computing Machinery, Inc. https://doi.org/10.1145/2971648.2971677
Mirtchouk, Mark ; Merck, Christopher ; Kleinberg, Samantha. / Automated estimation of food type and amount consumed from body-worn audio and motion sensors. UbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. Association for Computing Machinery, Inc, 2016. pp. 451-462 (UbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing).
@inproceedings{e8421965afd544bb861c4cffba7b17be,
title = "Automated estimation of food type and amount consumed from body-worn audio and motion sensors",
abstract = "Determining when an individual is eating can be useful for tracking behavior and identifying patterns, but to create nutrition logs automatically or provide real-time feedback to people with chronic disease, we need to identify both what they are consuming and in what quantity. However, food type and amount have mainly been estimated using image data (requiring user involvement) or acoustic sensors (tested with a restricted set of foods rather than representative meals). As a result, there is not yet a highly accurate automated nutrition monitoring method that can be used with a variety of foods. We propose that multi-modal sensing (in-ear audio plus head and wrist motion) can be used to more accurately classify food type, as audio and motion features provide complementary information. Further, we propose that knowing food type is critical for estimating amount consumed in combination with sensor data. To test this we use data from people wearing audio and motion sensors, with ground truth annotated from video and continuous scale data. With data from 40 unique foods we achieve a classification accuracy of 82.7{\%} with a combination of sensors (versus 67.8{\%} for audio alone and 76.2{\%} for head and wrist motion). Weight estimation error was reduced from a baseline of 127.3{\%} to 35.4{\%} absolute relative error. Ultimately, our estimates of food type and amount can be linked to food databases to provide automated calorie estimates from continuously-collected data.",
author = "Mark Mirtchouk and Christopher Merck and Samantha Kleinberg",
year = "2016",
month = "9",
day = "12",
doi = "https://doi.org/10.1145/2971648.2971677",
language = "English (US)",
series = "UbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing",
publisher = "Association for Computing Machinery, Inc",
pages = "451--462",
booktitle = "UbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing",

}

Mirtchouk, M, Merck, C & Kleinberg, S 2016, Automated estimation of food type and amount consumed from body-worn audio and motion sensors. in UbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. UbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Association for Computing Machinery, Inc, pp. 451-462, 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp 2016, Heidelberg, Germany, 9/12/16. https://doi.org/10.1145/2971648.2971677

Automated estimation of food type and amount consumed from body-worn audio and motion sensors. / Mirtchouk, Mark; Merck, Christopher; Kleinberg, Samantha.

UbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. Association for Computing Machinery, Inc, 2016. p. 451-462 (UbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Automated estimation of food type and amount consumed from body-worn audio and motion sensors

AU - Mirtchouk, Mark

AU - Merck, Christopher

AU - Kleinberg, Samantha

PY - 2016/9/12

Y1 - 2016/9/12

N2 - Determining when an individual is eating can be useful for tracking behavior and identifying patterns, but to create nutrition logs automatically or provide real-time feedback to people with chronic disease, we need to identify both what they are consuming and in what quantity. However, food type and amount have mainly been estimated using image data (requiring user involvement) or acoustic sensors (tested with a restricted set of foods rather than representative meals). As a result, there is not yet a highly accurate automated nutrition monitoring method that can be used with a variety of foods. We propose that multi-modal sensing (in-ear audio plus head and wrist motion) can be used to more accurately classify food type, as audio and motion features provide complementary information. Further, we propose that knowing food type is critical for estimating amount consumed in combination with sensor data. To test this we use data from people wearing audio and motion sensors, with ground truth annotated from video and continuous scale data. With data from 40 unique foods we achieve a classification accuracy of 82.7% with a combination of sensors (versus 67.8% for audio alone and 76.2% for head and wrist motion). Weight estimation error was reduced from a baseline of 127.3% to 35.4% absolute relative error. Ultimately, our estimates of food type and amount can be linked to food databases to provide automated calorie estimates from continuously-collected data.

AB - Determining when an individual is eating can be useful for tracking behavior and identifying patterns, but to create nutrition logs automatically or provide real-time feedback to people with chronic disease, we need to identify both what they are consuming and in what quantity. However, food type and amount have mainly been estimated using image data (requiring user involvement) or acoustic sensors (tested with a restricted set of foods rather than representative meals). As a result, there is not yet a highly accurate automated nutrition monitoring method that can be used with a variety of foods. We propose that multi-modal sensing (in-ear audio plus head and wrist motion) can be used to more accurately classify food type, as audio and motion features provide complementary information. Further, we propose that knowing food type is critical for estimating amount consumed in combination with sensor data. To test this we use data from people wearing audio and motion sensors, with ground truth annotated from video and continuous scale data. With data from 40 unique foods we achieve a classification accuracy of 82.7% with a combination of sensors (versus 67.8% for audio alone and 76.2% for head and wrist motion). Weight estimation error was reduced from a baseline of 127.3% to 35.4% absolute relative error. Ultimately, our estimates of food type and amount can be linked to food databases to provide automated calorie estimates from continuously-collected data.

UR - http://www.scopus.com/inward/record.url?scp=84991508711&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84991508711&partnerID=8YFLogxK

U2 - https://doi.org/10.1145/2971648.2971677

DO - https://doi.org/10.1145/2971648.2971677

M3 - Conference contribution

T3 - UbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing

SP - 451

EP - 462

BT - UbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing

PB - Association for Computing Machinery, Inc

ER -

Mirtchouk M, Merck C, Kleinberg S. Automated estimation of food type and amount consumed from body-worn audio and motion sensors. In UbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. Association for Computing Machinery, Inc. 2016. p. 451-462. (UbiComp 2016 - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing). https://doi.org/10.1145/2971648.2971677