PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Proc IEEE Workshop Mob Comput Syst Appl. Author manuscript; available in PMC 2010 May 4.
Published in final edited form as:
Proc IEEE Workshop Mob Comput Syst Appl. 2009; 2009: 5.
doi:  10.1145/1514411.1514416
PMCID: PMC2863147
NIHMSID: NIHMS137738

BALANCE: Towards a Usable Pervasive Wellness Application with Accurate Activity Inference

Abstract

Technology offers the potential to objectively monitor people’s eating and activity behaviors and encourage healthier lifestyles. BALANCE is a mobile phone-based system for long term wellness management. The BALANCE system automatically detects the user’s caloric expenditure via sensor data from a Mobile Sensing Platform unit worn on the hip. Users manually enter information on foods eaten via an interface on an N95 mobile phone. Initial validation experiments measuring oxygen consumption during treadmill walking and jogging show that the system’s estimate of caloric output is within 87% of the actual value. Future work will refine and continue to evaluate the system’s efficacy and develop more robust data input and activity inference methods.

General Terms: Design, Human Factors, Verification
Keywords: pervasive health, long term health monitoring, wellness, caloric balance

1. INTRODUCTION

The United States and many other industrialized countries face an epidemic of obesity due to plentiful energy-dense foods and a lack of opportunities for physical activity. Research from the authors and others has shown that most people underestimate their caloric intake – what they eat [3][10] – and overestimate their caloric expenditure – the calories they burn [6]. To help people make better lifestyle choices conducive to weight loss and weight control, we seek to provide an easy way to monitor the balance between their caloric intake and caloric expenditure. An ideal system would require a small time investment, minimize the cognitive burden on the user, and provide just-in-time feedback and encouragement. The easier the system is to use the more people will begin to use it, and continue to use it daily. Since this is a system that people will want to access continuously throughout their day, the mobile phone is an ideal platform for such a system. People carry their mobile phones with them at most times, meaning that a phone-based wellness system would provide ubiquitous access to information on caloric intake, expenditure, and balance. In addition to providing convenient access, it would be ideal for a wellness system to automatically detect the user’s activities and accurately infer the corresponding caloric expenditure objectively to reduce the error associated with self-reported activity levels.

We propose the BALANCE (Bioengineering Approaches for Lifestyle Activity and Nutrition Continuous Engagement) system, a mobile phone-based system that aims to integrate the above characteristics into a useful application for long term wellness management. A mobile phone interface with customization and adaptation makes food entry easier, and caloric expenditure is measured using a wearable sensor.

2. RELATED WORKS

2.1 Wellness Software

A number of commercial web applications exist – such as DietTV.com [5] – which facilitate nutritional awareness, dieting, and fitness. There are also smartphone and PDA-based applications for recording consumed foods and exercises, such as those sold by Keyoe [13]. The iPhone App Store hosts a number of applications for recording nutritional and exercise history, including iShape and Absolute Fitness. Nokia’s Research Center offers the Wellness Diary Error! Reference source not found. for download – an application for Nokia S60 phones that allows users to input and track health statistics, including exercise and food choices.

Tsai et al. [21] developed the PmEB software for a mobile phone with a similar goal to BALANCE: allowing easy user input and review of their relative caloric intake and caloric expenditure. The authors found that users preferred to record their eaten foods and exercise sessions via the PmEB software rather than via a paper diary1. Consolvo et al. [4] developed the UbiFit wellness system for use on a mobile phone. UbiFit automatically detects five activities – walking, running, bicycling, and using a commercially available elliptical trainer or stair-stepping machine – and allows users to enter in other performed activities that may not be measured directly, such as swimming. The users’ activities nurture an electronic garden display on the mobile phone, giving users immediate feedback on their exercise level.

Siek et al. [20] developed a PDA-based application for semiliterate patients with kidney disease that helps them monitor their nutrient intakes. The system offers barcode scanning and voice-recording as a means of simplified food entry, but the open-source UPC database used for the project contained only 60% of the participants’ scanned barcodes. HyperFit [9] also allows users to “scan” barcodes using a camera phone and image processing techniques. The authors suggest that in addition to using this scanning to speed up the food entry process, sheets of barcodes can be kept and used as a list of favorite meals, snacks, and exercises.

Other research and commercial food-tracking systems use photographic input from a camera or a camera phone. These images are sometimes stored for later reference by the user, as in the DietSense prototype [19] and Kaczkowski et al.’s work [12]. In other systems the photographs are sent off to a nutritionist for professional evaluation, as in Myca Nutrition’s Picture Food Journal Service (formerly MyFoodPhone) [17] and Wellnavi [22].

None of the above systems combine food input with objective activity inference. An upcoming product, Fitbit [7], purports to automatically detect activity levels and quality of sleep via a hip and wrist-mounted accelerometer, which it integrates with a web site where users can also log their food intake. The contribution of this work is providing a system which allows users to track both their food intake and their caloric expenditure in real-time on a device that accompanies the user (almost) everywhere. We believe that the omnipresence of the device will provide better opportunities to influence behavior when most appropriate: at the time when the user is making decisions.

2.2 Activity Recognition

Activity recognition using sensors has received a lot of attention in the research community over the past several years. Maurer et al. [16] use their multi-sensor platform for recognizing normal daily activities. Ganti et al. [8] use sensors embedded in clothing for activity recognition. Previous work from the authors [14] uses the Mobile Sensing Platform (MSP) for activity recognition. Work presented in this paper builds on our previous work by incorporating caloric expenditure estimations based on sensed physical activities with caloric inputs based on food intake.

Measuring caloric expenditure through physical activity is of great interest to the healthcare community and equipment vendors who market hardware to perform these measurements. The BodyMedia WMS and the Actical Physical Activity Monitor are two popular products in the market today that measure calorie expenditure. The system that is most closely related to ours is the IDEEA system, which attempts to measure both activity and energy expenditure by monitoring five sensors placed on different parts of the body [23]. Using their system the authors were able to measure caloric expenditures within 95% of actual values. However, the system requires the five sensors to be physically linked together with wires and then connected to a personal computer for data collection; this is an unwieldy configuration that cannot be used to monitor daily activities. In addition, the IDEEA thus far has only been validated to estimate the energy expenditure of basic movement patterns such as sitting, standing, leaning against a wall, and walking or jogging on level ground.

3. BALANCE SYSTEM

3.1 Activity Sensing

The current implementation of our system uses a Nokia N95 cell phone in conjunction with the MSP (Figure 1). The MSP combines an Intel XScale processor with 8 different sensing capabilities including 3-D accelerometry, barometric pressure, light sensors, humidity, sound, and position using a GPS. The unit has modest storage capacity to perform calculations in real-time. The hardware is packaged in a box and meant to be worn on the waist, as shown above.

Figure 1
Photographs of the system hardware. (left) The Mobile Sensing Platform (MSP) shown worn on the hip. (middle) The Nokia N95 mobile phone. (right) Test subject wearing apparatus attached to a metabolic cart for measuring oxygen consumption during treadmill ...

The MSP’s inference engine recognizes when a user is performing gross motor patterns such as sitting, walking, running and bicycling. Using the multiple sensors the unit can distinguish, for example, whether a person is walking up the stairs or riding an elevator up one flight. In initial proof-of-concept experiments, data was collected from various people and used to train a Naïve Bayes classifier. In our current validation experiments, when the system detects that the individual is walking or running it computes the step rate and tracks the number of steps the person has taken; this information is transmitted to the N95 over a Bluetooth link. In the current implementation, activity inferencing is done based only on data from the accelerometer.

3.1.1 Future Work

We have conducted tests on subjects to collect data about their caloric expenditure while walking and jogging on the treadmill at various speeds and elevations (see Section 4) and while performing activities of daily living, such as sweeping, in a field test. We will extend our caloric expenditure calculations to handle more common daily activities so as to be able to calculate the caloric expenditure for activities done throughout the day. For the treadmill tests, the surface’s gradient was obtained from the treadmill itself. For the field tests we plan to leverage the MSP’s barometer and GPS to obtain this information.

Given the increasing availability of sensors on mobile devices like cell phones, we are currently working on building an activity inference engine that would work with data from the phone’s built-in accelerometer. Additionally, we will use data from the built-in GPS receiver to infer the subjects’ mode of transportation and account for that in the caloric expenditure calculations.

3.2 Food Input Interface

The BALANCE system has a mobile phone interface for entering consumed foods, adding custom exercises that are not detected, such as swimming or primarily upper-body movements, and providing a quick summary of the user’s current food-exercise balance. The software deals with calories in terms of ’Calorie Hundreds Impact Points’ (CHIPs), a simplified unit developed for use in the PACE Project [2]; a CHIP is equal to 100 calories.

Figure 2a shows the home screen of the application which displays the user’s “Personal Fuel Gauge” – her relative caloric intake and caloric expenditure. The details of a day’s food consumption are shown in more detail on a separate screen, as shown in Figure 2b.

Figure 2
Screenshots from the BALANCE software.

To support ease of food entry, the BALANCE software consists of two primary food databases. The first is an extensive master database with detailed nutrition information. This database contains many very similar items, such as “Yogurt”, “Fruit Yogurt”, “Light Yogurt”, and so on. This makes it difficult and time-consuming to find specific foods when creating a food entry. Most people tend to eat a smaller number of foods fairly consistently, so there is an additional personal food database which consists of all foods that the individual user has ever entered. The first time the user eats a food, she is required to search the master database to find the item. The item is then copied into the personal food database, which makes it easier to re-enter in the future. The personal food database is also used to store custom entries, such as a favorite brand or flavor of yogurt that may not be in the master database.

In addition to being able to search the master and personal databases for foods that have been eaten, the software allows the specification of ‘Favorites.’ The ‘Favorites’ list includes both single food items and meals (or recipes) consisting of several food items. The user can construct a common meal – such as a typical breakfast of cornflakes with bananas, milk and orange juice – and then easily add all or part of the items to her daily food diary (Figure 2c). The software offers an interface for adding custom exercise entries that is similar to the food interface as well as a screen for viewing that day’s detected exercises (Figure 2d).

3.2.1 Future Work

The BALANCE software employs strategies to streamline food entry by keeping track of entries over time. Anecdotally, nutritionists we consulted with observed that people tend to have little variance in what they eat for breakfast – that is, they tend to eat very similar things for breakfast every day – slightly more variance in what they eat for lunch, and vary greatly in what they eat for dinner. We can take advantage of these patterns by providing ordered, targeted lists of foods based on time of entry.

Another strategy we are beginning to explore is the use of location and time of day data to streamline food entry. We can imagine that some locations have very consistent food entries— consider your favorite coffee shop where you usually have a tall skim latte and biscotti. When the user creates a food entry at that location, the software can present the most common food entries at the top of the list. In contrast, at home you may not eat the same food every night at your dining room table, but in the morning you may always eat one of your three favorite breakfasts.

In addition to using context and history to modify the food input interface to streamline entry, we plan to investigate the use of context and history to remind the user of a potential missed entry. Revisiting the coffee shop scenario, one can imagine that when the user leaves the coffee shop and has not entered anything, the phone reminds her to enter something if appropriate. We realize that this feature may be met with mixed appreciation, and it will need to be investigated thoroughly to identify a pattern of helpful reminders rather than bothersome interruptions. In addition, this reminder feature may be something that changes over time: when the user is beginning the process of tracking the software can provide frequent reminders; as the user becomes more comfortable with the process it may be appropriate to provide reminders less often.

3.2.2 User Studies

We are in the process of conducting paper prototype sessions to redesign the BALANCE application’s user interface. Paper prototype sessions have been conducted with 4 computer science graduate students; we are poised to perform a greater number of paper prototype sessions with a more diverse population.

In the near future we will conduct focus groups to receive feedback on the user experience of the BALANCE system. Approximately 5 participants at a time will be sent home with the system for 3 days and instructed to enter the foods and beverages consumed and any purposeful bouts of activity not currently sensed by the mobile platform. At the end of the 3 days the participants will meet with a researcher to answer questionnaires about their experiences and participate in group discussions about the system. Two user experience questionnaires will be administered to obtain scalar responses in addition to the open-ended feedback obtained during the focus groups

4. VALIDATION OF CALORIC OUTPUT MEASUREMENTS

To validate the accuracy of our approach, we designed and carried out several experiments comparing our calculated caloric expenditure against the actual measured value.

As a starting point for our energy estimation we used equations that estimated caloric expenditure for sitting, walking, and running based on the American College of Sports Medicine’s Guidelines for Exercise Testing and Prescription [15]. These equations are described in Figure 3. The three main components of the equations are: the resting metabolic rate (R), the horizontal component of movement (H), and the vertical component of movement (V). These equations provide an estimate of VO2 based on speed and grade, which are then multiplied by weight and duration2. VO2 is defined to be the volume of oxygen uptake for a person, in liters per minute. The VO2 estimate can be easily converted to caloric expenditure by multiplying the result by the caloric equivalent of 5 kcal/min [1].

Figure 3
Metabolic equations.

The equations require knowledge of the subject’s speed, the surface’s grade, and the subject’s weight. Weight is easy to measure and does not change significantly over short periods of time, so our goal was to accurately quantify speed and grade. We plan to use the change in barometric pressure to measure grade change in field tests, but the tests we used for this study were on a treadmill in a lab setting where barometric pressure does not change in response to elevation changes. Thus, we use the grade as reported by the treadmill. This left us with determining speed as our main measurement obstacle.

Our lab tests are designed to test activities such as sitting, standing, and walking and running at different speeds and grades. After being outfitted with an MSP and apparatus for measuring VO2 (Figure 1), our subjects were asked to complete a three-minute sitting stage, a two minute standing stage, six treadmill stages of three minute walking/running intervals, and a final five minute sitting stage. The sitting and standing stages at the beginning of the test allowed us to measure the subject’s base metabolic rate, and the sitting stage at the end allowed us to measure the rate of recovery from an active state to a normal resting state. The six walking/running stages, described in Table 1, were always executed in random order. During our tests subjects were also fitted with a heart rate monitoring device to detect a physiologic response to exercise. If at any time during the test the subject exceeded 85% of her age-predicted maximum heart rate, calculated as (220 – subject age), the remaining high intensity stages were skipped and the subject continued to the next low intensity stage. Thus, not all subjects completed all stages. This should have little effect on our results since the skipped stages forced the subject into an intensity level outside of what we expect her to engage in during a normal day.

Table 1
Experimental Walking/Running Stages

Our overall accuracy from our initial 10 treadmill tests was 87%. Generally our estimates were a little lower than actual, as shown in Figure 4. Changes in levels of activity cause the body to react by increasing or decreasing VO2 consumption relative to the increase or decrease in activity. However, this change is not instantaneous. There is a certain amount of lag associated with the change which is considerably more noticeable for large changes of activity – for example, going from sitting to walking briskly at 3.5 mph. This lag period also varies greatly from person to person depending on the person’s fitness level. A fit individual will have a much shorter lag period, as her body is able to reach a physiologic steady-state and recover from prior activity more quickly. An unfit individual’s lag will result in a much more gradual change towards steady state. Our system, however, reports the instantaneous change in activity, which is what caused a majority of our measurement error. Independent of the six treadmill stages, we performed experiments to compare the values reported by the two systems. Figure 5 shows a comparison between the actual VO2 reading and our estimate, illustrating some of the lags in caloric expenditure which occur in test subjects. Notably, the most significant divergence is in the sudden drop reported by our system, while the VO2 reading did not drop as much. This was due to the subject getting off the treadmill for a very short time period to adjust the test equipment. Since the interruption was very brief the VO2 did not change significantly, while the sensing equipment did detect the sudden drop in activity level.

Figure 4
Comparison of calculated energy expenditure as compared to actual energy expenditure for 10 subjects.
Figure 5
Estimated (red, rectilinear) vs. actual caloric expenditure, as measured by VO2, (blue, curvier) over the span of one test.

Our validation work is ongoing and we anticipate a final sample of 65 subjects of varying ages and body weights. As of this writing, field tests are in progress to collect data from subjects involved in normal daily activities. We expect better accuracy for field tests because the physiologic change from rest to activity is much more subtle, resulting in a more rapid increase in VO2 to steady-state.

5. CONCLUSION

The major innovation of this bioengineering technology lies in its ability to capture peoples’ movement through space and time under free-living conditions. We believe this work will lead to numerous applications: for example, development of a personal assistive device that could be used to monitor energy balance in real-time, coupled with detailed feedback on movement through space and time, to facilitate behavior changes favorable to weight loss and/or weight control.

Our year 2 work will be to test, debug, and refine the cell phone prototype using several turns of the iterative cycle. In the third year we will perform rigorous validation experiments on the final, fully completed BALANCE system using VO2 and doubly-labeled water as criterion measures.

ACKNOLWEDGEMENTS

This work is funded in part by NIH R21AG032232 (Duncan, GE), BALANCE: Bioengineering Approaches for Lifestyle Activity and Nutrition Continuous Engagement, and NIH R21AG028719 (Duncan, GE), Ubiquitous Computing for the Measurement of Physical Activity. Other members of the BALANCE team not included here are: Shirley Beresford, Barbara Bruemmer, Deonna Chandra Hughes, Philip Hurvitz, and Anne Vernez Moudon.

Footnotes

This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in Proceedings of the 10th workshop on Mobile Computing Systems and Applications (HotMobile ‘09)

1A paper diary is a traditional monitoring method in the nutrition sciences. The subject takes home the diary and uses it over a specified time period – usually 3–7 days – during which the subject either estimates or measures all consumed foods and beverages and records them in the diary [11].

2Note that for sitting, H and V are both 0, so we only count the resting metabolic rate.

REFERENCES

1. Whaley M senior, editor. American College of Sports Medicine. ACSM's guidelines for exercise testing and prescription. 7th ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2006.
2. Beresford SAA, Locke E, Bishop S, West BA, McGregor BA, Bruemmer B, Duncan GE, Thompson B. Worksite study promoting activity and changes in eating (PACE): design and baseline results. Obesity. 2007;15 Suppl 1:4S–15S. [PubMed]
3. Brunner E, Stallone D, Juneja M, Bingham S, Marmot M. Dietary assessment in Whitehall II: comparison of 7 d diet diary and food-frequency questionnaire and validity against biomarkers. British Journal of Nutrition. 2001;86(3):405–414. [PubMed]
4. Consolvo S, McDonald DW, Toscos T, Chen MY, Froehlich J, Harrison B, Klasnja P, LaMarca A, LeGrand L, Libby R, Smith I, Landay JA. Activity Sensing in the Wild: A Field Trial of UbiFit Garden. Proceedings of the Twenty-sixth Annual SIGCHI Conference on Human Factors in Computing Systems (SIGCHI ’08); Florence, Italy. 2008. pp. 1797–1806.
6. Duncan GE, Sydeman SJ, Perri MG, Limacher MC, Martin AD. Can sedentary adults accurately recall the intensity of their physical activity? Preventative Medicine. 2001;33:18–26. [PubMed]
7. Fitbit, Inc. http://www.fitbit.com/
8. Ganti RK, Jayachandran P, Abdelzaher TF, Stankovic JA. Satire: A Software Architecture for Smart Attire. Proceedings of the 4th International Conference on Mobile Systems, Application and Services; Uppsala, Sweden. 2006. pp. 110–123.
9. Järvinen P, Järvinen TH, Lähteenmäki L, Södergård C. HyperFit: Hybrid Media in Personal Nutrition and Exercise Management. Proceedings of the 2nd International Conference on Pervasive Computing Technologies for Healthcare.2008.
10. Johansson G, Wikman A, Ahrn AM, Hallmans G, Johansson I. Underreporting of energy intake in repeated 24-hour recalls related to gender, age, weight status, day of interview, educational level, reported food intake, smoking habits and area of living. Public Health Nutrition. 2001;4(4):919–927. [PubMed]
11. Johnson RK, Yon BA, Hankin JH. Research: Successful Approaches. Third Edition. American Dietetic Association; 2008. Dietary Assessment and Validation.
12. Kaczkowski CH, Jones PJH, Feng J, Bayley HS. Four-Day Multimedia Diet Records Underestimate Energy Needs in Middle-Aged and Elderly Women as Determined by Doubly-Labeled Water. Journal of Nutrition. 2000;130(4):802–805. [PubMed]
13. Keyoe, Inc. http://www.keyoe.com/
14. Lester J, Choudhury T, Borriello G. A Practical Approach to Recognizing Physical Activities. Proceedings of the Fourth International Conference on Pervasive Computing; Dublin, Ireland. 2006.
15. Mahler D, Froelicher V, Miller N, York T. American College of Sports Medicine's Guidelines for Exercise Testing and Prescription. Williams & Wilkins: 1995.
16. Maurer U, Smailagic A, Siewiorek D, Deisher M. Activity Recognition and Monitoring Using Multiple Sensors on Different Body Positions. Proceedings of the International Workshop on Wearable and Implantable Body Sensor Networks (BSN ‘06).2006.
17. Myca Nutrition. http://www.mycanutrition.com/
19. Reddy S, Parker A, Hyman J, Burke J, Estrin D, Hansen M. Image Browsing, Processing, and Clustering for Participatory Sensing: Lessons from a DietSense Prototype. Proceedings of the 4th Workshop on Embedded Sensor Networks (EmNets ’07); New York, NY,USA. 2007. pp. 13–17.
20. Siek K, Connelly KH, Rogers Y, Rohwer P, Lambert D, Welch JL. When Do We Eat? An Evaluation of Food Items Input into an Electronic Monitoring Application. Proceedings of the First International Conference on Pervasive Computing Technologies for Healthcare (Pervasive Health); Innsbruck, Austria. 2006.
21. Tsai CC, Lee G, Raab F, Norman GJ, Sohn T, Griswold WG, Patrick K. Usability and Feasibility of PmEB: A Mobile Phone Application for Monitoring Real Time Caloric Balance. Mobile Networks and Applications. 12(2–3):173–184.
22. Wang D, Kogashiwa M, Kira S. Development of a New Instrument for Evaluating Individuals’ Dietary Intakes. Journal of the American Dietetic Association. 2006;20:1588–1593. 2006. [PubMed]
23. Zhang K, Pi-Sunyer F, Boozer C. Improving Energy Expenditure Estimation for Physical Activity. Medicine and Science in Sports and Exercise. 2004 [PubMed]