|Home | About | Journals | Submit | Contact Us | Français|
Technology offers the potential to objectively monitor people’s eating and activity behaviors and encourage healthier lifestyles. BALANCE is a mobile phone-based system for long term wellness management. The BALANCE system automatically detects the user’s caloric expenditure via sensor data from a Mobile Sensing Platform unit worn on the hip. Users manually enter information on foods eaten via an interface on an N95 mobile phone. Initial validation experiments measuring oxygen consumption during treadmill walking and jogging show that the system’s estimate of caloric output is within 87% of the actual value. Future work will refine and continue to evaluate the system’s efficacy and develop more robust data input and activity inference methods.
The United States and many other industrialized countries face an epidemic of obesity due to plentiful energy-dense foods and a lack of opportunities for physical activity. Research from the authors and others has shown that most people underestimate their caloric intake – what they eat  – and overestimate their caloric expenditure – the calories they burn . To help people make better lifestyle choices conducive to weight loss and weight control, we seek to provide an easy way to monitor the balance between their caloric intake and caloric expenditure. An ideal system would require a small time investment, minimize the cognitive burden on the user, and provide just-in-time feedback and encouragement. The easier the system is to use the more people will begin to use it, and continue to use it daily. Since this is a system that people will want to access continuously throughout their day, the mobile phone is an ideal platform for such a system. People carry their mobile phones with them at most times, meaning that a phone-based wellness system would provide ubiquitous access to information on caloric intake, expenditure, and balance. In addition to providing convenient access, it would be ideal for a wellness system to automatically detect the user’s activities and accurately infer the corresponding caloric expenditure objectively to reduce the error associated with self-reported activity levels.
We propose the BALANCE (Bioengineering Approaches for Lifestyle Activity and Nutrition Continuous Engagement) system, a mobile phone-based system that aims to integrate the above characteristics into a useful application for long term wellness management. A mobile phone interface with customization and adaptation makes food entry easier, and caloric expenditure is measured using a wearable sensor.
A number of commercial web applications exist – such as DietTV.com  – which facilitate nutritional awareness, dieting, and fitness. There are also smartphone and PDA-based applications for recording consumed foods and exercises, such as those sold by Keyoe . The iPhone App Store hosts a number of applications for recording nutritional and exercise history, including iShape and Absolute Fitness. Nokia’s Research Center offers the Wellness Diary Error! Reference source not found. for download – an application for Nokia S60 phones that allows users to input and track health statistics, including exercise and food choices.
Tsai et al.  developed the PmEB software for a mobile phone with a similar goal to BALANCE: allowing easy user input and review of their relative caloric intake and caloric expenditure. The authors found that users preferred to record their eaten foods and exercise sessions via the PmEB software rather than via a paper diary1. Consolvo et al.  developed the UbiFit wellness system for use on a mobile phone. UbiFit automatically detects five activities – walking, running, bicycling, and using a commercially available elliptical trainer or stair-stepping machine – and allows users to enter in other performed activities that may not be measured directly, such as swimming. The users’ activities nurture an electronic garden display on the mobile phone, giving users immediate feedback on their exercise level.
Siek et al.  developed a PDA-based application for semiliterate patients with kidney disease that helps them monitor their nutrient intakes. The system offers barcode scanning and voice-recording as a means of simplified food entry, but the open-source UPC database used for the project contained only 60% of the participants’ scanned barcodes. HyperFit  also allows users to “scan” barcodes using a camera phone and image processing techniques. The authors suggest that in addition to using this scanning to speed up the food entry process, sheets of barcodes can be kept and used as a list of favorite meals, snacks, and exercises.
Other research and commercial food-tracking systems use photographic input from a camera or a camera phone. These images are sometimes stored for later reference by the user, as in the DietSense prototype  and Kaczkowski et al.’s work . In other systems the photographs are sent off to a nutritionist for professional evaluation, as in Myca Nutrition’s Picture Food Journal Service (formerly MyFoodPhone)  and Wellnavi .
None of the above systems combine food input with objective activity inference. An upcoming product, Fitbit , purports to automatically detect activity levels and quality of sleep via a hip and wrist-mounted accelerometer, which it integrates with a web site where users can also log their food intake. The contribution of this work is providing a system which allows users to track both their food intake and their caloric expenditure in real-time on a device that accompanies the user (almost) everywhere. We believe that the omnipresence of the device will provide better opportunities to influence behavior when most appropriate: at the time when the user is making decisions.
Activity recognition using sensors has received a lot of attention in the research community over the past several years. Maurer et al.  use their multi-sensor platform for recognizing normal daily activities. Ganti et al.  use sensors embedded in clothing for activity recognition. Previous work from the authors  uses the Mobile Sensing Platform (MSP) for activity recognition. Work presented in this paper builds on our previous work by incorporating caloric expenditure estimations based on sensed physical activities with caloric inputs based on food intake.
Measuring caloric expenditure through physical activity is of great interest to the healthcare community and equipment vendors who market hardware to perform these measurements. The BodyMedia WMS and the Actical Physical Activity Monitor are two popular products in the market today that measure calorie expenditure. The system that is most closely related to ours is the IDEEA system, which attempts to measure both activity and energy expenditure by monitoring five sensors placed on different parts of the body . Using their system the authors were able to measure caloric expenditures within 95% of actual values. However, the system requires the five sensors to be physically linked together with wires and then connected to a personal computer for data collection; this is an unwieldy configuration that cannot be used to monitor daily activities. In addition, the IDEEA thus far has only been validated to estimate the energy expenditure of basic movement patterns such as sitting, standing, leaning against a wall, and walking or jogging on level ground.
The current implementation of our system uses a Nokia N95 cell phone in conjunction with the MSP (Figure 1). The MSP combines an Intel XScale processor with 8 different sensing capabilities including 3-D accelerometry, barometric pressure, light sensors, humidity, sound, and position using a GPS. The unit has modest storage capacity to perform calculations in real-time. The hardware is packaged in a box and meant to be worn on the waist, as shown above.
The MSP’s inference engine recognizes when a user is performing gross motor patterns such as sitting, walking, running and bicycling. Using the multiple sensors the unit can distinguish, for example, whether a person is walking up the stairs or riding an elevator up one flight. In initial proof-of-concept experiments, data was collected from various people and used to train a Naïve Bayes classifier. In our current validation experiments, when the system detects that the individual is walking or running it computes the step rate and tracks the number of steps the person has taken; this information is transmitted to the N95 over a Bluetooth link. In the current implementation, activity inferencing is done based only on data from the accelerometer.
We have conducted tests on subjects to collect data about their caloric expenditure while walking and jogging on the treadmill at various speeds and elevations (see Section 4) and while performing activities of daily living, such as sweeping, in a field test. We will extend our caloric expenditure calculations to handle more common daily activities so as to be able to calculate the caloric expenditure for activities done throughout the day. For the treadmill tests, the surface’s gradient was obtained from the treadmill itself. For the field tests we plan to leverage the MSP’s barometer and GPS to obtain this information.
Given the increasing availability of sensors on mobile devices like cell phones, we are currently working on building an activity inference engine that would work with data from the phone’s built-in accelerometer. Additionally, we will use data from the built-in GPS receiver to infer the subjects’ mode of transportation and account for that in the caloric expenditure calculations.
The BALANCE system has a mobile phone interface for entering consumed foods, adding custom exercises that are not detected, such as swimming or primarily upper-body movements, and providing a quick summary of the user’s current food-exercise balance. The software deals with calories in terms of ’Calorie Hundreds Impact Points’ (CHIPs), a simplified unit developed for use in the PACE Project ; a CHIP is equal to 100 calories.
Figure 2a shows the home screen of the application which displays the user’s “Personal Fuel Gauge” – her relative caloric intake and caloric expenditure. The details of a day’s food consumption are shown in more detail on a separate screen, as shown in Figure 2b.
To support ease of food entry, the BALANCE software consists of two primary food databases. The first is an extensive master database with detailed nutrition information. This database contains many very similar items, such as “Yogurt”, “Fruit Yogurt”, “Light Yogurt”, and so on. This makes it difficult and time-consuming to find specific foods when creating a food entry. Most people tend to eat a smaller number of foods fairly consistently, so there is an additional personal food database which consists of all foods that the individual user has ever entered. The first time the user eats a food, she is required to search the master database to find the item. The item is then copied into the personal food database, which makes it easier to re-enter in the future. The personal food database is also used to store custom entries, such as a favorite brand or flavor of yogurt that may not be in the master database.
In addition to being able to search the master and personal databases for foods that have been eaten, the software allows the specification of ‘Favorites.’ The ‘Favorites’ list includes both single food items and meals (or recipes) consisting of several food items. The user can construct a common meal – such as a typical breakfast of cornflakes with bananas, milk and orange juice – and then easily add all or part of the items to her daily food diary (Figure 2c). The software offers an interface for adding custom exercise entries that is similar to the food interface as well as a screen for viewing that day’s detected exercises (Figure 2d).
The BALANCE software employs strategies to streamline food entry by keeping track of entries over time. Anecdotally, nutritionists we consulted with observed that people tend to have little variance in what they eat for breakfast – that is, they tend to eat very similar things for breakfast every day – slightly more variance in what they eat for lunch, and vary greatly in what they eat for dinner. We can take advantage of these patterns by providing ordered, targeted lists of foods based on time of entry.
Another strategy we are beginning to explore is the use of location and time of day data to streamline food entry. We can imagine that some locations have very consistent food entries— consider your favorite coffee shop where you usually have a tall skim latte and biscotti. When the user creates a food entry at that location, the software can present the most common food entries at the top of the list. In contrast, at home you may not eat the same food every night at your dining room table, but in the morning you may always eat one of your three favorite breakfasts.
In addition to using context and history to modify the food input interface to streamline entry, we plan to investigate the use of context and history to remind the user of a potential missed entry. Revisiting the coffee shop scenario, one can imagine that when the user leaves the coffee shop and has not entered anything, the phone reminds her to enter something if appropriate. We realize that this feature may be met with mixed appreciation, and it will need to be investigated thoroughly to identify a pattern of helpful reminders rather than bothersome interruptions. In addition, this reminder feature may be something that changes over time: when the user is beginning the process of tracking the software can provide frequent reminders; as the user becomes more comfortable with the process it may be appropriate to provide reminders less often.
We are in the process of conducting paper prototype sessions to redesign the BALANCE application’s user interface. Paper prototype sessions have been conducted with 4 computer science graduate students; we are poised to perform a greater number of paper prototype sessions with a more diverse population.
In the near future we will conduct focus groups to receive feedback on the user experience of the BALANCE system. Approximately 5 participants at a time will be sent home with the system for 3 days and instructed to enter the foods and beverages consumed and any purposeful bouts of activity not currently sensed by the mobile platform. At the end of the 3 days the participants will meet with a researcher to answer questionnaires about their experiences and participate in group discussions about the system. Two user experience questionnaires will be administered to obtain scalar responses in addition to the open-ended feedback obtained during the focus groups
To validate the accuracy of our approach, we designed and carried out several experiments comparing our calculated caloric expenditure against the actual measured value.
As a starting point for our energy estimation we used equations that estimated caloric expenditure for sitting, walking, and running based on the American College of Sports Medicine’s Guidelines for Exercise Testing and Prescription . These equations are described in Figure 3. The three main components of the equations are: the resting metabolic rate (R), the horizontal component of movement (H), and the vertical component of movement (V). These equations provide an estimate of VO2 based on speed and grade, which are then multiplied by weight and duration2. VO2 is defined to be the volume of oxygen uptake for a person, in liters per minute. The VO2 estimate can be easily converted to caloric expenditure by multiplying the result by the caloric equivalent of 5 kcal/min .
The equations require knowledge of the subject’s speed, the surface’s grade, and the subject’s weight. Weight is easy to measure and does not change significantly over short periods of time, so our goal was to accurately quantify speed and grade. We plan to use the change in barometric pressure to measure grade change in field tests, but the tests we used for this study were on a treadmill in a lab setting where barometric pressure does not change in response to elevation changes. Thus, we use the grade as reported by the treadmill. This left us with determining speed as our main measurement obstacle.
Our lab tests are designed to test activities such as sitting, standing, and walking and running at different speeds and grades. After being outfitted with an MSP and apparatus for measuring VO2 (Figure 1), our subjects were asked to complete a three-minute sitting stage, a two minute standing stage, six treadmill stages of three minute walking/running intervals, and a final five minute sitting stage. The sitting and standing stages at the beginning of the test allowed us to measure the subject’s base metabolic rate, and the sitting stage at the end allowed us to measure the rate of recovery from an active state to a normal resting state. The six walking/running stages, described in Table 1, were always executed in random order. During our tests subjects were also fitted with a heart rate monitoring device to detect a physiologic response to exercise. If at any time during the test the subject exceeded 85% of her age-predicted maximum heart rate, calculated as (220 – subject age), the remaining high intensity stages were skipped and the subject continued to the next low intensity stage. Thus, not all subjects completed all stages. This should have little effect on our results since the skipped stages forced the subject into an intensity level outside of what we expect her to engage in during a normal day.
Our overall accuracy from our initial 10 treadmill tests was 87%. Generally our estimates were a little lower than actual, as shown in Figure 4. Changes in levels of activity cause the body to react by increasing or decreasing VO2 consumption relative to the increase or decrease in activity. However, this change is not instantaneous. There is a certain amount of lag associated with the change which is considerably more noticeable for large changes of activity – for example, going from sitting to walking briskly at 3.5 mph. This lag period also varies greatly from person to person depending on the person’s fitness level. A fit individual will have a much shorter lag period, as her body is able to reach a physiologic steady-state and recover from prior activity more quickly. An unfit individual’s lag will result in a much more gradual change towards steady state. Our system, however, reports the instantaneous change in activity, which is what caused a majority of our measurement error. Independent of the six treadmill stages, we performed experiments to compare the values reported by the two systems. Figure 5 shows a comparison between the actual VO2 reading and our estimate, illustrating some of the lags in caloric expenditure which occur in test subjects. Notably, the most significant divergence is in the sudden drop reported by our system, while the VO2 reading did not drop as much. This was due to the subject getting off the treadmill for a very short time period to adjust the test equipment. Since the interruption was very brief the VO2 did not change significantly, while the sensing equipment did detect the sudden drop in activity level.
Our validation work is ongoing and we anticipate a final sample of 65 subjects of varying ages and body weights. As of this writing, field tests are in progress to collect data from subjects involved in normal daily activities. We expect better accuracy for field tests because the physiologic change from rest to activity is much more subtle, resulting in a more rapid increase in VO2 to steady-state.
The major innovation of this bioengineering technology lies in its ability to capture peoples’ movement through space and time under free-living conditions. We believe this work will lead to numerous applications: for example, development of a personal assistive device that could be used to monitor energy balance in real-time, coupled with detailed feedback on movement through space and time, to facilitate behavior changes favorable to weight loss and/or weight control.
Our year 2 work will be to test, debug, and refine the cell phone prototype using several turns of the iterative cycle. In the third year we will perform rigorous validation experiments on the final, fully completed BALANCE system using VO2 and doubly-labeled water as criterion measures.
This work is funded in part by NIH R21AG032232 (Duncan, GE), BALANCE: Bioengineering Approaches for Lifestyle Activity and Nutrition Continuous Engagement, and NIH R21AG028719 (Duncan, GE), Ubiquitous Computing for the Measurement of Physical Activity. Other members of the BALANCE team not included here are: Shirley Beresford, Barbara Bruemmer, Deonna Chandra Hughes, Philip Hurvitz, and Anne Vernez Moudon.
This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in Proceedings of the 10th workshop on Mobile Computing Systems and Applications (HotMobile ‘09)
1A paper diary is a traditional monitoring method in the nutrition sciences. The subject takes home the diary and uses it over a specified time period – usually 3–7 days – during which the subject either estimates or measures all consumed foods and beverages and records them in the diary .
2Note that for sitting, H and V are both 0, so we only count the resting metabolic rate.