SRI’s Human Sleep Lab puts consumer wearables to the test
As part of a broad range of new consumer sleep-tracking technology, wearable devices are at the frontier of new technologies, with companies innovating new applications and features for devices at breakneck speed. Clinicians and researchers have taken note, leading many to wonder whether using commercial devices could advance their work and complement the use of standard clinical and research methods.
Dr. Massimiliano de Zambotti understands the promise of commercial wearables for clinical science. As a scientist in the SRI International Human Sleep Research Program, Dr. de Zambotti recalled the first time he encountered consumer-focused wearable technology and realized its enormous potential:
“At the time, we were doing a laboratory study with full in-lab polysomnography in adolescents. One of the research participants arrived at the lab wearing a Jawbone UP. The Jawbone device was originally designed for counting steps and measuring activity, but I realized that it also had a sleep feature. This device came with a companion app. From the app, you could see the sleep pattern. At that time, we were using a research-grade version of actigraphy, a simple wrist accelerometer that allowed us to analyze the subject’s typical sleep/wake patterns before bringing them to the lab. However, this device was expensive — several hundred dollars — and bulky. Adolescents didn’t want to wear them. We also needed a technician to set up and distribute the device and retrieve the data.”
Intrigued by the possibility of leveraging a compact, beta-tested, app-enabled, $90 commercial device in his study, Dr. de Zambotti first had to determine whether the technology satisfied certain performance criteria for clinical research. He explained, “How do you know how it’s performing and whether it’s useful for more than amusement? These devices need to be evaluated through a rigorous process.”
Dr. de Zambotti and his colleagues were concerned by the lack of consistency and standardization in assessing device performance in clinical settings. “There were no standards. Some people would run correlations to prove the accuracy of a device; some people would run specific analytics. We tested based on what we needed to know to decide whether we could trust this device.”
This lack of coordination and uniformity often left researchers with potentially valuable device performance data that, unfortunately, couldn’t be directly compared to data assessed by different methods. The inefficiency was frustrating.
Standardizing wearable device evaluations
To address this, the SRI Sleep Lab engaged in a series of studies evaluating different devices and methods to assess sleep and sleep-related physiology and events, to promote informed discussion of how commercial sleep technology can be used in a research and clinical setting.
Of note, Dr. de Zambotti, in collaboration with Dr. Luca Menghini, developed a standardized testing and analysis pipeline that could accurately assess the capabilities of sleep-tracking technology and allow researchers to compare the performance of each technology against reference gold standard methods. “We wanted to have a structured way of doing these studies,” Dr. de Zambotti remarked.
The publication detailing the team’s assessment framework and open-source code has been well-received. Notably, this initiative required building a bridge between clinical care, research and commercial technology. “This evaluation pipeline is already being used by other experts in the field and by major players in the industry,” said Dr. Menghini.
Sleep Health to host rigorous performance evaluation of sleep technology
Dr. de Zambotti and Dr. Orfeu Buxton, Editor-in-Chief of Sleep Health, the journal of the National Sleep Foundation, agreed that regulation and standardization are needed in the sleep technology space to avoid the uninformed and indiscriminate use of unregulated technology in sleep research and clinical sleep medicine. To begin addressing this issue, they promoted an initiative to properly evaluate sleep technology, opening a new article type for performance evaluation studies. An interdisciplinary team of experts joined the initiative, including Drs. Meredith Wallace, Luca Menghini, Michael Grander, Susan Redline and Ying Zhang, forming the Sleep Health Expert Task Force on evaluating wearable technology. This task force published an article in the journal to explain the rationale, priorities and standards for studies assessing wearable technology.
“We believe that authors, reviewers and readers will all appreciate a systematic and rigorous approach to evaluating the performance of sleep health devices and related algorithms,” said Dr. Buxton. “These standards should provide an achievable and clear bar for acceptable quality that will contribute to high-quality publications, improved understanding of sleep and acceleration of the pace of innovation in Sleep Health.”
Dr. Buxton explained: “The standards we set in Sleep Health ensure consistency in how researchers report study outcomes. So bottom line, any clinical researcher who wants to use a device can search for a structured article that shows immediately, even in the abstract, the main metrics of the study and how the device performed. Anyone who utilizes the data will have a high bar in terms of the quality of the study.”
Dr. de Zambotti commented, “We set the benchmark to rigorously evaluate any new sleep technology from a research/clinical point of view, but we also welcome and encourage device engineers and manufacturers to properly test their technology. If companies want to keep pace with the benchmark, they should consider these guidelines, which would improve the value of their work and also increase transparency and quality for end-users.”
recommended analytical steps flow chart
The future of sleep technology
TheSleep Health Expert Task Force will continue to play an essential role in bridging the clinical research and commercial worlds.
“At Sleep Health, a multidisciplinary journal that explores sleep’s role in population health and elucidates the social science perspective on sleep and health,” commented Dr. Buxton. “We are excited about this initiative. With the rigor and standards proposed, new sleep technology may ultimately drive new and innovative sleep health research at the pace expected today.”
Dr. de Zambotti is a Principal Scientist at SRI International and co-founder and Chief Scientific Officer of Lisa Health. Dr. de Zambotti’s work in sleep technology is done under the SRI International Human Sleep Research Program led by Dr Fiona Baker.
Dr. Buxton is the Elizabeth Fenton Susman Professor of Biobehavioral Health, a Social Science Research Institute co-funded faculty and Director of the Sleep, Health & Society Collaboratory at Pennsylvania State University. He also serves as the Editor-in-Chief of Sleep Health.
