The algorithm is subject to change without notice.
These products sometimes had data problems, and the Apple Watch’s team wanted to test their severity before starting. So they checked the heart rate data exported by Hassan Dawood, a researcher at Brigham and Women’s Hospital, and his Daily data export. Two heart rate variability data: one on September 5, 2020, and the second on April 15, 2021.
For experimentation, they analyzed the data collected from early December 2018 to September 2020 during the same period. Since the two exported data sets contain data for the same period, the data in the two data sets should theoretically be the same.
Onnela said he expected some differences. Apple Watch’s “black box” of portable algorithms is an ongoing challenge for researchers. Unlike the raw data collected by display devices, products usually only allow researchers to export information after analysis and filtering using specific algorithms. The company will periodically change the algorithm without warning, so the September 2020 export may contain data parsed using a different algorithm than the April 2021 export.
Apple Watch’s Constant change of algorithms
“The amazing thing is that they are so different,” he said. “This is probably the most obvious example of this phenomenon I have ever seen,” he added. He published the data on his blog last week. Apple did not respond to a request for comment. “It’s amazing to see such a noticeable difference,” said Olivia Walch, a sleep researcher who studies data from the university’s portable devices and apps.
Vouch has long urged researchers to use raw data, which is data obtained directly from device sensors, rather than data leaked through its software. “This is confirmed because I entered my small soapbox with raw data, and I am glad to have a concrete example that is really important,” he said. Walch says that constantly changing algorithms make it almost unaffordable to use commercial handheld devices for sleep research.
Sleep research is expensive. Everyone runs different versions of the software and then compares them? Companies are motivated to change their algorithms to improve their products. “They don’t have much motivation to tell us how things have changed,” he said. This is a question that needs to be investigated. Onnela compares this to monitoring body weight.
Apple Watch may be riskier to rely on
“If I want to use the scale every week, I will use the same scale every time,” he said. If he adjusts these scales without his knowledge, daily weight changes will be unreliable. Only occasionally interested in tracking your health, this may be fine. The difference is not important. But consistency is important in research. “This is a problem,” he said. For example, someone can use wearable devices to conduct some research and draw conclusions. It’s about how people’s sleep patterns change due to changes in the surrounding environment.
But this conclusion only applies to this specific version of the portable software. “If you just use a different model, you may get completely different results,” Volch said. Dawood’s Apple Watch data is not a research result. But an informal example shows the importance of being cautious about commercial devices that don’t allow access to raw data. Onnela said this is enough for his team, too.
There’s a plan to use equipment in the studio. Wearable devices should only be used when raw data is available, or researchers know when the algorithm changes. In some cases, data from portable devices are still useful.
Image courtesy of Marques Brownlee/YouTube