Researchers propose platform for evaluating disease-forecasting AI methods
source link: https://venturebeat.com/2021/02/11/researchers-propose-platform-for-evaluating-ai-disease-forecasting-methods/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Data: Meet ad creative
From TikTok to Instagram, Facebook to YouTube, and more, learn how data is key to ensuring ad creative will actually perform on every platform.
Register NowSince the start of the pandemic, there’s been an influx of papers on epidemic forecasting. Indeed, as of February, a search for “COVID forecasting” on Google Scholar yields over 14,000 results. But while many researchers compare their approaches against traditional modeling strategies, forecasts are highly sensitive to the implementation, which requires a well-defined benchmark.
Researchers at the University of Southern California propose a benchmark in EpiBench, which focuses on retrospective forecasting — i.e., forecasting when ground truth is already available. While the platform is in the preliminary stages, the researchers believe that it could benefit real-time forecasting efforts by the U.S. Centers for Disease Control and other agencies around the world that drive governments’ responses.
Modeling is only a part of the forecasting process and can’t be considered a method to compare against. It describes only the epidemic portion rather than data preprocessing and model architecture development. Therefore, the researchers say, without a platform like EpiBench, claiming a new AI forecasting approach performs better than another isn’t possible.
As a proof of concept, the researchers developed a prototype of EpiBench called “COVID-19 forecast-bench,” which keeps a daily record of COVID-19 cases and deaths as reported by Johns Hopkins University. Researchers are required to provide details of their methodologies regarding data preprocessing, modeling techniques, and learning strategy, and these methodologies are evaluated for 1-, 2-, 3-, and 4-week-ahead forecasts.
In an experiment, the researchers compared 3 AI and machine learning forecasting methods and 30 methodologies pulled from published research using EpiBench. They found that while many of the forecasts reportedly used the same model (SEIR), they predicted “drastically” different outcomes. Moreover, two methodologies identical except that one smoothed data over 14 days versus the other’s 7 days varied “significantly” in their performance, suggesting that data preprocessing played a nontrivial role.
The researchers believe that EpiBench will help the research community in making decisions by analyzing methodologies based on how they arrive at their forecasts. Expanding from the prototype, the researchers hope to launch a website, a GitHub repository, and a Slack channel for discussion. If all goes according to plan, developers will be able to view evaluations for submitted models and upload sets of forecasts for benchmarking, as well as upload code to reproduce results and run code on more datasets.
“EpiBench can help us identify which decisions (data preprocessing, modeling choice, learning strategy, hyper-parameter tuning, etc.) in the forecasting approach are critical in epidemic forecasting and help direct the research accordingly,” the researchers wrote in a paper describing EpiBench. “In the future, we wish to expand EpiBench to various epidemic forecasting tasks. The platform will be available to the AI and machine learning researchers and epidemiologists.”
VentureBeat
VentureBeat's mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact.
Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:
- up-to-date information on the subjects of interest to you
- our newsletters
- gated thought-leader content and discounted access to our prized events, such as Transform
- networking features, and more
Recommend
-
7
News Researchers Propose A New “Unhackable” Transistor one day ago by Jake Hertz Protecting IP is a serious concern in the semiconductor ind...
-
4
Transform 2021 Join us for the world’s leading event about accelerating enterprise transformation with AI and Data, for enterprise technology decision-makers, presented by the #1 publisher in AI...
-
12
3:51 PM / by Tony TranResearchers Propose Sprinkling Hundreds of Chips Into Human BrainThey believe it could help treat epilepsy and Parkinson’s. ...
-
8
March 22, 2022
-
7
News The Block DOE-backed researchers propose a solution to a stablecoin paradox: an electricity peg
-
8
August 24, 2022 ...
-
5
Researchers propose infrared popcorn popper Infrared requires less energy than microwaves or air poppers By
-
6
Researchers propose a fourth traffic light for autonomous vehicles Light would instruct humans to do whatever the robocars do By
-
12
Researchers propose new host configuration for lithium anodes using nuclear track method by LIU Jia,...
-
11
Scientists propose new rapid analysis methods for reliability of standby redundant systems by Zhang Nannan,...
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK