Journal Article > Pre-PrintFull Text
bioRxiv. 18 August 2017; DOI:10.1101/177451
Funk S, Camacho A, Kucharski AJ, Lowe R, Eggo RM, et al.
bioRxiv. 18 August 2017; DOI:10.1101/177451
Real-time forecasts based on mathematical models can inform critical decision-making during infectious disease outbreaks. Yet, epidemic forecasts are rarely evaluated during or after the event, and there is little guidance on the best metrics for assessment. Here, we propose an evaluation approach that disentangles different components of forecasting ability using metrics that separately assess the calibration, sharpness and unbiasedness of forecasts. This makes it possible to assess not just how close a forecast was to reality but also how well uncertainty has been quantified. We used this approach to analyse the performance of weekly forecasts we generated in real time in Western Area, Sierra Leone, during the 2013–16 Ebola epidemic in West Africa. We investigated a range of forecast model variants based on the model fits generated at the time with a semi-mechanistic model, and found that good probabilistic calibration was achievable at short time horizons of one or two weeks ahead but models were increasingly inaccurate at longer forecasting horizons. This suggests that forecasts may have been of good enough quality to inform decision making requiring predictions a few weeks ahead of time but not longer, reflecting the high level of uncertainty in the processes driving the trajectory of the epidemic. Comparing forecasts based on the semi-mechanistic model to simpler null models showed that the best semi-mechanistic model variant performed better than the null models with respect to probabilistic calibration, and that this would have been identified from the earliest stages of the outbreak. As forecasts become a routine part of the toolkit in public health, standards for evaluation of performance will be important for assessing quality and improving credibility of mathematical models, and for elucidating difficulties and trade-offs when aiming to make the most useful and reliable forecasts.
Journal Article > Pre-PrintFull Text
bioRxiv. 1 February 2019; DOI:10.1101/533851
Saran K, Masini T, Chikwanha I, Paton G, Scourse R, et al.
bioRxiv. 1 February 2019; DOI:10.1101/533851
BACKGROUND
Tuberculosis (TB) poses a global health crisis requiring robust international and country-level action. Adopting and implementing TB policies from the World Health Organization (WHO) is essential to meeting global targets for reducing TB burden. However, many high TB burden countries lag in implementing WHO recommendations. Assessing the progress of implementation at national level can identify key gaps that must be addressed to expand and improve TB care.
METHODS
In 2016/2017, Médecins Sans Frontières and the Stop TB Partnership conducted a survey on adoption and implementation of 47 WHO TB policies in the national TB programs of 29 countries. Here we analyze a subset of 23 key policies in diagnosis, models of care, treatment, prevention, and drug regulation to provide a snapshot of national TB policy adoption and implementation. We examine progress since an analogous 2015 survey of 23 of the same countries.
RESULTS
At the time of the survey, many countries had not yet aligned their national guidelines with all WHO recommendations, although some progress was seen since 2015. For diagnosis, about half of surveyed countries had adopted the WHO-recommended initial rapid test (Xpert MTB/RIF). A majority of countries had adopted decentralized models of care, although one-third of them still required hospitalization for drug-resistant (DR-)TB. Recommended use of the newer drugs bedaquiline (registered in only 6 high-burden TB countries) and delamanid (not registered in any high-burden country) was adopted by 23 and 18 countries, respectively, but short-course (9-month) and newer pediatric regimens by only 13 and 14 countries, respectively. Guidelines in all countries included preventive treatment of latent TB infection for child TB contacts and people living with HIV/AIDS, but only four extended this to adult contacts.
CONCLUSION
To reach global TB targets, greater political will is needed to rapidly adopt and implement internationally recognized care guidelines.
Tuberculosis (TB) poses a global health crisis requiring robust international and country-level action. Adopting and implementing TB policies from the World Health Organization (WHO) is essential to meeting global targets for reducing TB burden. However, many high TB burden countries lag in implementing WHO recommendations. Assessing the progress of implementation at national level can identify key gaps that must be addressed to expand and improve TB care.
METHODS
In 2016/2017, Médecins Sans Frontières and the Stop TB Partnership conducted a survey on adoption and implementation of 47 WHO TB policies in the national TB programs of 29 countries. Here we analyze a subset of 23 key policies in diagnosis, models of care, treatment, prevention, and drug regulation to provide a snapshot of national TB policy adoption and implementation. We examine progress since an analogous 2015 survey of 23 of the same countries.
RESULTS
At the time of the survey, many countries had not yet aligned their national guidelines with all WHO recommendations, although some progress was seen since 2015. For diagnosis, about half of surveyed countries had adopted the WHO-recommended initial rapid test (Xpert MTB/RIF). A majority of countries had adopted decentralized models of care, although one-third of them still required hospitalization for drug-resistant (DR-)TB. Recommended use of the newer drugs bedaquiline (registered in only 6 high-burden TB countries) and delamanid (not registered in any high-burden country) was adopted by 23 and 18 countries, respectively, but short-course (9-month) and newer pediatric regimens by only 13 and 14 countries, respectively. Guidelines in all countries included preventive treatment of latent TB infection for child TB contacts and people living with HIV/AIDS, but only four extended this to adult contacts.
CONCLUSION
To reach global TB targets, greater political will is needed to rapidly adopt and implement internationally recognized care guidelines.
Journal Article > Pre-PrintFull Text
bioRxiv. 17 February 2021; DOI:10.1101/2021.02.17.431606
Jarvis CI, Gimma A, Finger F, Morris TP, Thompson JA, et al.
bioRxiv. 17 February 2021; DOI:10.1101/2021.02.17.431606
The fraction of cases reported, known as ‘reporting’, is a key performance indicator in an outbreak response, and an essential factor to consider when modelling epidemics and assessing their impact on populations. Unfortunately, its estimation is inherently difficult, as it relates to the part of an epidemic which is, by definition, not observed.
We introduce a simple statistical method for estimating reporting, initially developed for the response to Ebola in Eastern Democratic Republic of the Congo (DRC), 2018-2020. This approach uses transmission chain data typically gathered through case investigation and contact tracing, and uses the proportion of investigated cases with a known, reported infector as a proxy for reporting. Using simulated epidemics, we study how this method performs for different outbreak sizes and reporting levels. Results suggest that our method has low bias, reasonable precision, and despite sub-optimal coverage, usually provides estimates within close range (5-10%) of the true value.
Being fast and simple, this method could be useful for estimating reporting in real-time in settings where person-to-person transmission is the main driver of the epidemic, and where case investigation is routinely performed as part of surveillance and contact tracing activities.
We introduce a simple statistical method for estimating reporting, initially developed for the response to Ebola in Eastern Democratic Republic of the Congo (DRC), 2018-2020. This approach uses transmission chain data typically gathered through case investigation and contact tracing, and uses the proportion of investigated cases with a known, reported infector as a proxy for reporting. Using simulated epidemics, we study how this method performs for different outbreak sizes and reporting levels. Results suggest that our method has low bias, reasonable precision, and despite sub-optimal coverage, usually provides estimates within close range (5-10%) of the true value.
Being fast and simple, this method could be useful for estimating reporting in real-time in settings where person-to-person transmission is the main driver of the epidemic, and where case investigation is routinely performed as part of surveillance and contact tracing activities.