Improving accuracy of HMIS District Coverage Estimates using Annealing Technique in Nepal

Department of Health Services (DoHS), Ministry of Health and Population (MoHP), Nepal; and Nepal Health Sector Programme 3/Monitoring, Evaluation and Operational Research (MEOR). 2019. Improving Accuracy of District Coverage Estimates of Health Management Information System by Annealing it with Lot Quality Assurance Sampling Community Survey Data in Nepal. Kathmandu, Nepal: DoHS, MoHP, Nepal.

This study was implemented under the aegis of the Integrated Health Information Management Section (IHMIS), Management Division, Department of Health Services (DoHS), Ministry of Health & population (MoHP). the UK Department for International Development Nepal Health Sector Program 3 (NHSP3), Monitoring, Evaluation & Operational Research (MEOR) Project in collaboration with Liverpool School of Tropical Medicine (LSTM) provided technical assistance to implement the study and was financed by DFID Nepal.

EXECUTIVE SUMMARY : The Health Management Information System (HMIS) provides a strong base for evidence informed decision making in the Nepal health system. The information from HMIS is used for planning, monitoring and evaluation of the health system at all levels, which includes monitoring of the Nepal Health Sector Programme III (NHSP3) and Nepal Health Sector Strategy (NHSS) 2015-2020. Nepal’s health system needs accurate, comprehensive and disaggregated data to measure its performance, and to identify disparities among social groups and geographic areas. Thus, HMIS is expected to provide valid data to inform strategic and policy level decisions. However, the data that comes from HMIS often has several reporting and recording issues from the point of data generation to the analysis and needs to be assessed from time to time to verify the quality of data. This study was designed to identify and test a mechanism to improve the accuracy of the HMIS. This was achieved by combining a probability sample (lot quality assurance sampling; LQAS) with HMIS indicators using an innovative statistical annealing technique (AT).

Eleven health coverage indicators relevant to the Safe Motherhood Programme: antenatal care (three indicators: four antenatal care check-ups, iron folic acid supplementation, and use of anti-helminthics), safe delivery (three indicators: institutional delivery, delivery by skilled birth attendant, and caesarean section) and newborn care and postnatal care (five indicators: chlorhexidine ointment application within one hour, breastfeeding within one hour, low birth weight, postnatal care within 24 hours, and postnatal care per protocol) were selected in consultation with Ministry of Health and Population (MoHP) stakeholders. Study team conducted a LQAS survey in Jumla, Arghakhanchi, and Mahottari districts; one each from the three ecological regions of the country. Study team gathered related information from all the municipalities and rural municipalities of the selected districts. A total of 551 women with a child 0-5 months age were interviewed from the 29 municipalities across these three districts. In addition to the primary data from the probability sample using LQAS, study team also collected the HMIS data for the same reference period which was used to calculate the hybrid prevalence of the selected indicators. Data was analysed using statistical AT in the Stata SE version 15 and R version 3.4.1.

For each municipality-level indicator coverage, AT constructs a hybrid estimate as a weighted linear combination of an HMIS estimate and a LQAS probability sample survey estimate, providing a standard error (SE) for both the HMIS and combined estimates. Study team reported two different calculations of the low birth weight indicator from the probability sample data, thus there are 12 combined indicators for each municipality – that is 12*29=348 in total. The results show that the combined estimate differs from the LQAS estimate by no more than 14.5%, while decreasing the SE by 0% to 5.8%. The majority (327/348=94%) of HMIS and combined estimates, however, can differ by up to 90%. In 5 instances the HMIS estimate is missing. In the remaining 16 instances the difference ranges between 93% and 2050%, and the HMIS coverage ranges between 101% and 2114%. This occurs for the breast-feeding indicator (Patrasi municipality in Jumla, nine municipalities in Mahottari district: Sonma, Bhangaha, Aurahi, Ramgopalpur, Samsi, Loharpatti, Manra Siswa, Pipra, Matihani), the Navi Malam indicator (4 municipalities in Mahottari district: Aurahi, Loharpatti, Pipra, Jaleswor), the iron folic acid indicator (Gaushala municipality in Mahottari district), and the anthelminthic indicator (Loharpatti municipality in Mahottari). Standard errors for the HMIS estimate are between 1 and v 78% times larger than those calculated for the combined estimate. The HMIS weighting factor ranges between 0.001 and 0.50 in Arghakanchi, between 0.009 and 0.99 in Jumla and between 0 and 0.996 in Mahottari.

In the HMIS data, it was found that 22.4% of indicators over-estimated and 32.5% under-estimated coverage. The results indicate that HMIS needs substantial strengthening to improve reliability to capture the data for the target population. Specific indicators can be targeted as well due to their substantial divergence from the probability sample. The study has shown that using AT with a probability sample could be a potential strategy for strengthening health information systems. The findings from this study are expected to provide ways to generate more accurate information to be used for decision making. Using AT with a probability sample could be one of the ways to generate more precise coverage estimates for appropriate decision making, and we may also have to improve the quality of recording and reporting.

EXECUTIVE SUMMARY 

Summary Results

• The majority (327/348=94%) of HMIS and combined estimates can differ by up to 90%.
• In 5 instances the HMIS estimate is missing.
• In the remaining 16 instances the difference ranges between 93% and 2050%, and in these cases HMIS coverage ranges between 101% and 2114%
• When the HMIS estimate is compared with the combined CI at the municipality level,
  22.4% of indicators over-estimated and 32.5% under-estimated the municipality-level coverages.
• Many of the HMIS indicators report a coverage of more than 100% at municipality level

Key Programmatic Implications

• HMIS coverage estimates could be further strengthened
• Selected municipalities could be targeted for improvement based on the current HMIS status
• The difference could be due to population, seeking services in facilities other than catchment area,
  –it could also be due to poor recording and reporting
• Using a small population based sample on a regular basis can identify weaknesses/inaccuracies of HMIS for the target population

DOWNLOAD: REPORT, PRESENTATION & RAW DATA: DOHS

Additional information about the study may be obtained from the IHIMS, Management Division, DoHS, MoHP, Teku Kathmandu; Telephone: +977-1-4257765; internet: http://www.mddohs.gov.np; or NHSP3/MEOR project, Lalitpur-10, Jwagal, Lalitpur, 44700, Nepal; telephone: +977-1-5544822; email: meor@abtassoc.com.au.

SOURCE OF INFORMATION:Department of Health Services (DoHS)(Improving accuracy of HMIS District Coverage Estimates using Annealing Technique in Nepal)

Deputy Director – SAARC Tuberculosis and HIV/AIDS Centre

Call for applications TDR Postgraduate Scholarship in Implementation Research

CAS-TWAS President’s PhD Fellowship Programme

Admission open! MPH Program, Faculty of Health Sciences, Pokhara University