Abstract
Background: A well-performing health information system (HIS) provides timely, complete, accurate and easily retrievable data. However, HIS in low- and middle-income countries (LMICs), including Ethiopia, is highly complex and influenced by pressures from donors, politics and technical factors. Hence, these countries experience persistent challenges in producing quality data and difficulties using health management information system (HMIS) data from their HISs.
Objectives: This study aimed to evaluate how HMIS was perceived and utilised in HIV and AIDS monitoring in Ethiopia, and views regarding the influence of determinants on the use of HMIS.
Method: A qualitative evaluative case study using focus group discussions with data producers and users was conducted in selected health facility in Addis Ababa. A purposive critical case sampling was used to recruit participants.
Results: Key findings revealed that HIV and AIDS-specific indicators, information and communication technology (ICT) and other related resources were critical barriers to the successful use of the HMIS. Participants believed these technical issues impacted the quality of data adversely and, subsequently, the conversion of that data to information and using it to monitor the HIV and AIDS programme’s performance.
Conclusion: Technical factors affected all strategic decisions taken by the organisation. The health facilities did not process information as expected. However, staff performed the HMIS tasks with the tools available as they tried to make sense of data.
Contribution: This study contributed to the body of knowledge by identifying the technical factors on data quality and use of HMIS for HIV and AIDS monitoring.
Keywords: technical determinants; Health Information System; Health Management Information System; HIV and AIDS; monitoring and evaluation.
Introduction
A well-performing Health Information System (HIS) provides timely, complete, accurate and easily retrievable data. It also reflects the impact of guidelines and policies on the functioning of the health system and population health (Ministry of Health [Ethiopia] 2013). The HIS has multiple uses, such as enabling planning, supporting patient and health facility management and supporting global reporting. However, the information from such a system will be of little value if it is not available in formats that meet the needs of multiple users, namely, policymakers, planners, managers, healthcare providers, communities and individuals (WHO 2010).
According to the WHO (2016), country monitoring is the basis for regional and global monitoring of priority health issues. Monitoring and evaluation (M&E) is needed for reporting progress on health-related sustainable development goals (SDGs). Countries need functional surveillance mechanisms and accountability to ensure that priority health programmes are implemented as planned against stated objectives and desired results. The MEASURE Evaluation (2014) identified that strengthening was required for the HIV M&E system. Indicators that are too complex challenge the effectiveness of monitoring the HIV programme (Gloyd et al. 2016).
The Health Management Information System (HMIS) is a sub-system of a larger HIS, where other information systems interact. The HMIS, as a component of the HIS, brings together data from multiple data sources. These data constitute denominators used in calculating health indicators (Macfarlane & AbouZahr 2019). The HMIS has a critical role in the monitoring and evaluation of a health system, as it integrates data from all healthcare levels and provides information for the management of programmes, facilities, policies and resources (Qazi & Ali 2011). Ending the AIDS epidemic will stimulate wider international health and development efforts, indicating what can be realised through evidence-based action and multi-sectorial partnerships (UNAIDS 2014). HIV and AIDS care providers’ preferences and clinical workflows reflected through an integrated system intended for this purpose should provide evidence-based information. This will permit its usage in specific contexts, for specific users to solution of specific questions, thereby improving data torrents’ quality and efficiency and leading to upgraded patient care and health outcomes (Mgbere et al. 2018).
Globally, most of the eastern Mediterranean region countries have national HIS plans. Still, some plans have limited scope, particularly in data collection, analysis and capacity-building (WHO 2014). Health Information System in low- and middle-income countries (LMIC) are highly complex and influenced by pressures from donors, political and administrative. Therefore, accountability and good governance are imperative (Koumamba et al. 2021).
In south and eastern African region, Botswana struggled to create a functional HIS over the years. Substantial challenges happened in all aspects of the system undermining growth. Prolonged challenges such as the absence of central coordination, feeble leadership, poor policy and regulatory frameworks and insufficient resources were observed (Seitio-Kgokgwe et al. 2015). A study undertaken in Kenya indicates that most of the HISs collect aggregate data that do not allow patient-level quality of care evaluation (Gathara 2015).
Studies in Ethiopia reveal technical and behavioural factors related to reporting, such as poor documentation, poor competency (a skills gap) and lack of experience, resulting in poor data quality and below the national targets (Endriyas et al. 2019; Kebede et al. 2020). The focus of this study was on how the system was used to generate quality data. Therefore, this study reports on technical and organisational factors and their influence on the use of HMIS in HIV and AIDS monitoring and evaluation in Ethiopia.
The study recognised the need to gain an understanding of contextual factors and how they impact the health care interventions and adopted Performance of Routine Information System Management (PRISM) framework from Aqil, Lippeveld and Hozumi (2009), revised by MEASURE Evaluation (2019). The framework is guided by technical, organisational and behavioural factors to measure performance (MEASURE Evaluation 2019).
Research methods and design
Study setting
This study was conducted in two settings, namely, hospitals and health centres, referred to as health facilities, and governed by the Addis Ababa Regional Health Bureau. Key to the setting were three factors: HIV and AIDS service, HMIS implementation and accessibility. Three public hospitals and six health centres were chosen as the research sites. Hospitals were purposefully selected based on the burden of HIV cases, experience with HMIS implementation and HIV and AIDS management. Health centres were selected mainly because of their long-term experience with HIV management and varying levels of antiretroviral treatment (ART) of HIV and AIDS burden.
Design, sampling and study population
A qualitative evaluative case study was conducted to explore and describe how data generators and users utilised HMIS to generate relevant and reliable data for HIV and AIDS monitoring. This approach permitted the examination of data management processes, thus giving insights into the complexities of generating quality and useful data from HMIS. Therefore, HMIS use was the unit of analysis or the case. Health facilities represented a natural setting. Yin (2013) regards the goals of a case study as understanding a social phenomenon and real-life events such as organisational and managerial processes. Yazan (2015) contends that researchers have yet to have a full consensus on the design and implementation of the case study.
The population for this study was professionals operating at public health facilities, using the country’s HIS and the HMIS, such as those who mainly produce data (HMIS staff like data clerks, HIT and the HMIS/M&E focal person) and others who mainly use data for the HIV and AIDS programme monitoring and evaluation (clinicians and the facility management team). A non-probability purposive critical case sampling was used to recruit well-informed and experienced participants who worked on the HMIS and mainly gave service, support and decision-making in the HIV and AIDS programme.
The rationale for using critical case sampling was that a few informants could be the source of theoretical saturation, thus being decisive in explaining HMIS processes in these facilities (Patton 2015). Specific criteria were also set for recruitment, and only professionals working in the health facilities stipulated above were included in the research sample. The recruitment of research participants continued until data saturation. Therefore, the sample size was not predetermined.
Data collection instruments/tools
The interview guide was developed in English and then translated into Amharic. During translation, the researcher and language professional translated it independently and compared different versions for consistency and similarity of wording (Tsang, Royse & Terkawi 2017). The main focus of the questionnaire was on their perceptions of HMIS and how it was used to generate relevant and reliable data for HIV and AIDS monitoring and participants’ views regarding the influence of technical, organisational and behavioural factors on the use of HMIS.
Data collection procedure
Semi-structured focus group interviews were conducted with 64 participants over eight groups from 01 December 2018 to 31 March 2019. Focus groups were particularly suited for this study because they are flexible, inexpensive, elaborative and capable of producing rich data (Streubert, Speziale & Carpenter 2011). The participants who were willing to participate in the study signed informed consent forms, which outlined the purpose and processes of the study. The researcher outlined the rules for the discussions, such that only one person was allowed to talk at a time, mobile phones had to be put on silent and participants had to raise a hand when needing to speak.
Data collection was started only after ethical clearance was obtained from the university and the regional health bureau granted permission. A central grand tour question was planned to allow probing. Researchers considered participants’ comfort, ease of access to the venue and levels of interruption during interviews (Nyumba et al. 2018). The meetings were conducted in a suitable venue that provided privacy and was convenient for most participants.
Field notes were taken by an assistant who signed a confidentiality agreement with the researcher to ensure the ethical principles were observed. Based on the participation, each group interview took roughly 1–2 hours. Follow-up sessions were conducted until data saturation was achieved. Finally, eight Focus Group Discussion (FGD) were conducted with 64 participants from nine public health facilities. The interviews were audio recorded.
Data analysis
The audio-recorded data were transcribed verbatim and translated into English by the language professional and the researcher. The transcripts were numbered separately for each FGD. Data producers’ focus groups were numbered and labelled: FG1 data producers to FG4 data producers. Data users’ focus group transcripts were also numbered one to four and labelled: FG1 data users to FG4 data users. MMR verified the transcripts, and after preliminary analysis, saturation was determined, and data from all groups were merged.
Thematic analysis was carried out using ATLAS.ti. Version 8 software to develop broad topics condensed into descriptive codes. Coded sections were repeatedly read, cross-checked to derive meanings and aligned with the research objectives. The codes were categorised into themes, subthemes and categories that represented participants’ views on HMIS and technical, organisational and behavioural factors that influence the use of HMIS for HIV and AIDS monitoring and evaluation.
The analysis process was theory driven and pattern matching, following key concepts of the PRISM framework. The themes that emerged were cross-checked to derive meaning and were aligned with the research aim. The researchers reassessed and revised the codes, categories and themes until a consensus was reached.
Trustworthiness
The study used the Lincoln and Guba model (1985) criteria for trustworthiness: credibility, dependability, transferability and confirmability. There was prolonged engagement with participants during focus group interviews until data saturation occurred. They communicated in the language in which they felt comfortable. Dependability was ensured by double-checking data and summarising key findings for participants to confirm they were accurate reflections. A detailed description of the process used, such as the procedure and purpose of selecting participants, the method and length of the data collection procedure, data analysis and presentation, was produced to enhance transferability. MMR assessed the research methods and procedures. Participants’ verbatim statements enhanced dependability.
Ethical considerations
Full ethical approval was granted by the University of South Africa (UNISA) on 6 December 2017. The ethics approval number is HSHDC/792/2017. Permission to conduct the study was granted by the Addis Ababa Health Bureau. Written informed consent was obtained from all participants. Privacy and confidentiality were assured throughout the study; the rights of participants were protected and the integrity of the study was maintained.
Results
This study intended to examine how the HMIS generates relevant and reliable data and explore stakeholders’ views regarding technical and organisational factors that influence the use of HIV and AIDS health information.
Two themes with related categories emerged from the data: the organisation and structure of HMIS in Ethiopia and the use of routine health information for HIV and AIDS monitoring.
Biographical data of participants
Sixty-four participants, comprising 33 data producers and 31 data users, participated in eight focus group discussions. They were all recruited from nine health facilities. Nineteen were male participants, and 45 were female participants. The average age of participants was 32 years, and the level of education was higher in the data users’ group than in data producers; 33 participants had a diploma and 30 participants had a BSc degree. Finally, one participant had a master’s degree. The experience level in the HMIS ranged from 1 year to 10 years, respectively. More data producers received training on data quality checks, reporting systems and format use. Experience in using HMIS ranged from 2.03 years (medical directors) to 5.25 years (voluntary counselling and testing heads), respectively.
Theme 1: Organisation and structure of Health Management Information System in Ethiopia
Use of the Health Management Information System at the facility level
Data users understood that the HMIS is a management function used to provide users with reliable information to support decisions regarding the performance of the HIV and AIDS programme and to manage local factors that influence service utilisation. Data producers confirmed and said that routine facility reporting is critical for effective data management of antiretroviral therapy. Data flow within the facilities follows routine reporting processes, with each group having specific roles:
‘HMIS is implemented in every activity in the facility and includes vital registration, providing information on program related factors. A comprehensive software is being utilized and the recently implemented DHIS2 is a huge resource for data management.’ (FG4 Data users)
‘Mostly data clerks do data capturing and review for consistencies. We do paperwork and produce reports and there is an HMIS officer who also analyses the data.’ (FG2 Data users)
‘Voluntary counselling and testing registers and tally sheets are the main HMIS record format available to capture data. Summary sheets for daily work are used to provide monthly reports.’ (FG1 Data users)
Electronic and paper-based data reporting
Participants indicated that currently, the reporting uses both manual and electronic inputs. As mentioned earlier, some units, such as voluntary counselling and testing and prevention of mother-to-child transmission, still use paper-based registers and tally sheets. On the contrary, data clerks use SmartCare software. The HMIS focal persons enter all data in the DHIS2 (Web-District Health Information System). However, various challenges related to paper-based and parallel reporting were expressed.
Most believe manual reporting compromises quality, takes time and increases the risks of data entry errors. Some believe that paper-based reporting is still maintained because of software incompatibility with the reporting system. Others contended that the paper-based system could be a backup during power shortages or system failures:
‘We are manually working on some reports to HMIS. We use prevention of mother to child transmission register and tally sheet. The register is an MNCH integrated part that is used in antenatal care.’ (FG2 Data users)
‘We use SmartCare software and it has more detailed information.’ (FG4 Data Producers)
‘Doing manual may result in a lot of mistakes, quality will be compromised … paper-based reporting system takes time through each level of reporting and takes months to reach federal level.’ (FG1 Data Producers)
‘Paper-based system is needed for backup purposes, in case software gets corrupted. Hard copy is used when there is power cut.’ (FG3 Data producers)
Data-capturing system
Most participants indicated that using the correct data capture and recording tools is important for producing useful information in HIV and AIDS monitoring. Data from tallies and registers are collected and entered into the DHIS2 system in the HMIS room. Routine data are collected daily using paper-based recording (registers and tally sheets). They found it useful in providing information for planning, costing and research, which is also used to evaluate the performance of the HIV and AIDS programme:
‘We need to follow specific formats and registers when we capture data. HIV-related data are collected mostly in the antiretroviral therapy department at the data clerk level, then sent to the HMIS unit for compilation.’ (FG2 Data producers)
‘Register and tallies are collected from each department to enter [the data] in the DHIS2 [system] in the HMIS room.’ (FG3 Data producers)
‘We find it useful in that information from HMIS can be used for planning, budgeting, monitoring and research ….’ (FG4 Data users)
Challenges with reporting
Participants raised several issues related to the generation of quality reports. Some were related to health professionals, timeliness, reporting formats and misunderstandings between health professionals and the HMIS unit. They also emphasised the mismatch between the software and report format regarding data elements. In addition, they lamented the outdated software, which does not match the level of performance needed for accurate data analysis, especially medication records. Participants recommended appropriate software to replace manual reporting. Most participants mentioned that the prevention of mother-to-child transmission reporting format changes regularly, and it can cause confusion and errors among those who are not trained.
The following quotes support the findings:
‘This software reports mostly the type of drugs only. But viral load data is being done manually in formats … the printing from [Firms/Suppliers] takes from 3–6 months and that causes delays.’ (FG3 Data Producers)
‘Software has problems in wrongly reducing the number of reports. DHIS requires an update in data analysis by method, ages, and categories.’ (FG4 Data Producers)
‘Provision of medical service for communicable disease report is still in hard copy and has drawback … reporting electronically is better.’ (FG1 Data Producers)
‘In PMTCT [prevention of mother-to-child transmission] unit, the ministry supporting partner has their own reporting format. The data are not integrated and errors happen due to different types of reporting formats.’ (FG2 Data users)
Management of updates
Almost all data producers believed that SmartCare needs updating, and they also need regular training on updates. Data such as viral load, cohort chart and appointments were sometimes lost, influencing everything from client dosages to tracking lost to follow-up cases. The updates were perceived as necessary to enhance capacity and confidence in data management processes. However, they lamented frequent updates in routine data collection:
‘…It is good to provide training parallel with new updates. … all facility problems are related to updates and training. Specific training needs to be done.’ (FG1 Data Producers)
‘We need to be skilled in updates on indicator display, use of the system to track distribution and use of resources … refresher training needs to be planned in every year by the health bureau.’ (FG4 Data Producers)
‘The prevention of mother-to-child transmission format is frequently updated. For (a) new untrained person, it will be difficult to understand and work with.’ (FG2 Data users)
Usability of the system
Most users expressed that the HMIS is very useful for registrations using specific data elements such as age and disease codes. They found it helpful in providing information for planning, costing and research, which is also used to evaluate the performance of the HIV and AIDS programme. However, they suggested it can also be complicated, considering the number of inexperienced users. The main issues were the system’s complexity, related disease classification and too many indicators. These issues affect the efficiency of generating quality data and continuous use of information.
The following quotes support the findings:
‘HMIS supports the use of information for various reasons, such as having the right data based on accurate registrations. When reporting, disease classification and age data support the measurement of the disease’s prevalence. We find it useful in that information from HMIS can be used for planning, budgeting, monitoring and research ….’ (FG4 Data users)
‘After HMIS was started, new terminology was brought and resulted in many disease codes … Staffs have difficulty to comprehend all the disease codes. In the past, HMIS reporting format was simple and less paperwork.’ (FG3 Data users)
IT infrastructure and system
All groups emphasised that the Information Technology (IT) infrastructure needed to be designed to support data management processes, including the use of information for monitoring and evaluation. Currently, the system lacks functionalities, especially appointments, CD4 (Cluster of Differentiation 4) counts and viral load. Participants reported that some facilities had no computers, and staff had to report using manual formats. On the contrary, the data producers highlighted issues with connectivity.
The following verbatim quotes illustrate the findings:
‘The system does not have a functionality to save the appointment, CD4 [Cluster of Differentiation 4] count and viral load data. It does not produce the report accordingly. A child who is overweight will take [an] adult drug regimen, but the system is rigid and doesn’t recognize and record INH data.’ (FG3 Data users)
‘The other part, we don’t have the computer system to report …we requested one with the software application. HMIS staffs [sic] are expected to save reports on compact disk and flash if there is a lack of internet service to send the report … we have started to use a data display chart for monitoring the patient load ….’ (FG4 Data users)
Data quality assurance
All participants agreed that data generated from health facilities should be of high quality to use information effectively. They believed that electronic reporting and the HMIS help in maintaining quality data. They described various ways adopted to assess quality. Data clerks mainly collect, compile, analyse and validate data from different sources. The validation takes place after every shift and monthly and is usually done by two people to check the accuracy and consistency for control. They also use a lot of quality assurance sampling to select a sample of files to check the quality, using HIV and AIDS indicators.
The general agreement among them was that lot of quality assurance sampling ensures efficiency at different levels. However, they also indicated that quality is affected by the level of job satisfaction of the staff:
‘Electronic HMIS supports effective communication and quality. What is required is sending updated information, which is very important.’ (FG1 Data Producers)
‘The data will be checked according to the proportion of files by comparing the previous and current month [and if the] difference is huge or exaggerated, then indicators will be checked.’ (FG3 Data Producers)
‘We work with monthly data that come from the registers. We review to see how many clients are on treatment. We compile data at the end of our shift, identify improper recording and sometimes we find incompatible data from the recordings.’ (FG1 Data users)
‘There are things to be done to control data quality, like lot quality assurance sampling, by selecting 12 indicator data sets, we check data source documents such as tally and register for accuracy, that is, differences and similarities.’ (FG4 Data Producers)
Theme 2: Use of routine health information for HIV and AIDS monitoring
Ministry-level data-driven decisions
Participants acknowledged the usefulness of the HMIS in planning and mobilising resources at the sub-city health office, city health bureau and ministry of health levels. They described the data flow from facilities to the ministry level and recognised the significance of policy formulation based on data emanating from facilities.
The Ministry of Health uses the information to manage healthcare programmes and the overall health systems. Most participants were aware of their important role in improving population health outcomes and viewed the HMIS as a system that enabled them to make a contribution.
They elaborated as follows:
‘It is important to provide the ministry with useful data to enable them to set national targets and review the indicators. The city health bureau has its own targets in terms of resources allocation. We need to be mindful of different levels of information needs.’ (FG3 Data users)
‘Management team looks at data and take(s) actions to improve the service, for example, counselling, training, and staff.’ (FG4 Data users)
Case management
Participants confirmed that data have proven very vital in managing HIV cases. It gives much information that helps with managing drug therapy. Case managers provide an essential service. Lost cases are being followed up using specific algorithms, testing is monitored, and plans are developed based on the results.
The guidelines provide definitions regarding tracing, and clients are followed within prescribed days of missing treatment. The plans mostly involve the initiation of treatment and counselling:
‘In the antiretroviral therapy department, we have a case manager, who will follow the linked cases every week. On a monthly basis, they will call the lost cases, find out reasons for not following the treatment.’ (FG3 Data users)
‘The test and treat algorithm help the adherence team to decrease defaults and lost cases. We are able to see if we are reaching our targets.’ (FG3 Data users)
‘Patients are followed up based on the information received from other units. During reporting time, an HMIS team evaluates the facility performance in terms of targets. In January, HMIS was revised, the medical director or HMIS focal person usually does the assessments to identify gaps.’ (FG4 Data users)
Specific indicators for the HIV and AIDS programme
Participants recognised the significance of indicators in information use and linked them to the programme’s performance. Specific departments have unique indicators, for example, prevention of mother-to-child transmission, ART and voluntary counselling and testing.
The annual plans are used as benchmarks to monitor the level of achievement by performance review teams. This allows them to self-correct and learn from each other:
‘The indicators are number of pregnant women attending at least one ANC visit at a prevention of mother to child transmission site and acceptance of HIV testing, percentage of pregnant women testing positive for HIV, number of women testing positive who receive ART prophylaxis.’ (FG2 Data users)
‘Indicators are useful to measure implementation level of the HIV and AIDS programme and to determine resources required, for us, we report to the ministry who take actions on supply of resources.’ (FG4 Data users)
Performance monitoring
Participants confirmed the existence of some monitoring processes in the units, such as monthly review meetings. The composition of teams was similar across facilities. Departmental heads participate in teams chaired by the medical directors. Major items for review are testing, initiation levels and adherence to ART. The HMIS reports form the basis for discussions using various forms, indicators and logbooks. They all agreed that monitoring is a team effort that includes inputs from different departments:
‘In the prevention of mother to child transmission department, there is (a) cohort monitoring chart system for checking the number of clients, defaulters, fatality rate, and children diagnosed and lost … The reasons and/or causes are discussed in our monthly meetings.’ (FG2 Data users)
‘In my hospital, we hold these review meetings to make sure that we use data from HMIS effectively and use annual plans for monitoring. The HMIS focal person makes sure that HMIS reports are done timely and completely for each department to check their performance. Finance and HR heads are also represented.’ (FG4 Data users)
Specialised skills
All participants agreed that adequate knowledge and competence in the HMIS are critical. They indicated that one of the biggest challenges in healthcare is inadequate human resources, knowledge and lack of specialised software skills, which greatly impacts the quality of information using HMIS protocols:
‘We didn’t have a data manager for the past three years, who will do displays and analyse data for the facility … there is high staff turnover due to retirement and separation. Every year, staff rotates and no one is fixed in one place.’ (FG2 Data producers)
‘We have one HIT person for all departments in this facility and he has difficulty of supporting every room. 4–5 HIT persons are expected to be assigned. In the past, even one person was not assigned, which created problems in sending quality reports.’ (FG1 Data producers)
Culture of information use
Data users from various units expressed different perceptions on the promotion of the culture of information use. They described processes that are being followed at the facility level as an indication of some level of promoting a culture of information use. However, the majority believed that much more could be done by the management as, currently, there are no incentives for information use.
There are no guidelines, and training is inadequate for users to have a holistic understanding of the purpose of the HMIS:
‘The information I get from HMIS staff is beneficial; we work together to make sure that information is used, it is not perfect, but there is a steady progress from where we were.’ (FG2 Data users)
‘Data clerks are very helpful in providing information for the compilation of reports and tracking the HIV and AIDS program progress. However, it would be good to have them participate in our tracking activities as well.’ (FG3 Data users)
‘Information use is at the start level and I cannot say we are fully utilising data for decision-making. We do support staff. Recent implementation of DHIS2 may assist in getting data quickly to use for monitoring the program.’ (FG4 Data users)
Discussion
Gimbel et al. (2017) state that HMISs are planned to serve all users at each health system level with reliable information to support decisions. The study identified the significance of an in-depth understanding of the HMIS for programme follow-up and information. Participants indicated that the design of the HIS was critical in the successful implementation of the HMIS. The assumption was that the HMIS understanding is a foundation for its proper use and implementation through management inducing staff to embrace an innovative way of understanding (Mohamadali & Aziz 2017).
Findings revealed that Ethiopia uses a hybrid model comprising paper-based and electronic reporting. There were various views. Some expressed that the use of paper-based reporting did not support timely reporting, and others indicated that the paper trail provided a reliable backup, especially during power failures that seemed to be frequent. The system seemed hampered by reporting formats, the load and the tools used to capture data. The non-alignment between the software and paper-based generated data seemed to be an issue, especially in reporting certain data elements.
The software did not accurately capture the level of performance. Routinely collected health service data must be reported timely to higher levels, facilitating evidence-based decision-making at all levels of the health pyramid, especially at the point of collection (Cheburet & Odhiambo-Otieno 2016). MEASURE Evaluation (2012) indicates that the continuous usage of paper-based systems results in poor data quality regarding availability, reliability, completeness and reporting timeliness, affecting health service delivery. Mozambique also reported the use of paper and electronic systems reporting formats, the incompatibility of reporting formats and a shortage of skilled professionals as the main challenges that could be improved through standardisation of reporting (Hochgesang et al. 2016).
The involvement of external agencies also seemed to be an issue. For example, the Ministry of Health [Ethiopia] had its mandates for the data management process, but external agencies also bring software incompatible with the local IT infrastructure. Some weaknesses were highlighted in the systems updates. More specifically, SmartCare needed to be updated to avoid data loss. Participants believed that capacity building should be effected every time there is an update or change in reporting formats.
Participants perceived HMIS as very useful in generating data for managerial functions, including evaluating the performance of the HIV and AIDS programme at their level. Findings reveal that having a usable system can significantly help practitioners to implement the HMIS successfully, especially if they are knowledgeable about the disease classification system and indicators. In addition, this system needs to be compatible with the SmartCare software. Health Information System will face challenges in any country if fragmented and incompatible with other donor-funded systems. Umezuruike, Nwankwo and Kareyo (2017) confirm that a lack of standards for various software and hardware tools is a recipe for incompatibility among myriads of devices applicable to health information technology.
Therefore, management might need to review the current reporting formats and software. In developing countries, the HIS exists with many parallel central programmes, which leads to the disintegration of the HIS (Kumar et al. 2017).
ICT especially software, networking and other related resources were recognised as critical for efficient and effective monitoring of a health programme such as HIV and AIDS. Timely, complete and accurate information and a functional IT system are critical to producing quality data. Having reliable data on the health system’s performance supports effective strategies to implement and measure interventions (Kebede et al. 2020). Some functionalities such as clients’ appointments, viral load and CD4 count were missing, and manual reporting closed that gap. Braa et al. (2007) report that antiretroviral therapy reporting is complex, and paper-based systems make it even more difficult to handle.
The purpose of HMIS is to generate quality data that are useful. Timely, accurate, error-free reporting and uploading of data are considered as the quality benchmarks of an ideal HMIS (Samal & Dehury 2016). The significance of electronic reporting using systems such as DHIS2 was acknowledged. However, they also believed that users’ satisfaction needed to be taken into consideration. They used various mechanisms to validate data, including using Lot Quality Assurance system (LQAS); they also expressed various levels of confidence in conducting data quality checks. The study assumed that capacity building in data quality check would be beneficial.
Facilities need to have an adequate number of qualified employees and infrastructure to produce quality data (Moore et al. 2014). This will support the analysis and interpretation, and the use of information at the district and facility levels (Nicol et al. 2017). Cheburet and Odhiambo-Otieno (2016) argue that the LQAS has a great potential for monitoring health programmes to improve data quality.
Most participants acknowledged the importance of guidelines concerning understanding definitions of indicators to support the use of information. Key indicators generate data, such as the number of individuals tested and counselled for HIV and who received their test results. In the prevention of mother-to-child transmission department, the focus seemed to be on the number of HIV-infected women on HIV care and using a modern family planning method in addition to all the indicators used at the antiretroviral therapy department. The interconnected roles between data producers and users supported using data to manage cases.
This study contends that when data users and producers collaborate, they become more attentive to the data collection processes, methods, data sources and data quality. They can analyse and interpret data cooperatively to respond to programmatic questions (Nutley & Reynolds 2013). The antiretroviral therapy department generates data on key indicators, such as patients currently on ARVs, differentiated by the type of regimen, age category and sex. All these units used the HMIS to generate relevant data and plan activities accordingly. The HMIS team evaluated the performance according to targets.
Users require different information in varying details to track the HIV and AIDS programme performance. The performance review meetings reflected some commitment to data quality and developing a culture of information. Lippeveld and Hagan (2017) agree that there must be a plan at every health system level to create a culture of data-driven decisions. Different units tracked their performance routinely through data generated by the HMIS. This was achieved by comparing their plans with data.
Participants perceived these meetings as vital and their roles as significant for the continued use of the HMIS in monitoring the programme’s performance. Collaboration across various government agencies, specifically data management units, is essential to allow this sharing and use of data (Hosseinpoor et al. 2017).
The review meetings provided a platform to collaborate and monitor the programme performance using various tools. Silas (2017) found the importance of monitoring performance and following up on information usage through performance review meetings. Alshamari (2016) indicates that DHIS2 utilisation is critical for properly implementing data management and facilitating the use of information in HIV and AIDS monitoring. Findings show that HMIS users need specialised skills. Increasing the number of skilled health information management professionals in the HMIS strengthens the health system (Makinde et al. 2016).
Different users will require different types of information in varying details to support decision-making. The study found that at the facility level, users needed information to manage issues related to the programme and resources. Lloyd et al. (2011) found that the major challenge is to apportion resources for training and to advance the necessary skills regarding information in ideal formats.
The study revealed that the sustainability of the HMIS depends on all processes affecting organisational information culture. Harikumar (2012) agrees that advancing an information culture cultivates programme implementation.
Participants acknowledged some support in the HMIS task. However, they lamented their inadequacy. They would like incentives for information use and more adequate guidelines for users to understand the bigger set of HMIS fully. Wude et al. (2020) indicate that a perceived culture of information enhances routine health information utilisation. Ojo and Popoola (2015) also confirm that management and leadership advocacy and an organisation’s commitment to promoting a culture of information affect the success of the HIS.
Conclusion
Ethiopia uses a hybrid model of paper-based and electronic reporting. The use of paper-based reporting was deemed to be beneficial during power or connectivity failures. However, it appeared that paper-based reporting did not support the significant dimension of data quality: timeliness of reporting. For information to be effectively used in monitoring and evaluation, data must reach intended recipients on time. Health Management Information System was well perceived and understood. The main challenge appeared to be the inadequacy of information technology infrastructure. The study found that DHIS2 helped provide information for planning, costing, researching and monitoring the performance of the HIV and AIDS programme. However, this system needed to be compatible with the existing SmartCare software. Health Information System would face challenges in any country if fragmented and incompatible with other donor-funded systems. The findings leaned towards showing the local experience of the HMIS as is itself an innovation for the organisation implementing it. Such information reforms require well-structured change management.
The complexity of reporting formats, interoperability of software used, non-alignment with paper-based tools, the DHIS2, HIV and AIDS-specific indicators, ICT and other related resources were highlighted as critical barriers to the successful use of the HMIS. Participants believed these technical issues impacted the quality of data adversely and, subsequently, the conversion of that data to information and using it to monitor the programme’s performance. The actual impact on producing useful information still needs to be quantified. Developing countries, including Ethiopia, need to implement mechanisms to facilitate systems integration to achieve universal coverage and reach the SDGs, especially those related to providing infrastructure to support health systems. The importance of indicators to monitor the programme’s performance to show whether it is achieving its intended objectives is acknowledged. The common fact is the need for clear indicators with a simple disease classification system for improved data management and use in HIV and AIDS monitoring.
The study showed that the culture of health information was perceived as lacking. However, users did use the information for annual work plans and setting priorities. Therefore, the gaps identified need to be addressed through adequate training and capacity building, especially after HIS updates. In addition, users need to believe that the organisation promotes the culture of information use and creates an enabling environment that encourages staff engagement. The study notes that performance monitoring teams played a role in ensuring the quality of HIV and AIDS data and utilisation of HMIS data for decision-making. There was a clear interplay between technical and behavioural factors. Data showed that data analysis, interpretation and presentation need technical skills and knowledge. Therefore, there is a need to design appropriate skills development programmes to ensure staff is capacitated adequately.
In conclusion, the study acknowledges that the HMIS is a management tool that can be used at any health system level. Its use is influenced by the information needs of each stakeholder group, in this case, data producers and users. The study concludes that the interpretation and meaning users had of the HMIS in Ethiopia influenced how they used it and, subsequently, how they perceived support from the organisation. The importance of high-quality data in monitoring and evaluating the HIV and AIDS programme is acknowledged.
Acknowledgements
The authors acknowledge all participants, the health centre and hospital directors and translators for their valuable study support. They would also like to acknowledge Prof. Phil Botha’s contribution to editing the manuscript. They also acknowledge the publication of sections (61%) of this manuscript to a preprint server: https://www.researchsquare.com/article/rs-2756791/v1.
Competing interests
The authors declare that they have no financial or personal relationship(s) that may have inappropriately influenced them in writing this article.
Authors’ contributions
B.E.D. conceived and designed the study. B.E.D. conducted the fieldwork, collected and analysed the data and wrote the manuscript. M.M.R. advised and supervised the data analysis and reviewed the manuscript. B.E.D. and M.M.R. read and approved the final manuscript.
Funding information
This research received a specific grant from the University of South Africa for data collection and analysis.
Data availability
The data sets generated and/or analysed during this study are not publicly available but are available from the corresponding author, B.D., upon reasonable request.
Disclaimer
The views and opinions expressed in this article are those of the authors and are the product of professional research. It does not necessarily reflect the official policy or position of any affiliated institution, funder, agency, or that of the publisher. The authors are responsible for this article’s results, findings, and content.
References
Alshamari, M., 2016, ‘Usability factors assessment in health information system’, Intelligent Information Management 8(6), 170–180. https://doi.org/10.4236/iim.2016.86012
Aqil, A., Lippeveld, T. & Hozumi, D., 2009, ‘PRISM framework: A paradigm shift for designing, strengthening and evaluating routine health information systems’, Health Policy and Planning 24(3), 217–228. https://doi.org/10.1093/heapol/czp010
Braa, J., Hanseth, O., Heywood, A., Mohammed, W. & Shaw, V., 2007, ‘Developing health information systems in developing countries: The flexible standards strategy’, MIS Quarterly 31(2), 381–402. https://doi.org/10.2307/25148796
Cheburet, S.K. & Odhiambo-Otieno, G.W., 2016, ‘Process factors influencing data quality of routine health management information system: Case of Uasin Gishu County referral Hospital, Kenya’, International Research Journal of Public and Environmental Health 3(6), 132–139, viewed 30 November 2019, from http://journalissues.org/wp-content/uploads/2016/06/Cheburet-and-Odhiambo-Otieno.pdf.
Endriyas, M., Alano, A., Mekonnen, E., Ayele, S., Kelaye, T., Shiferaw, M. et al., 2019, ‘Understanding performance data: Health management information system data accuracy in Southern Nations Nationalities and People’s Region, Ethiopia’, BMC Health Services Research 19, 175. https://doi.org/10.1186/s12913-019-3991-7
Gathara, D.N., 2015, ‘The use of routine case record data to evaluate quality of inpatient hospital care for newborns and children in Kenya’, PhD thesis, University of Amsterdam, viewed 11 December 2021, from http://dare.uva.nl/personal/pure/en/publications/the-use-of-routine-case-record-data-to-evaluate-quality-of-inpatient-hospital-care-for-newborns-and-children-in-kenya(02bd5816-9b6a-4522-b894-e9b12b5ee58f).html.
Gimbel, S., Mwanza, M., Nisingizwe, M.P., Michel, C., Hirschhorn, L. & the A.H.I., PHIT Partnership Collaborative, 2017, ‘Improving data quality across 3 sub-Saharan African countries using the Consolidated Framework for Implementation Research (CFIR): Results from the African Health Initiative’, BMC Health Services Research 17, Article #828. https://doi.org/10.1186/s12913-017-2660-y
Gloyd, S., Wagenaar, B.H., Woelk, G.B. & Kalibala, S., 2016, ‘Opportunities and challenges in conducting secondary analysis of HIV programmes using data from routine health information systems and personal health information’, Journal of the International AIDS Society 19(5 Suppl 4), 20847. https://doi.org/10.7448/IAS.19.5.20847
Harikumar, S., 2012, ‘Evaluation of health management information systems – A study of HMIS in Kerala’, MPH dissertation, Achutha Menon Centre for Health Science Studies, Sree Chitra Tirunal Institute for Medical Sciences and Technology, Thiruvananthapuram, viewed 16 November 2020, from https://www.semanticscholar.org/paper/Evaluation-of-Health-Management-Information-Systems-Harikumar/6c29bbd2fba15668daa0617aa957fd7d161f08c1.
Hochgesang, M., Zamudio-Haas, S., Moran, L., Nhampossa, L., Packel, L., Leslie, H. et al., 2016, ‘Scaling-up health information systems to improve HIV treatment: An assessment of initial patient monitoring systems in Mozambique’, International Journal of Medical Informatics 97, 322–330. https://doi.org/10.1016/j.ijmedinf.2016.11.002
Hosseinpoor, A.R., Nambiar, D., Tawilah, J., Schlotheuber, A., Briot, B., Bateman, M. et al., 2018, ‘Capacity building for health inequality monitoring in Indonesia: Enhancing the equity orientation of country health information systems’, Global Health Action 11(22), 7–12. https://doi.org/10.1080/16549716.2017.1419739
Kebede, M., Adeba, E. & Chego, M., 2020, ‘Evaluation of quality and use of health management information system in primary health care units of east Wollega zone, Oromia regional state, Ethiopia’, BMC Medical Informatics and Decision Making 20(1), 1–12. https://doi.org/10.1186/s12911-020-01148-4
Koumamba, A.P., Bisvigou, U.J., Ngoungou, E.B. & Diallo, G., 2021, ‘Health information systems in developing countries: Case of African countries’, BMC Medical Informatics and Decision Making 21, Article #232. https://doi.org/10.1186/s12911-021-01597-5
Kumar, M., Mostafa, J. & Ramaswamy, R., 2017, ‘Federated health information architecture: Enabling healthcare providers and policymakers to use data for decision-making’, Health Information Management 47(2), 85–93. https://doi.org/10.1177/1833358317709704
Lincoln, Y.S. & Guba, E.G., 1985, Naturalistic inquiry, Sage, Beverly Hills, CA.
Lippeveld, T. & Hagan, O., 2017, ‘Routine health facility and community information systems: Creating an information use culture’, Global Health: Science and Practice 5(3), 338–340. https://doi.org/10.9745/GHSP-D-17-00319
Lloyd, S., Collie, J., McInnes, A., King, K., Lollback, A. & Garland, A., 2011, ‘Smart use of data, information and communication: The INFORM-ed Best Local Practice Project – Grafton Base Hospital’, Health Information Management 40(3), 26–30. https://doi.org/10.1177/183335831104000304
Macfarlane, S.B. & AbouZahr, C., 2019, The Palgrave handbook of global health data methods for policy and practice, Palgrave, No place.
Makinde, O.A., Mami, M.I., Oweghoro, B.M., Oyediran, K.A. & Mullen, S.M., 2016, ‘Investing in health information management: The right people, in the right place, at the right time’, Health Information Management Journal 45(2), 90–96. https://doi.org/10.1177/1833358316639447
MEASURE Evaluation, 2012, ‘Using data to improve service delivery: Training tool kit for pre-service nursing education’, MEASURE Evaluation, University of North Carolina, Chapel Hill, NC, viewed 15 October 2018, from https://measureevaluation.org/resources/publications/cl-19-021.html.
MEASURE Evaluation, 2014, A case study to measure national HIV monitoring and evaluation system strengthening Côte d’Ivoire, viewed 18 October 2018, from https://www.measureevaluation.org/resources/publications/sr-14-102.
MEASURE Evaluation, 2019, Performance of Routine Information System Management (PRISM) toolkit: PRISM tools, MEASURE Evaluation, University of North Carolina, Chapel Hill, NC, viewed 10 October 2021, from https://www.measureevaluation.org/resources/publications/tl-18-12/at_download/document.
Mgbere, O., Khuwaja, S., Vigil, K.J., Patel, S.M., Wang, J. & Essien, E.J., 2018, ‘Sources of clinical information used in HIV care and treatment: Are providers’ choices related to their demographic and practice characteristics?’, Health Informatics Journal 25(4), 1572–1587. https://doi.org/10.1177/1460458218788906
Ministry of Health [Ethiopia], 2013, Overview of the health sector, viewed 25 June 2018, from https://www.moh.gov.et/en/node/35?language_content_entity=en.
Mohamadali, N.A. & Aziz, N.F.A., 2017, ‘The technology factors as barriers for sustainable health information systems (HIS) – A review’, Procedia Computer Science 124, 370–378. https://doi.org/10.1016/j.procs.2017.12.167
Moore, M., Anthony, C.R., Lim, Y.-W., Jones, S.S., Overton, A. & Yoong, J.K., 2014, ‘The future of health care in the Kurdistan Region – Iraq: Toward an effective, high-quality system with an emphasis on primary care’, Rand Health Quarterly 4(2), 1. https://doi.org/10.7249/MG1148-1
Nicol, E., Bradshaw, D., Uwimana-Nicol, J. & Dudley, L., 2017, ‘Perceptions about data-informed decisions: An assessment of information-use in high HIV-prevalence settings in South Africa’, BMC Health Services Research 17(2), 765. https://doi.org/10.1186/s12913-017-2641-1
Nutley, T. & Reynolds, H.W., 2013, ‘Improving the use of health data for health system strengthening’, Global Health Action 6, 20001. https://doi.org/10.3402%2Fgha.v6i0.20001
Nyumba, T., Wilson, K., Derrick, C.J. & Mukherjee, N., 2018, ‘The use of focus group discussion methodology: Insights from two decades of application in conservation’, Methods in Ecology and Evolution 9(1), 20–32. https://doi.org/10.1111/2041-210X.12860
Ojo, A.I. & Popoola, S.O., 2015, ‘Some correlates of electronic health information management system success in Nigerian teaching hospitals’, Biomedical Informatics Insights 7, 1–9. https://doi.org/10.4137/BII.S20229
Patton, M.Q., 2015, Qualitative research and evaluation methods: Integrating theory and practice, 4th edn., Sage, Thousand Oaks, CA.
Qazi, M.S. & Ali, M., 2011, ‘Health management information system utilization in Pakistan: Challenges, pitfalls and the way forward’, Bioscience Trends 5(6), 245–254. https://doi.org/10.5582/bst.2011.v5.6.245
Samal, J. & Dehury, R.K., 2016, ‘Perspectives and challenges of HMIS officials in the implementation of Health Management Information System (HMIS) with reference to maternal health services in Assam’, Journal of Clinical and Diagnostic Research 10(6), 7–10. https://doi.org/10.7860/JCDR/2016/16921.7955
Seitio-Kgokgwe, O., Gauld, R.D., Hill, P.C. & Barnett, P., 2015, ‘Development of the national health information systems in Botswana: Pitfalls, prospects and lessons’, Online Journal of Public Health Informatics 7(2), e210. https://doi.org/10.5210/ojphi.v7i2.5630
Silas, N.K., 2017, ‘Factors influencing the performance of routine health information system: The case of Garissa subcounty, Kenya’, Thesis, Master of Arts in Project Planning and Management, University of Nairobi.
Streubert, H.J. & Carpenter, D.R., 2011, Qualitative research in nursing advancing the humanistic imperative, Wolters Kluwer, Philadelphia, PA.
Tsang, S., Royse, C.F. & Terkawi, A.S., 2017, ‘Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine’, Saudi Journal of Anaesthesia 11(Suppl. 1), S80–S89. https://doi.org/10.4103/sja.SJA_203_17
Umezuruike, C., Nwankwo, W. & Kareyo, M., 2017, ‘Implementation challenges of health management information systems in Uganda: A review’, Journal of Multidisciplinary Engineering Science and Technology 4(7), 7726–7731, viewed 12 March 2020, from http://www.jmest.org/wp-content/uploads/JMESTN42352312.pdf.
Joint United Nations Programme on HIV/AIDS, 2014, Guidance on capacity building for HIV monitoring and evaluation, UNAIDS, Geneva.
World Health Organization, 2010, Monitoring and evaluation of health system strengthening, viewed 22 May 2017, from https://iris.who.int/bitstream/handle/10665/258734/9789241564052-eng.pdf?sequence=1.
World Health Organization, 2014, Reinforcing health information systems, World Health Organization Regional Office for the Eastern Mediterranean, viewed 13 December 2021, from https://apps.who.int/iris/handle/10665/252220.
World Health Organization, 2016, World health statistics: Monitoring health for the SDGs, viewed 27 December 2021, from https://www.who.int/docs/default-source/gho-documents/world-health-statistic-reports/world-heatlth-statistics-2016.pdf.
Wude, H., Woldie, M., Melese, D., Lolaso, T. & Balcha, B., 2020, ‘Utilization of routine health information and associated factors among health workers in Hadiya Zone, Southern Ethiopia’, PLoS One 15(5), 1–11. https://doi.org/10.1371/journal.pone.0233092
Yazan, B., 2015, ‘Three approaches to case study methods in education: Yin, Merriam, and Stake’, The Qualitative Report 20, 134–152. https://doi.org/10.46743/2160-3715/2015.2102
Yin, R.K., 2013, ‘Validity and generalization in future case study evaluations’, Evaluation 19(3), 321–332. https://doi.org/10.1177/1356389013497081
| 
Hide
Show all
Email this article (Login required)
