Level of Agreement on HIT Adoption and Use
Adoption of health information technology (HIT) is often assessed in surveys of organizations. The validity of data from such surveys for ambulatory clinics has not been evaluated. Researchers from RAND (Robert Rudin, Shira Fischer, Paul Shekelle, M Susan Ridgely, Cheryl Damberg) and Penn State University (Yunfeng Shi, Alejandro Amill-Rosario, Bethany Shaw) compared level of agreement between one ambulatory statewide survey and two other data sources. The principal finding of this study is that quantitative and qualitative data provide some evidence supporting the validity of some of the survey items included in two enterprise surveys related to the presence or absence of HIT functions in ambulatory clinics operated by medical groups, however qualitative data raised serious questions about the validity of other survey items.
Their findings appeared online in JAMIA Open on March 7, 2019
As use of electronic health records (EHR) and other forms of HIT increases, there has been continued interest in measuring the adoption and use of specific HIT capabilities, and examining the relationships between HIT and outcomes such as reductions in safety events, reductions in costs, and improvements in quality of care. Data to support these measurement efforts are typically collected through enterprise surveys of health systems and care.
Previous studies have questioned the validity of HIT utilization surveys for hospitals, but a more recent study of a range of HIT functionalities in hospitals comparing survey responses to Medicare Meaningful Use reports as a gold standard, found a more favorable assessment. Similar validation assessments have not been conducted on enterprise surveys used to acquire data from ambulatory providers, so to address the gap, the authors assessed one ambulatory HIT survey’s level of agreement with a second survey encompassing similar items that cover a range of functionalities, and with survey respondents’ stated perspectives of the survey items when interviewed.
Methods of study
The study relied on 2016 data from two surveys of ambulatory providers in Minnesota—the Healthcare Information and Management Systems Society (HIMSS) survey and the Minnesota HIT Ambulatory Clinic Survey (MN HIT)—and primary data collected through qualitative interviews with survey respondents. The authors conducted a concurrent triangulation mixed-methods assessment of the MN HIT survey by assessing level of agreement between it and HIMSS, and a thematic analysis of interview data to assess the respondent’s understanding of what was being asked and their approach to responding.
Conclusion of findings
Analyses of HIT survey findings that inform public policy should take into consideration the validity of the data used to develop estimates of prevalence and use. Responses to survey items that inquire about more widely adopted and well-established functionalities are more likely to be valid. Adequately capturing data on less established functionalities, degree of usage, variation across clinics within a health system, and barriers to implementation will likely require a more in-depth understanding of provider experience with HIT and a better understanding of how respondents to surveys arrive at their answers.
This work was supported through the RAND Center of Excellence on Health System Performance, which is funded through a cooperative agreement (1U19HS024067-01) between the RAND Corporation and the Agency for Healthcare Research and Quality.
For more information contact Robert S. Rudin, PhD - rrudin@rand.org
