Developing and Testing an Automated Qualitative Assistant (AQUA) to Support Qualitative Analysis

Developing and Testing an Automated Qualitative Assistant (AQUA) to Support Qualitative Analysis

Published: Nov 29, 2021
Publisher: Family Medicine and Community Health, vol. 9, supplement 1

Robert P. Lennon

Robbie Fraleigh

Lauren J. Van Scoy

Bethany L. Snyder

Erin L. Miller

William A. Calo

Aleksandra E. Zgierska

Christopher Griffin

Qualitative research remains underused, in part due to the time and cost of annotating qualitative data (coding). Artificial intelligence (AI) has been suggested as a means to reduce those burdens, and has been used in exploratory studies to reduce the burden of coding. However, methods to date use AI analytical techniques that lack transparency, potentially limiting acceptance of results. We developed an automated qualitative assistant (AQUA) using a semiclassical approach, replacing Latent Semantic Indexing/Latent Dirichlet Allocation with a more transparent graph-theoretic topic extraction and clustering method. Applied to a large dataset of free-text survey responses, AQUA generated unsupervised topic categories and circle hierarchical representations of free-text responses, enabling rapid interpretation of data. When tasked with coding a subset of free-text data into user-defined qualitative categories, AQUA demonstrated intercoder reliability in several multicategory combinations with a Cohen’s kappa comparable to human coders (0.62–0.72), enabling researchers to automate coding on those categories for the entire dataset. The aim of this manuscript is to describe pertinent components of best practices of AI/machine learning (ML)-assisted qualitative methods, illustrating how primary care researchers may use AQUA to rapidly and accurately code large text datasets. The contribution of this article is providing guidance that should increase AI/ML transparency and reproducibility.

How do you apply evidence?

Take our quick four-question survey to help us curate evidence and insights that serve you.

Take our survey