Data quality is essential to ensure the rigor of comparative effectiveness research. Electronic health records capture real clinical encounters and document considerations in treatment decisions for the general patient population. For these reasons, electronic health record data are increasingly utilized to provide real-world evidence of clinical effectiveness. However, electronic health record systems are primarily designed for clinical care, not research. Data extracted from electronic health records are not readily available for research. Our case study aimed to compare the effectiveness of consensus treatment plans recommended by the Childhood Arthritis and Rheumatology Research Alliance for patients diagnosed with polyarticular-course juvenile idiopathic arthritis using electronic health record data. Here, we discuss challenges and share experiences encountered during the process of transforming electronic health record data into a research quality dataset.