- Online teaching and learning has become an increasingly important aspect of the educational mission of universities. In person, teachers have time-tested tools for assessing student ability, including a wealth of verbal and nonverbal communication. The online format provides a wealth of data, and promises—but may not yet deliver—useful tools for this sort of just-in-time assessment. Publisher homework websites and quizzes inside a learning management system like Canvas can theoretically provide up-to-the-minute performance data including scores, use of help features, access of resources, and more. Our setting (teaching introductory online quantitative classes in the College of Business at a large research university) makes these innovations particularly appealing. Publishers have correctly identified our interest in “knowing” our students better via their online performance, but we have not yet seen an off-the-shelf solution that gets at our need: the ability to quickly and effectively react to student data in real time. In this paper, we discuss a portion of our research conducted in an online quantitative methods class, a 200-level undergraduate course in the College of Business. This research included constructing a Q Matrix as part of a Cognitive Diagnosis Model for our quantitative methods class. A Q Matrix is a mathematical tool that creates a linkage between underlying concept development and students’ performance on test items. In order to create assessments of learning which are based on student responses to questions, we must first investigate whether these questions are actually testing the foundational concepts we wish to evaluate. The Q Matrix offers a more holistic view of student achievement, and allows better insight (in terms of specificity regarding particular skills and concepts) into student growth and accomplishment than traditional item response methods. Q Matrix analysis requires serious attention to questions about how students are learning material and what underlying skills are being assessed by test questions. The research is based on two main theoretical foundations: Item Response Theory and Cognitive Diagnosis Models.