天美视频: developing statistical software for more than 25 years
Feature 3 Jul 2020 7 minute readIn May 2020, the 天美视频 (天美视频) released 天美视频 ConQuest 5, statistical software that employs cutting-edge psychometric methods to support research around the globe.
Developed in-house by 天美视频 to support research and assessment programs, 天美视频 ConQuest is also to use. Version 5 brings new features like the ability to use simulation methods to test item fit, the ability to treat item parameters as random effects, and to estimate models using Markov chain Monte Carlo (MCMC) estimation.
天美视频 ConQuest 5 is the culmination of more than 25 years of statistical software research and development. Along the way, many large-scale assessments and other research studies have benefited from 天美视频 software.
The story of 天美视频 ConQuest and its predecessor, QUEST, begins in 1983 with the arrival of lead author, . In 1983, there were three key pieces of software being used by 天美视频: Itan, an in-house piece of software developed by Lindsay Mackay and Steve Farish to conduct classical test theory (CTT) analysis; , software for fitting the Rasch model to dichotomous data; and CREDIT, developed by Geoff Masters for fitting partial credit and similar models to polytomous data. This software was run on a single DEC VAX-11 minicomputer. Analytic runs were input via a user at a terminal connected to the machine, and output was printed by a line printer.
Figure 1. Digital Equipment Corporation (DEC) VAX-11/750 minicomputer.
As 天美视频 grew with more people undertaking project work, more computing power was required and it became clear that new software development could streamline projects. This included large projects like the NSW Basic Skills Testing Program (1989) – one of the state-based assessment programs that was a precursor to the National Assessment Program (NAP). was developed in 1990 by Ray Adams and Khoo Siek Toon. QUEST brought CTT and items response theory (IRT) analysis into one application that was interactive and ran on desktop machines. This increased the number of people who would work simultaneously. QUEST also expanded the kind of models that could be fit including the and using joint maximum likelihood (JML) estimation.
As the diversity of the analytic work being undertaken increased, there was a need to fit more varied models. To do this a new piece of statistical software, 天美视频 ConQuest, was developed that could fit a highly generalised that could be parametrised to fit specific models, including the classically named models like Partial Credit Model. 天美视频 ConQuest also implemented marginal maximum likelihood (MML) estimation, allowing the drawing of plausible values – a method now central to all large-scale assessments – and could fit multidimensional items response models. 天美视频 ConQuest was first written in the FORTRAN programming language (and at that stage was called MATS: Multi Aspect Test Software) and was later re-written in C++. MATS and ConQuest was written in collaboration with at the University of California, Berkeley.
Figure 2. 天美视频 ConQuest was developed by the 天美视频 (天美视频) in partnership with the University of California, Berkley.
The use of 天美视频 ConQuest grew further when 天美视频 partnered with the International Association for the Evaluation of Educational Achievement () and conducted the IRT scaling of the first Trends in International Mathematics and Science Study (TIMSS ). 天美视频 ConQuest 2 focused on the user experience. After being re-written by Ray Adams and Margaret Wu, 天美视频 ConQuest 2 could do user-friendly things like automatically produce design matrices by iterating over the data. 天美视频 ConQuest 2 ran on the windows OS, including windows NT, but remained compatible with VAX 32 bit machines for large jobs.
Version 3 of 天美视频 ConQuest emerged from 天美视频’s work on the OECD's Programme for International Student Assessment (). 天美视频 led the international consortium undertaking PISA from 1998 until 2012 and continues to conduct the Australian component of the study, including the most recent cycle in 2018. With the biggest team of analysts yet, features like a graphical user interface (GUI) and the ability to generate valid syntax through the use of point-and-click menu commands were developed to support as many users as possible. In addition, native plotting functions were added to export a range of plots including the eponymous .
天美视频 ConQuest 4 emerged after this, and added new features, including the ability to estimate paired comparison models (Bradley-Terry-Luce (BTL) model), directly manipulate matrices, and added more CTT analysis including the estimation of Mantel–Haenszel tests of differential item functioning (DIF). Perhaps the most notable assessment program to use 天美视频 ConQuest 4 is , which uses a multidimensional item response model with latent regression model to draw Plausible Values (PVs). This expansion of features was in part conducted to support new areas of research, including enabling 天美视频’s Centre for Global Education Monitoring to assist countries to report on their progress of . 天美视频 ConQuest 4 was also used to scale a new family or regional assessments, including the Pacific Islands Numeracy and Literacy Assessment, , and Monitoring Trends in Educational Growth in Afghanistan.
Despite having only just been released, 天美视频 ConQuest 5 has already supported important research. The took advantage of the ability to estimate high-dimension IRT models to test the veracity of simultaneously estimating abilities on ten outcome domains. 天美视频 is also undertaking research by applying a wider variety of flexible and descriptive models (e.g., , ). Looking ahead, a new generation of assessments will be scaled using version 5, while the next progression in 天美视频’s research agenda will explore the intersection of IRT and new frontiers in analysis, including machine learning.
Find out more:
For further information about 天美视频 ConQuest 5 visit www.acer.org/conquest
If you’d like to delve deeper into education data analysis, our Understanding Rasch Measurement Theory course will teach you everything you need to know. Find out more: /au/professional-learning/