Bayesian Scoring Theoretical Foundations

Overview

Bayesian Scoring provides the means to increase the accuracy of a student ability estimate (Lexile or Quantile measure) using both historical and current test experience information. Like all statistical results, a measure of uncertainty accompanies the estimate. As additional information is added into the estimate the uncertainty associated with the estimate is also updated. Both prior information and current test results are used to build likelihood probability models in determining a student’s Lexile/Quantile measure. These models also make use of the uncertainty associated with these estimates. Uncertainty arises because test performance may be influenced by a variety of factors that are independent of a student’s ability, such as a test environment filled with distractions, family discord, or whether the student had a good night’s sleep. Even if we are certain about the test performance once the results are in, we remain uncertain about our estimate of the underlying ability that produced the performance.

However, since it is also likely that some amount of time has passed since the last assessment experience, an allowance must be made for an uncertain amount of growth in ability that likely occurred between when the last test was administered and the most recent assessment experience. This allowance is accomplished by means of a growth model that estimates both the growth in ability and the increase in uncertainty as a function of the elapsed time between the assessments.

Basic Principles

Bayesian methodology provides a paradigm for combining prior information with current data, both subject to uncertainty, to come up with an estimate of current status, which is again subject to uncertainty. Uncertainty is modeled mathematically using probability.

When a student is ready to encounter an assessment instrument, i.e. test, the prior information could come from grade level or from previous assessments. The current data in this context is the performance on the test, which can be summarized as the number of items answered correctly out of the total number of items attempted.

Both prior information and current data are represented via probability models reflecting uncertainty. The need for incorporating uncertainty when modeling prior information is intuitively clear. The need for incorporating uncertainty when modeling test performance is, perhaps, less intuitive. Once the test has been taken and scored, and assuming that no scoring errors were made, the performance, (i.e. raw score), is known with certainty. Uncertainty arises because test performance is associated with but not determined by the ability of the student and it is that ability, rather than the test performance per se, that we are endeavoring to measure. Thus, although we are certain about the test performance once the results are in, we remain uncertain about the ability that produced the performance.

The uncertainty associated with prior knowledge is modeled by a probability distribution for the ability parameter. This distribution is called the prior distribution and it is usually represented by a probability density function, e.g., the normal bell-shaped curve. The uncertainty arising from current data is modeled by a probability function for the data when the ability parameter is held fixed. When roles are reversed so that the data are held fixed and the ability parameter is allowed to vary, this function is called the likelihood function. In the Bayesian paradigm, the posterior probability density for the ability parameter is proportional to the product of the prior density and the likelihood, and this posterior density is used to obtain the new ability estimate along with its uncertainty.

Implementation via the Rasch model

The Likelihood. As a statistical model, the Rasch model involves ability parameters for students and difficulty parameters for items. Both sets of parameters can be estimated from current data, but our interest here is focused on the abilities, so we will assume the difficulties are given, either from theory or past data. For the data record from an individual student, consisting of a pattern of right and wrong answers for a set of items with given difficulties, the likelihood function for that student’s ability is the product of factors, one per item, where the factors are determined using the Rasch model.

The prior. In the Bayesian paradigm, prior information and its associated uncertainty is represented by a probability distribution, called the prior distribution. A formula for the density function provides a complete and precise specification of the prior distribution. However, it is more common for the prior information to be specified by an estimated ability and a measure of the uncertainty associated with the ability estimate.

Modeling Growth and its impact on the prior. When no prior assessments are available for a student, the prior distribution can be inferred from the student's grade and/or age. Once a posterior has been obtained from current data, that posterior can serve as the prior for an immediate repeat assessment. If a substantial amount of time has passed since the last assessment, however, then allowance should be made for an uncertain amount of growth in ability since the last assessment. This allowance is accomplished by means of a growth model, which estimates as a function of elapsed time both the growth in ability and the augmentation in uncertainty.

Calculating Uncertainty for an Individual. We assume that maximum uncertainty is 1.1056 (within grade standard deviation in logits from prior research). The assumption is that after three years we are again at maximum uncertainty about a student’s ability. Time is measured in days and ability is measured in logits.

Growth over Time. The rate of growth in ability is variable. Based on prior research, typically younger students grow at a faster rate than older, more-experienced students. Modeling the growth rate as a decreasing function of current ability incorporates this feature. This function assumes that the minimum growth rate per year is 10.