What does it do?

Multi-stage tests (MST) must be historically the earliest attempt to achieve adaptivity in testing. In a traditional, non-adaptive test, the items that will be given to the examinee are completely known before testing has started, and no items are added until it is over. In adaptive testing, the items asked are, at least to some degree, contingent on the responses given, so the exact contents of the test only becomes known at the end. (Bejar 2014) gives a nice overview of early attempts at adaptive testing in the 1950s. Other names for adaptive testing used in those days were tailored testing or response-contingent testing. Note that MST can be done without any computers at all, and that computer-assisted testing does not necessarily have to be adaptive.

When computers became ubiquitous, full-scaled computerized adaptive testing (CAT) emerged as a realistic option. In CAT, the subject’s ability is typically reevaluated after each item and the next item is selected out of a pool, based on the interim ability estimate. In MST, adaptivity is not so fine-grained: items are selected for administration not separately but in bunches, usually called modules. In the first stage of a MST, all respondents take a routing test. In subsequent stages, the modules they are given depend on their success in previous modules: test takers with high scores are given more difficult modules, and those with low scores are given easier ones – see e.g., Zenisky, Hambleton, and Luecht (2009), Hendrickson (2007), or Yan, Lewis, and Davier (2014).

Hence, MST can be seen as middle ground between CAT and linear testing, and can be approached from either side: as a CAT that has been somewhat coarsened, or as adding adaptivity to the more traditional designs. We prefer to go the latter way because our experience is so much larger with linear, multi-booklet designs.

Below is a schematic representation of the data from a very simple, linear, one-booklet test. We assume that the columns have been sorted by increasing item difficulty, and the rows have been sorted by increasing person ability.

A first example

Note the two corners that have been highlighted in blue. When we ask very easy items to very able examinees, the responses are quite predictable, and hence not very informative. Most will be correct, and the few wrong ones might be due to momentary lapse of attention (slipping). When we ask difficult items to less able examinees, the result is the opposite yet similar, and the few correct responses might be attributed to lucky guessing.

What if we simply refuse to collect this worst, least informative part of the data? We end up with something akin a very simple MST. The central part that has been given to all examinees may be regarded as a routing test, and the two incomplete parts as the follow-up modules: A for the less able examinees, B for the others. If we don’t ask items that are much too easy or much too difficult for the examinee, we avoid demotivation and frustration, correspondingly, and with them much of the motivation to approach the data with a 3- or 4-parameter IRT model. It seems preferable to downsize slipping and guessing physically than to try to accommodate them mathematically.

This is a somewhat primitive and not entirely accurate example – for example, the routing test does not have to be confined to items of average difficulty. Still, it explains some of the basic motivation for adaptivity and MST. To get closer to actual work with MST, it is more convenient to represent the test design with a tree diagram. A very simple example is shown below:

The tree is read from top to bottom. The root represents the first stage where all examinees take the first module (the routing test). In the second stage, examinees with a score lower than or equal to 5 on the routing test take module 2, whereas examinees with a score higher than 5 on the routing test take module 3. Every path corresponds to a booklet. In this MST, there are two booklets: the first one, booklet M1-M2, should be relatively easy while the other, M1-M3, is more difficult.

Unlike CAT, MST is not all about algorithms. Most concepts, steps, procedures known from linear testing and familiar from dexter are still in place, but in a modified form: some are a bit more complicated, others quite a bit, a few become meaningless. We next review the basic workflow, which is similar to dexter with a few important additions, and we discuss some of the differences in more detail.

How do I use it?

There are not too many workflows to manage and analyze test data. In dexter, the procedure is, basically:

1. Start a project
2. Declare the scoring rules for all items
3. Input data
4. Examine data, assess quality with classical statistics, make necessary adjustments
5. Estimate item parameters
6. DIF, profile analysis …
7. Estimate and analyze proficiencies

In dexterMST, we follow more or less the same path except that we must, between steps 2 and 3, communicate the MST structure to the program: the modules, the booklets, and the routing rules.

db = create_mst_project(":memory:")

add_scoring_rules_mst(db, scoring_rules)

routing_rules = mst_rules(
  easy = Mod_1[0:5] --+ Mod_2, 
  hard = Mod_1[6:10] --+ Mod_3)

create_mst_test(db,
                test_design = design,
                routing_rules = routing_rules,
                test_id = 'ZwitserMaris')

add_booklet_mst(db, bk1, test_id = 'ZwitserMaris', booklet_id = 'easy')
add_booklet_mst(db, bk2, test_id = 'ZwitserMaris', booklet_id = 'hard')

fi = fit_inter_mst(db, test_id = 'ZwitserMaris', booklet_id = 'hard')

plot(fi, item_id='item21')
plot(fi, item_id='item45')

f = fit_enorm_mst(db)

coef(f)

## 1 item properties updated for 50 items

## Warning: Removed 33 rows containing missing values (geom_point).

abl = ability_tables(f, method='MLE')
abl

f2 = fit_enorm_mst(db, item_id != 'item21')

CML estimation with MST

dexter’s estimation routines cannot be directly applied in MST where the design is not fixed in advance but evolves during testing based on the subject’s responses. Ordinary CML, in particular, gives biased results under the circumstances and was long believed to be impossible for MST (Eggen and Verhelst (2011), Glas (1988)). This was proven wrong by Zwitser and Maris (2015) who demonstrated that CML estimation is indeed possible, provided that one takes the design into account. Furthermore, they argued that sensible model aka those that admit CML will in general fit quite well to MST data.

In the three years since, the results of Zwitser and Maris (2015) have not been mentioned in any of the recent edited volumes on MST, and dexterMST is, to our best knowledge, the first publicly available attempt at a practical implementation. MST data are usually analyzed with marginal maximum likelihood (MML), which is available in a number of R-packages, notably mirt. MML estimation makes assumptions about the shape of the ability distribution (usually a normal distribution is assumed), and it can produce unbiased estimates if these assumptions are fulfilled. CML, on the other hand, does not need any such assumptions at all, so it can be expected to perform well in a broader class of situations.

That MML gives biased results if the ability distribution is misspecified has been shown quite convincingly by Zwitser and Maris (2015). We reproduce their example here without the code but note that the data have been simulated with an ability distribution that is not normal but a mixture of two normals. Here is a density plot.

Note that a distribution that is not normal is not, in any way, ab-normal. In education, skewed or multi-modal distributions like this do occur as a result of many kinds of selection procedures. Below are the estimated item difficulty parameters plotted against the true parameters:

The results illustrate the well-known fact that both naive CML and MML estimates can be severely biased. The latter are not biased because of the MST design but because the population distribution was misspecified.

Note that the colors indicate whether the items occurred in module 1, module 2 or module 3. It will be clear that the modules are appropriately, albeit not perfectly, ordered in difficulty. Judging from the p-values, the (simulated) respondents would have been comfortable with the test.

tia_tables(get_responses_mst(db))$testStats %>%
  select(booklet_id, meanP)

tia_tables(get_responses_mst(db))$testStats %>%
  select(booklet_id, meanP) %>%
  kable(caption='mean item correct')

Note how the dexter function tia_tables was used. To wit, we first got the response data using get_responses_mst and used these as input. This detour was necessary because the data-bases of dexter and dexterMST are not directly compatible.

In the same way, we can use dexter’s plausible_values function to show that the original distribution of person abilities can be approximated quite well by the distribution of the plausible values:

rsp_data = get_responses_mst(db)
pv = plausible_values(rsp_data, parms = f)

plot(density(pv$PV1), main='plausible value distribution', xlab='pv')

Note that the output from fit_enorm_mst can be used in dexter functions without further tricks. For further reading, we refer the reader to the dexter vignette on plausible values.

Predicates and MST

Response-contingent testing, to use the charming term from the past, introduces many intricate constraints and dependencies in the test design. As a result, not only the complicated techniques, but even such apparently trivial operations as removing items from the analysis or changing a scoring rule can become something of a minefield. Things are never as easy as in a linear test!

In CML calibration, it is essential that we know the routing rules so, to remove some items from analysis, one needs to infer the MST design without these items. What happens internally in dexterMST is that a new MST is defined for each possible score on the items that are left out, with the routing rules that follow from the ones originally specified. Consider, for example, the design that corresponding to the analysis with item21 left out.

design_plot(db, item_id!="item21")

As one can see, the tree is split to accommodate the examinees who answered this item correctly, and those who did not. While more complex predicates are allowed, it will be clear that these may involve complicated bookkeeping which takes time and might slow down the analysis.

At the time of writing, it is not possible to change the scoring rules, except for items in the last modules of the test. As usual, the advice is to avoid this. Predicates that remove complete modules from a test, e.g. module_id != 'Mod_1' will cause an error and should be avoided.

Beyond two stages: ‘all’ and ‘last’ routing

So far, we have considered the simplest MST design with just two stages. dexterMST can handle much more complex designs involving many stages and modules. As an example, we show a diagram corresponding to one of the MSTs used in the 2018 edition of Cito’s Adaptive Central End Test (ACET):

One ACET test

With more than two stages, two different methods of routing become possible:

Last: Use the score on the present module only.
All: Use the score on the present and the previous modules.

dexterMST fully supports both types of routing. We do require that a routing type applies to a complete test and is specified when the test is created, for example create_mst_test(..., routing='last'). In this vignette we used ‘all’, which is the default value.

It is worth noting that the ACET project includes both MST and linear tests. The linear tests are simply entered as a single module MST with a trivial routing rule, e.g.:

lin.test_design = data.frame(module_id='mod1', item_id=paste('item',1:30), item_positon = 1:30)
lin.rules = mst_rules(lin.booklet = mod1)
create_mst_test(db, lin.test_design, lin.rules, test_id = 'linear test')

ACET is also the biggest project we know. In total, the project database contains the responses of 97225 students to 3622 items spread over 169 tests and including six distinct MSTs. Could dexterMST analyse the data? Sure! On a Windows 64-bit laptop with an 2.9 GHz processor, this took about 1.5 minutes to calibrate.

dexter vs. dexterMST: A summary for dexter users

dexterMST is a companion to dexter. It loads dexter automatically, and many of that dexter’s functions can be used immediately, notably those for ability estimation. When that is not the case, there will be some kind of warning. The new functions relevant for MST have mst in their names. In addition, we have tried to keep the general logic and workflow as similar to dexter as possible. Thus, experienced dexter users should find dexterMST easy to understand. Some of the most important differences are listed below.

• It is no longer possible to infer the test design automatically from the scoring rules and the data; the user must specify the test design explicitly through the modules and the routing rules before the response data can be input.

• In dexterMST there are restrictions on altering the scoring rules. Specifically, it is not possible to change scoring rules for any items unless they only appear in the last stage of a test. Attempts to circumvent this restriction by changing the scoring rules midway will lead to wrong calibration results.

• dexter can work with or without a project database. Due to the extra dimensions of the data in MST designs, dexterMST requires a project database.

• The data bases created by dexter and dexterMST are not compatible or exchangeable but the parameter object that is the output from fit_enorm_mst can be used in dexter functions like plausible_values and ability as we have illustrated earlier.

For your convenience, a function import_from_dexter is available that will import items, scoring rules, persons, test designs and responses from a dexter database into the dexterMST database.