Exam development
Your questions, your bank, your statistics.
The first time a candidate disputes a result, somebody at your body has to answer a question the scoring system cannot: is this a good question? Scoring a paper and knowing whether the paper measures anything are different capabilities.
01 The item bank
Authored and reviewed where it lives
- Authoring and review are separate acts by separate people, on the record
- Revisions keep every version; relationships and testlets keep items together
- Seven formats, each with a scoring contract fixed when the item is written
- Written responses are marked by people against a rubric, double-marked with adjudication when you say so
Questions are written, reviewed and held in your own bank, under your body's control. Subject-matter experts work in a secure space rather than on a document emailed around, and a review workflow records who looked at an item and what they concluded. Items carry their content area, so a form can be assembled to a blueprint rather than by hand. Every revision is kept; items that belong together — a shared stimulus, a testlet, an enemy pair that must never share a form — are related in the bank rather than in somebody's memory.
- Selected responseSingle best answer, multiple response, true/false.
- Structured formatsMatching, ordering, numeric within a tolerance, and short constructed responses — each with a versioned scoring contract that says exactly how points are awarded, frozen when the item goes to review and applied by one function everywhere a response is scored.
- Written responsesScored by people against a rubric fixed when the question was written: named criteria, whole-point levels with descriptors. One marker, or two independent markers within an agreement tolerance and a third person to adjudicate. Marker agreement is reported per question and per marker.
02 Blueprints and assembly
Assembled deliberately
- A content outline with weights and tolerances; assembly proposes forms against it
- Anchor items recur so forms can be linked; exposure caps and overlap limits rest the rest
- Pretest items ride a live form and are scored for nobody
- A window can hand each candidate one published form from a pool
A form is built from sections, each drawing on the bank, against a blueprint: the content outline, its weights and the tolerance each may miss by. Assembly proposes forms that satisfy it, then applies your program's published assembly policy — an exposure cap so a question thousands of candidates have seen may rest, an overlap limit with each live form, and a minimum and maximum of designated anchor items so forms can be linked by an equating study. Exposure is derived from the papers candidates were actually shown, never from a counter that could drift.
Pretest items can be carried on a live form and scored for nobody — the only honest way to gather statistics on a new question is to put it in front of real candidates under real conditions, and the only honest way to do that is to make sure it cannot affect their result. How many pretest items a form may carry is capped per program, and whether they are visible as such is your decision.
A window can give each candidate one published form from a pool, decided at admission — whole forms that were reviewed and published, never a paper assembled per candidate that nobody reviewed and no equating study describes. Everything after that point is unchanged: the paper is frozen from that form, scored with that form's conversion, reconstructed from that form.
03 Item analysis
Including the statistic that finds a miskeyed question
- Difficulty, point-biserial and upper–lower 27% discrimination, distractor analysis — per item, per sitting
- Miskey alarms when the strongest candidates prefer a distractor
- Flags and statistics travel with the item into review
After a sitting, every item is analyzed.
- DifficultyThe proportion who answered correctly. Cheap, and most useful at the extremes: an item everybody passes and an item everybody fails both consume a candidate's time and separate nobody.
- Discrimination, two waysThe point-biserial correlates getting this item right with doing well on the rest of the paper; the upper–lower 27% index says how much more often the strongest quarter got it right than the weakest. The first is the better statistic and the second is the one you can explain to a board in a sentence. A body defending a cut score needs both.
- Distractor analysisWhich wrong answers were chosen, and by whom. A distractor that the strongest candidates pick more often than the key is the signature of a miskeyed item, and a miskeyed item does not merely fail to measure: it marks the people who know the material wrong. This is the alarm that finds one.
04 When a candidate challenges an item
A route that ends in a real decision
A candidate may challenge a specific question within a window you set. The challenge goes to your people with the item and its statistics in front of them. You decide, and the decision propagates to every score that item touched — rescoring the cohort if that is what the decision means, rather than leaving one candidate corrected and the rest wrong.
05 Course Studio
Learning content built from grounded sources
Course Studio builds learning content from grounded sources — the standards, texts and materials you supply — into sessions with figures, narration and practice checks, reviewed by your people and published as editions. A published edition is never rewritten; a successor supersedes it and enrollments carry their own edition. Sessions can be offered to learners with a price or without one, and a completion can stand as evidence against an eligibility requirement.
- Sessions, figures, narration and practice checks, generated from sources you supply
- Published as editions that are never rewritten; a successor supersedes
- A session can be sold, enrolled and completed; completion can feed eligibility
Related
- Practice analysisJob-task analysis: the study design, the expert panel, the outline, the validation survey and the results behind the blueprint.More about Practice analysis
- Standard settingThe study behind the cut score — recorded, approved and locked before candidates register.More about Standard setting
- PsychometricsReliability and SEM, decision consistency, fairness review, equating, scaled scoring and technical reports with a frozen evidence manifest.More about Psychometrics
- Online deliveryBrowser delivery of frozen forms with server-side scoring, autosave and a complete sitting record.More about Online delivery
Ask about bringing your bank across
Tell us about your program.
Tell us how many candidates you certify and what you run today. You will get a written quote against the published plans — not a discovery call before a number.