By Ritvars Mētra, Founder of ReadinessCompassTwo readiness surveys go out in the same week to comparable populations. One asks whether people agree that they are “well prepared for the upcoming changes”. The other asks whether they have “completed the new process end to end without help”.
The first comes back at 71 per cent. The second comes back at 24 per cent. Nothing about the two populations differs. The instrument produced the difference.
This is a studied effect, not an anecdote
Norbert Schwarz set out the mechanism in Self-reports: How the questions shape the answers in American Psychologist. Self-reports of behaviour and attitudes are strongly influenced by features of the instrument — question wording, format and context — and the underlying cognitive and communicative processes are systematic and increasingly well understood.
The important word is systematic. These are not random errors that average out across a large enough sample. They push in a consistent direction, which means a badly worded readiness item does not merely add noise — it produces a confident, reproducible, wrong number.
Respondents also treat a survey as a conversation. They assume the questions are relevant, that the response options describe the expected range of answers, and that the researcher is asking in good faith. Every one of those assumptions is an opportunity to mislead them accidentally.
Six defects, and their repairs
| The item | What goes wrong | Better |
|---|---|---|
| “I have received adequate training and support” | Double-barrelled. Two things, one answer. A person well trained and unsupported cannot respond honestly | Split into two items |
| “I feel prepared for the changes” | Undefined referent. Prepared for what, and by whose standard? Each respondent answers a different question | “Have you completed [specific task] in the new system without help?” |
| “I have been kept informed recently” | Vague reference period. “Recently” means a week to some and a quarter to others | “In the last two weeks…” |
| “The new system will improve how we work” | Leading. Asserts the benefit and invites agreement | “What effect do you expect on your own work? Better / no change / worse / too early to say” |
| “I am confident in my ability to use the system” | Confidence is not capability. It measures self-assessment, which is weakly related to performance | Pair it with a behavioural item and compare |
| “Change is handled well in this organisation” | Attitude to the organisation, not readiness for this change. Moves with unrelated events | Ask about this change, this team, this month |
The second row is the one that costs programmes most. “I feel prepared” is the single most common readiness item in circulation, and it is close to uninterpretable — because the respondent is comparing themselves against a standard they invented while reading the question.
Agreement scales invite agreement
Almost every readiness survey is built on agree/disagree items, and that format carries a known bias. Among the satisficing strategies Jon Krosnick documented in Applied Cognitive Psychology is simply agreeing with the assertion made by the question — acquiescence — alongside endorsing the status quo and failing to differentiate across a set of items.
Three consequences follow for a readiness instrument:
- A page of positively worded statements will score positively regardless of the truth, because agreeing is the low-effort response and every item points the same way.
- Straight lines are data, not laziness. Non-differentiation across diverse items is a documented satisficing strategy. Flag those responses; do not average them in silently.
- Item-specific response options beat generic agreement. “How many times have you used the new process?” with counted options cannot be acquiesced to.
The standard textbook remedy — reverse-wording some items so agreement is not always the positive answer — helps detect the problem but introduces its own: reverse items are more cognitively demanding and are themselves misread under time pressure. Use one or two as a check, not throughout.
Order and context change the answer
Schwarz’s work covers context effects in attitude measurement, and they are easy to create accidentally in a readiness survey.
Ask three questions about workload and stress, then ask about confidence in the change, and the confidence answer will be lower than if the same item had come first. Ask about the benefits of the programme, then about personal readiness, and readiness reads higher. Neither ordering is dishonest; both produce a different number.
The practical rules are simple. Put behavioural and factual items before attitudinal ones, since facts are less susceptible to priming. Keep the order identical between waves — if you reorder a survey between pulses, the change you observe may be entirely instrumental. And group items by topic so respondents are not asked to switch frame repeatedly, which raises cognitive load and therefore satisficing.
A cheap test that catches most of it
Before a survey goes out, sit with four people from the target population and have them think aloud while answering. Not a review of the document — watch them answer, and ask after each item what they took it to mean and what they were comparing themselves against.
Half an hour reliably surfaces the items where two people read the same words differently, which is the defect that no amount of statistical treatment can repair afterwards. It is the same principle that makes watching people attempt a task better evidence than asking them about it: observation catches what self-report cannot.
The uncomfortable implication of all this is that a readiness number is a property of the instrument as much as of the organisation. Two competent teams can measure the same population and disagree by thirty points without either doing anything wrong. That is a reason to write questions carefully and to keep them identical over time — not a reason to distrust measurement, but a reason to treat any single score as one input rather than a verdict.
More on readiness measurement in the Change Readiness Hub, or start with how to design the survey itself.