5 Survey design & study types

What to criticise in a survey

Leading / loaded questions that push a response.
Non-response bias. Those who don’t reply may differ systematically.
Self-selection. Online polls, phone-ins are not representative.
Recall / social-desirability bias in the answers themselves.

⭐ Extending the results beyond who was actually studied

A very common flaw in reports, and one examiners plant deliberately. The study measures one group and the conclusion quietly applies to everybody.

Watch for: a sample of university students generalised to all adults; volunteers generalised to the public; one country or season or age group generalised to all; animal studies generalised to humans.

Name it explicitly: “the results can only be extended to the population the sample was drawn from — here, ___ — so the claim about ___ is not justified.”

⭐ What the specification says these questions will ask

Questions “will require candidates to evaluate claims or conclusions made in the report” — so always end with a verdict on the claim, not just a description of the study.
Identify and discuss potential sources of error — sampling error, non-response, self-selection, measurement and question wording.
Calculate and interpret margins of error, using the rule of thumb MoE ≈ 1/√n.
Consider study design and the type of inference it supports — see below.

Type of inference: what the design lets you claim
Design	Valid inference	NOT valid
Random sample from the target population	Generalise to that population	Any causal claim
Observational study (no random allocation)	An association between variables	That one variable caused the other
Randomised experiment	A causal claim for the groups studied	Generalising beyond the group unless it was randomly sampled
Self-selected / convenience sample	A description of those people only	Generalising to anyone else

Random sampling and random allocation are different things

Random sampling is how you choose who is in the study — it earns you the right to generalise. Random allocation is how you decide who gets which treatment — it earns you the right to claim cause.

A study can have one without the other, and the inference you are allowed changes accordingly. Saying “it was random” without saying which kind is the fastest way to lose this mark.

⭐ Population, sampling frame, sample — say which you mean

Target population: everyone the study wants to draw a conclusion about. 2024: “The target population is all UK men and women.”
Sampling frame: the list actually used to draw the sample from.
Sample: who was really measured.
Marks come from the gaps between them. If the frame misses part of the target population, the sample cannot represent it — 2025 asked exactly this, for a clear description of the difference between the target population and the sampling frame.

⭐ Naming the variables

Explanatory variable: the one the researcher changes or groups by. 2024: “footwear — socks over the top of footwear, or no socks.” 2025: “artwork creator: human-created or AI-created.”
Response variable: the one measured as the outcome.
Name both with their actual levels, in context. Writing “footwear” alone is weaker than naming the two groups being compared.

Observational vs experimental

Observational study: just measures: can show association but not causation (lurking variables).
Experiment: researcher assigns treatments (ideally randomised, controlled): can support causation.
Only a well-designed randomised experiment justifies a causal claim.