Reliability questions can come up in Section D of both papers. The big idea is consistency: would the research give the same kind of result if it was done again, or if someone else did the judging?
Sign up to access the complete lesson and track your progress!
Unlock This CourseThink about a bathroom scale. If you step on it three times in a row and it says 50 kg, then 58 kg, then 46 kg, you cannot trust it. The scale is not consistent. In research, a procedure or a measure that gives inconsistent results has poor reliability.
Key terms:
You do not need to know how to measure inter-rater or inter-observer reliability. You only need to know what they mean, how to spot good or poor reliability in a scenario, and how to improve it.
Reliability is about consistency. The last lesson showed that validity is about measuring the right thing. A measure can be consistent and still measure the wrong thing, so the two ideas are different.
Two teachers each mark the same set of pupils' answers to a question. If they give very similar marks, there is agreement and inter-rater reliability is high. If one gives most answers full marks and the other gives most answers very low marks, it is low.
Two researchers watch the same children playing and each record how often a child shares a toy. If they record about the same amount, inter-observer reliability is high. If one records 12 shares and the other records 3, it is low.
Both ideas are about the same thing: when more than one person judges or watches the same thing, do they agree? Raters judge responses, such as answers, drawings or interview replies. Observers watch behaviour as it happens.
Poor agreement is a problem because you cannot tell which person is right. The results may say more about who did the judging than about the participants.
In the exam you will meet a study you have not seen before. Ask yourself:
Two researchers watch the same nursery for an hour. Each tries to record how many times children are "kind" to each other. One records 14 kind acts and the other records 5.
Judgement: low inter-observer reliability.
Reason: the two observers watched the same behaviour but did not agree, so the results are not consistent. "Kind" has probably been understood in different ways.
Improvement: give both observers the same clear list of what counts as kind, such as "offers a toy" or "helps a child who has fallen", and practise using it together before the study starts.
Run the study in exactly the same way each time, with the same steps, timings and wording. Then any difference in results is less likely to come from the procedure changing.
Define exactly what counts, such as "hits another child with a hand" instead of "is rough". Observers and raters then use the same rules, so they are more likely to agree.
Train observers or raters before the study. Let them practise on example behaviour or answers and discuss any differences, so everyone uses the rules in the same way.
Use two or more people and compare what they record or score. If they disagree, the researchers can see there is a problem and put it right. One person alone cannot be checked.
Match the improvement to the problem in the scenario. If two raters disagree about what a good answer is, give them clear rules and training. If a study was run differently for each participant, standardise the procedure.
Two psychologists want to find out whether children who watch a short cartoon share more sweets. Each psychologist watches the same group of children and records how many times a child "acts generously". Psychologist A records 18 acts and Psychologist B records 7.
(a) Explain what the results suggest about the inter-observer reliability of this study. [2 marks]
(b) Suggest one way the psychologists could improve the reliability of this study. [2 marks]
(a) Inter-observer reliability is low (1) because the two observers watched the same behaviour but did not agree on how many generous acts there were (1).
(b) Give both observers a clear list of exactly what counts as acting generously, such as handing a sweet to another child (1), so they use the same rules and are more likely to agree (1).
For a reliability question, always mention agreement or consistency, and say who is being compared, such as the two observers.