5. Research Design, Evidence, and Core Sections
5. Research Design, Evidence, and Core Sections
The decision
Research design is the logic connecting a question to an answer. Methods are the procedures inside that logic.
Design determines what comparison is possible; the comparison determines what claim is credible.
Learning outcomes
You will be able to:
- match descriptive, comparative, causal, and interpretive questions to designs;
- turn abstract concepts into observable evidence;
- identify confounding, selection, measurement, and transfer limits;
- make every paper section perform a distinct research job.
1. Match question and design
| Question | Possible design | Evidence produced | Main limit |
|---|---|---|---|
| What changes after feedback? | before–after study | within-student change | time and practice effects |
| How do two feedback forms differ? | comparative cohort | between-group difference | group selection/confounding |
| What is the causal effect? | randomised or strong quasi-experiment | estimated counterfactual difference | compliance, attrition, external validity |
| How is feedback used? | revision coding or process tracing | observed uptake pathway | coding and interpretation |
| How do students experience it? | interviews/focus groups | situated meanings | accounts are not performance effects |
Mixed methods are justified when two forms of evidence answer different parts of the same question—not because “more data is better.”
2. Operationalise the key ideas
Use a construct-to-evidence table.
| Construct | Indicator | Measurement rule | Threat |
|---|---|---|---|
| abstract quality | rubric score | two blinded raters; four criteria | rater disagreement |
| feedback structure | prompt condition | fixed rubric questions versus usual comments | groups may ignore prompts |
| uptake | visible revision linked to a comment | code accepted, adapted, or rejected | silent reasoning is unobserved |
An indicator is evidence for a construct, not the construct itself. A rubric score captures selected dimensions of quality; it does not exhaust “good writing.”
3. Compare three designs for the same case
Design A — one-group before and after
Draft → structured feedback → final draft
Can support: scores improved after the activity.
Cannot establish alone: feedback caused the improvement; students also practised, received teaching, and had more time.
Design B — existing seminar groups
Group 1: structured feedback
Group 2: usual feedback
Both: draft and final ratings
Can support: change differed between these groups.
Main threat: groups may differ in teacher, timetable, prior skill, or motivation.
Design C — random assignment within the class
Eligible students → random allocation → two feedback forms → blind outcome rating
Can better support: an average causal effect in this class, if allocation, adherence, missing data, and blinding are credible.
Still cannot establish: the same effect in every discipline, institution, or postgraduate population.
The strongest design is the strongest feasible and ethical design, not automatically the most complex one.
4. Audit inference threats
| Threat | Diagnostic question | Possible response |
|---|---|---|
| selection | who enters each group, and why? | randomise; match; measure baseline differences |
| confounding | what else changes with the intervention? | hold teaching constant; collect covariates |
| measurement | would another rater/code give the same result? | blind raters; pilot rubric; report agreement |
| attrition | who is missing at the end? | compare missingness; sensitivity analysis |
| researcher flexibility | could analysis choices follow the result? | specify outcomes and analysis before inspecting results |
| transfer | which feature of this setting may change the effect? | state population and contextual boundary |
A limitation is useful only when it changes interpretation or suggests a concrete check.
5. Calibrate the verb
| Evidence | Prefer | Avoid without stronger design |
|---|---|---|
| cross-sectional association | relates to, is associated with | causes, leads to |
| before–after change | increased after, changed following | produced the change |
| controlled non-random comparison | is consistent with an effect | proves |
| credible randomised estimate | increased on average in this study | always works |
| qualitative accounts | participants described, interpreted as | demonstrates prevalence or effect size |
Precision is not timid writing. It tells the reader exactly how far the evidence travels.
6. Give every section one job
Empirical paper
| Section | Job | Reader's question |
|---|---|---|
| introduction | define problem, question, and contribution | Why this study? |
| literature/theory | establish pattern, mechanism, and gap | What should we expect, and why? |
| method | make the inference auditable | How could this evidence answer the question? |
| results | report observations without moving the goalposts | What was found? |
| discussion | interpret, test alternatives, bound the claim | What does it mean and where does it stop? |
| conclusion | state the answer and consequence | What should be retained? |
Review or argument paper
Replace “method/results” with:
- scope and selection — how the evidence base was formed;
- thematic or analytical sections — how evidence was compared;
- synthesis — what conclusion follows and with what uncertainty.
The labels may vary by discipline; the research jobs do not.
7. Write the method as decisions
Weak:
A quantitative method was used and data were analysed statistically.
Stronger:
Students were assigned within seminar groups to structured or usual feedback. Two raters, blinded to condition and draft stage, scored each abstract using the same four-criterion rubric. The primary comparison was the difference in draft-to-final score change between conditions.
The stronger version lets a reader inspect assignment, measurement, and comparison.
Practice: claim–design hearing
Complete one row, then defend it aloud in 90 seconds.
| Question | Desired claim | Comparison/interpretation | Data | Main threat | Honest verb |
|---|---|---|---|---|---|
Your listener asks only:
- What observation would contradict the claim?
- What rival explanation remains?
- Which population is the conclusion about?
Revise until each answer is explicit.
Quality check
- each construct has an indicator and measurement rule;
- the design creates the comparison the question requires;
- major threats have checks or acknowledged consequences;
- causal language is reserved for a credible counterfactual design;
- every section advances the question rather than repeating background.
Next: make claims traceable through citation, paraphrase, and responsible AI use.