2. Search, Evaluate, and Read Evidence
2. Search, Evaluate, and Read Evidence
The decision
Searching is not “finding papers.” It is creating a defensible path from a question to a body of evidence.
Learning outcomes
You will be able to:
- translate a question into searchable concepts and synonyms;
- record databases, dates, queries, and screening decisions;
- judge relevance separately from credibility;
- read a paper in passes and extract only decision-useful information.
1. Turn the question into concepts
Do not paste the full question into a database. Build concept blocks.
| Block | Search terms for the running case |
|---|---|
| intervention | “peer feedback” OR “peer review” OR “peer assessment” |
| product | “academic writing” OR abstract* OR “research writing” |
| population | undergraduate* OR “first-year student*” OR “higher education” |
| design/outcome, if needed | rubric OR “writing quality” OR revision |
Example:
("peer feedback" OR "peer review")
AND ("academic writing" OR abstract* OR "research writing")
AND (undergraduate* OR "higher education")
Start with two or three blocks. Add a fourth only when irrelevant results dominate; too many blocks can hide useful work.
2. Search in layers
| Layer | Purpose | Action |
|---|---|---|
| orientation | learn vocabulary and major debates | read one recent review and one handbook entry |
| database search | retrieve reproducibly | search disciplinary databases and the library catalogue |
| citation chaining | locate the conversation | inspect references and later papers citing strong sources |
| gap search | challenge your emerging view | search rival terms, null results, critiques, and other contexts |
Discovery tools suggest neighbours; they do not establish coverage. For a systematic or scoping review, follow a field-appropriate protocol and report every source and screening stage. PRISMA is a reporting guideline, not a badge for an ordinary essay.
3. Keep a search log
| Date | Source | Exact query | Filters | Results | Kept | Change next |
|---|---|---|---|---|---|---|
| 12 Oct | education database | "peer feedback" AND "academic writing" | 2018–2026; article | 146 | 11 | add undergraduate terms |
| 12 Oct | citation search | papers citing a 2019 meta-analysis | none | 83 | 7 | inspect task design |
The log prevents accidental cherry-picking and lets another researcher repeat the route.
A practical stopping rule
For coursework, stop expanding when:
- new searches mostly return sources already seen;
- each major claim has more than one relevant source;
- disagreements and limitations are represented;
- one further search round changes no section of the outline.
This is saturation for a bounded assignment, not proof that the literature is complete.
4. Screen relevance before quality
Use the same inclusion logic for every result.
Include if: higher education + peer feedback on an academic text + observable writing or revision evidence
Exclude if: school setting only; feedback is teacher-only; outcome is satisfaction with no writing evidence
At title/abstract stage ask: Could this source answer any part of my question?
At full-text stage ask: Which claim could it support, and under what limit?
Record a reason for exclusion. “Not useful” is too vague; “population outside scope” is auditable.
5. Evaluate credibility laterally
Do not remain inside the document. Open new tabs and investigate:
- Origin — Who produced it, and for what purpose?
- Method — What was observed, compared, or interpreted?
- Measurement — Do measures represent the stated concepts?
- Inference — Does the conclusion exceed the design?
- Corroboration — Do independent sources agree, qualify, or contradict it?
Professional fact-checkers often evaluate unfamiliar online sources by leaving the page and checking the wider information environment—a practice called lateral reading (Wineburg & McGrew, 2019). Apply the same principle to reports, datasets, preprints, and AI-generated answers.
6. Read in three passes
Pass 1 — Decide whether to continue
Read title, abstract, headings, figures/tables, and conclusion. Write:
Question:
Design/data:
Main answer:
Possible use:
Pass 2 — Reconstruct the evidence
Read method, results, and limitations. Ask:
- What exactly was compared?
- How were constructs measured?
- What uncertainty is reported?
- Which alternative explanation remains?
Pass 3 — Place it in the conversation
Check references, cited-by links, supplementary material, corrections, and competing studies. Decide whether the source is foundational, confirming, conflicting, methodological, or contextual.
7. Make a source card
Example based on the 2024 peer-feedback review:
Source: Wei & Liu (2024)
Question: What benefits and challenges are reported for peer feedback in higher-education academic writing?
Evidence: systematic review; 60 empirical articles published 2014–2024
Finding: benefits were reported across cognitive, behavioural, affective, social, and metacognitive domains
Boundary: the review reports a predominance of undergraduate and qualitative studies
Use: motivates the topic and identifies an evidence/design gap
Do not claim: structured peer feedback causes higher abstract scores
The final line is crucial: it prevents a relevant source from being used for a claim it did not test.
Practice: search-and-screen lab
- Build three concept blocks for your question.
- Run two searches and record them exactly.
- Screen 20 results with one-sentence reasons.
- Read three retained sources in three passes.
- Produce one source card for the strongest and one for the strongest counter-source.
Quality check
- search terms include synonyms and rival language;
- dates, databases, filters, and decisions are recorded;
- inclusion rules were written before seeing preferred results;
- relevance and credibility are judged separately;
- every retained source has a possible use and a “do not claim” boundary.