12 Method Chapter
The method chapter is written in past tense and must be detailed enough that another researcher could fully replicate your study. It is highly standardized in structure.
Descriptions of your findings in the result section → past tense (“The results showed that…”)
12.1 Quantitative research (survey and experiment)
12.1.1 Sample and Participants
- Method of recruitment (convenience sampling, snowballing - How were participants recruited?)
- Sample size, including justification (a priori power analysis — see below)
- Exclusions: exactly how many participants were excluded and for what reason(s)
- The final N (must match the N reported in the results section)
- Demographics: frequency and percentage for gender, age, education, etc.
Example:
“Participants were recruited via Prolific Academic. A power analysis based on an expected small-to-medium effect (r = .20, α = .05, power = .80) indicated a required sample of N = 197. A total of 231 participants were recruited. After excluding 19 participants who failed two or more attention checks and 12 who completed the survey in under four minutes, the final sample consisted of N = 200 participants (54% female; Mage = 29.8, SD = 7.6, range: 18–61; 51% with a university degree).”
A priori power analysis
Every quantitative study must justify its sample size with an a priori power analysis — conducted before data collection, not after. The logic is straightforward: four quantities are mathematically linked, and fixing three determines the fourth.
| Quantity | Typical value | What it means |
|---|---|---|
| Effect size | Estimated from prior work | The smallest effect you want to be able to detect |
| Alpha (α) | .05 | Acceptable Type I error rate (false positive) |
| Power (1 − β) | .80 | Probability of detecting the effect if it truly exists |
| Sample size (N) | → Solve for this | The output of the power analysis |
Choosing and justifying an effect size: The effect size estimate is the most consequential input and must be explicitly justified — not assumed. Use one of the following approaches:
- Prior literature: Report the effect size from the most relevant prior study (or a meta-analytic estimate if available). Explain why that study is a reasonable benchmark for yours.
- Smallest effect size of interest (SESOI): Define the minimum effect that would be theoretically or practically meaningful, and explain why smaller effects would not matter.
- Conservative estimate: If prior evidence is sparse, use a small effect size and explain that choice.
In Communication Science, effects are typically small to medium (r = .10–.30, d = 0.20–0.50). Do not default to a “medium effect” without justification — this inflates your assumed power and may leave your study underpowered for the effects you actually expect.
Post-hoc power analysis — calculating power after data collection using the observed effect size — is circular and uninformative. It tells you nothing beyond the p-value you already have. Always run power analysis a priori, before collecting data.
Reporting: Report the expected effect size and its justification, α, desired power, the resulting N, and the software used:
“Based on prior work by Masur et al. (2021), who found a moderate effect of peer disclosure on disclosure intentions (β = .24), we expected a small-to-medium effect (r = .20). A power analysis (α = .05, power = .80) indicated a required sample of N = 197 (calculated using the
pwrpackage in R; Champely, 2020).”
For the R code to run the power analysis, see the R for Data Analysis chapter. For a detailed conceptual introduction, see this blog post.
12.1.2 Design
- State the type of design clearly: survey, experiment, between-subjects, within-subjects, mixed
- Describe the task if there was one (e.g., “Participants were exposed to a news article and then completed a questionnaire”)
- Name the independent and dependent variables
- For experiments: describe factors and conditions clearly using design notation (e.g., “a 2 [logo color: red, blue] × 2 [medium: TV, newspaper] between-subjects design”), and confirm randomization
12.1.3 Procedure
Describe in chronological order:
- What participants were exposed to (especially condition differences in experiments)
- What tasks and measurements were completed
- Key instructions given (purpose of study, debriefing)
12.1.4 Materials
- Describe experimental stimuli or materials, including exact differences between conditions
- All materials must be provided in the appendix (or even in the text itself if it helps the reader)
12.1.5 Measures
For each variable, report:
- Variable name (in bold or italics)
- What it measures
- Source reference (original scale)
- Number of items
- Example item(s) — or all items if few; refer to appendix for longer scales
- Response format (e.g., “5-point Likert scale from 1 = strongly disagree to 5 = strongly agree”)
- M and SD in your sample, along with minimum and maximum values
- Reliability (Cronbach’s α and/or McDonald’s omega), but also results of confirmatory factor analyses if applicable!
- How the final score was computed (e.g., average of items; recoded items)
- Direction (e.g., “Higher scores indicate greater vaccine hesitancy”)
- For experiments: a manipulation check!
Eventually organize measures under subheadings: Independent variables, Dependent variables, Control variables.
Example:
Privacy concerns were measured with four items adapted from Malhotra et al. (2004; e.g., “I am concerned about threats to my personal privacy online”). Items were answered on a 5-point Likert scale (1 = strongly disagree; 5 = strongly agree). Confirmatory factor analysis confirmed a single-factor structure (CFI = .97, RMSEA = .05). The scale showed good reliability (ω = .84; M = 3.12, SD = 0.91, range: 1–5). Higher scores indicate greater privacy concerns. The final score was computed as the mean of all four items.
The variable names in your method section must be identical to those used in your hypotheses, conceptual model, and results section. This is one of the most common sources of errors in MA theses.
12.2 Content analysis
A content analysis method chapter is structured similarly to a survey/experiment chapter but with different building blocks. The core sections below should appear in this order.
12.2.1 Research design
State explicitly that this is a content analysis and whether it is manual, automated, or hybrid (e.g., automated classification with a human-validated subsample). Briefly state the purpose: what is being classified and why.
12.2.2 Corpus and sampling
Describe how the corpus was assembled. Be precise:
- Sources: which platforms, outlets, archives, or datasets
- Date range: exact start and end dates
- Search/inclusion criteria: keywords, hashtags, account lists, filters
- Sampling strategy: full population (census), random sample, stratified sample (e.g., across time or sources), or systematic sample
- Final N: total number of items in the corpus, and how many were excluded and why
- Replication: state whether the corpus is shareable (and if not, why — e.g., platform ToS)
12.2.3 Unit of analysis
State explicitly what is being coded — this is a frequent gap in MA theses. Common units:
- Document level: full article, full post, full video
- Paragraph/sentence level: useful when one document contains multiple frames or stances
- Utterance/turn: in interviews or dialogue data
- Image or modality slice: for multimodal content
Justify the choice: the unit should match the granularity of your construct (e.g., if you study “stance toward a policy,” the sentence may be more appropriate than the article).
12.2.4 Coding approach
Describe the analytical procedure. Manual and automated approaches have different reporting requirements.
Manual coding
- Codebook: which categories, with definitions and decision rules. The full codebook belongs in the appendix.
- Coders: how many, who (background, training)
- Training: how coders were trained on the codebook (number of pilot rounds, refinement steps)
- Double-coding: what proportion of the corpus was coded by ≥ 2 coders for reliability (typically 10–20%)
- Disagreement resolution: how disagreements were resolved (discussion, third coder, retention as disagreement)
Automated coding
State which approach you used and why. The main families, in increasing order of flexibility and resource cost:
| Approach | When to use | Tools |
|---|---|---|
| Dictionary / keyword-based | Concept is well-defined by a closed list of terms | LIWC, custom R/Python dictionaries |
| Supervised machine learning | Sufficient hand-coded training data; concept is too fuzzy for a dictionary | scikit-learn, quanteda |
| Transformer models (e.g., BERT) | Modest training data; want strong generalization | HuggingFace, transformers in R/Python |
| Large language models (zero/few-shot) | Limited labeled data, but concept can be described to a model | OpenAI API, Anthropic API, open-weight LLMs |
For all approaches, report: the model/tool used, version, prompt or training data (in appendix), and preprocessing steps.
Every automated method — including dictionaries, ML, BERT, and LLMs — must be validated against a human-coded gold standard. An unvalidated automated classification is not a measurement; it is a guess at scale. The CADC course materials and the CCS Amsterdam tutorials provide guidance on building validation sets.
LLMs are convenient but not always the best tool. Small, fine-tuned transformer models (e.g., BERT-NLI) often outperform general-purpose LLMs on narrow classification tasks and are reproducible and cheap to run. If you do use an LLM, the prompt is part of your method and must be reported in full — small wording changes can shift results substantially.
12.2.5 Reliability and validation
| Approach | What to report | Threshold |
|---|---|---|
| Manual | Krippendorff’s α (preferred) or Cohen’s κ for each category | α/κ ≥ .70 acceptable; ≥ .80 good |
| Automated | Precision, Recall, F1, and Accuracy against a hand-coded gold standard | F1 ≥ .70 acceptable for most constructs; higher for high-stakes claims |
Report reliability per category, not as an aggregate — categories often differ substantially in difficulty.
Example (manual content analysis):
“We conducted a manual content analysis of N = 800 Instagram posts collected via the CrowdTangle API between January and March 2025. The unit of analysis was the individual post (image + caption). Two trained coders independently coded all posts using a codebook developed through three iterative pilot rounds (see Appendix B). Twenty percent of the sample (n = 160) was double-coded to assess intercoder reliability. Krippendorff’s α was .82 for frame type, .76 for affective tone, and .91 for the presence of a call to action — all above the conventional threshold of .70 (Krippendorff, 2018). Disagreements on the remaining 80% were resolved through discussion between the two coders.”
Example (automated content analysis):
“Posts were classified for the presence of misinformation cues using a fine-tuned RoBERTa-base model. A gold-standard validation set of n = 500 posts was hand-coded by two annotators (κ = .81). The model achieved precision = .84, recall = .79, F1 = .81 on this validation set, indicating acceptable performance for downstream analysis. The training data, prompts, and validation set are available at [OSF link].”
12.2.6 Reporting checklist
12.3 Qualitative research
A qualitative method chapter is structured around six subsections. Every choice — from the framework to the coding approach — must be supported by methodological literature with proper APA citations. Avoid the common mistake of citing only your substantive literature; you also need to ground your methods (Braun & Clarke, Charmaz, Lincoln & Guba, etc.).
12.3.1 Methodology
State and justify the qualitative framework you used. The most common in Communication Science MA theses are:
| Framework | When to use | Key reference |
|---|---|---|
| Thematic analysis | Identifying patterns across cases; flexible, not tied to one epistemology | Braun & Clarke (2006, 2022) |
| Grounded theory | Building theory inductively from data, with no strong prior framework | Charmaz (2014); Strauss & Corbin (1998) |
| Discourse analysis | Studying how language constructs meaning, identity, or power | Potter & Wetherell (1987); Fairclough (2003) |
| Phenomenology | Exploring lived experience of a phenomenon | Smith et al. (2009) |
| Case study | In-depth examination of a single case (org, event, platform) bounded in time/space | Yin (2018); Stake (1995) |
State your epistemological position briefly (e.g., constructivist, critical realist, interpretivist) if it shapes your analytical choices. For case studies, also justify the case: why this case, why now, why it can speak to your RQ.
12.3.2 Participants and sampling
- Recruitment: how and where participants were recruited (e.g., snowballing via networks, social media advertisements, organisational gatekeepers)
- Sampling strategy: name and justify it — purposive (selected for relevant characteristics), theoretical (selected to develop emerging concepts in GT), maximum variation (diverse cases), or snowball
- Inclusion/exclusion criteria: explicit criteria participants had to meet
- Sample size and saturation: state the final N and how saturation was assessed. Typical MA ranges: 10–20 in-depth interviews, 4–6 focus groups, or one detailed case
- Demographics: provide a participant table in the appendix (pseudonymised ID, age range, gender, relevant background) and summarise key characteristics in the text
Sample size in qualitative research is not justified by power but by theoretical or thematic saturation — the point at which new data no longer yields new themes or categories. State explicitly when and how you judged saturation to have been reached.
12.3.3 Data collection
Describe the method of data collection in concrete, replicable detail:
- Method: semi-structured interviews, focus groups, participant observation, document analysis, AI-conducted interviews (an emerging option — see Masur’s wiki on AI as interviewer for considerations)
- Setting: where the data were collected (online via Zoom/Teams, in-person at participant’s home/office, public space)
- Topic guide / interview protocol: a brief description in text and the full guide in the appendix. Note how it was developed (literature + pilot interviews)
- Duration: average length and range (e.g., “Interviews lasted between 45 and 80 minutes, M = 62 min”)
- Recording and transcription: how data were recorded (audio/video) and transcribed (verbatim, intelligent verbatim, automated + corrected). Note any software used (e.g., Whisper, Otter.ai, Amberscript)
- Language: the language interviews were conducted in, and whether/how quotes were translated for the thesis
12.3.4 Data analysis
Describe the analytical procedure step by step. The level of detail should allow another researcher to follow your reasoning.
Thematic analysis (Braun & Clarke)
Report the six phases:
- Familiarisation with the data (reading, re-reading)
- Generating initial codes
- Searching for themes
- Reviewing themes
- Defining and naming themes
- Producing the report
Specify whether coding was inductive (codes emerge from data) or deductive (codes derived from theory), or a hybrid approach.
Grounded theory
Report the coding stages:
- Open coding: line-by-line generation of initial codes
- Axial coding: grouping codes into categories and identifying relationships
- Selective coding: identifying the core category and integrating around it
Also report use of memos, constant comparison, and theoretical sampling — these are not optional features in GT.
Software
State whether coding was done manually or with software (NVivo, Atlas.ti, MAXQDA, Taguette, Dedoose). Software does not do the analysis — it organises it. State which functions you used (coding, queries, code co-occurrence, visualisations).
12.3.5 Trustworthiness and reflexivity
Qualitative work requires explicit discussion of quality criteria. The standard framework is Lincoln & Guba (1985):
| Criterion | Quantitative analogue | How to address it |
|---|---|---|
| Credibility | Internal validity | Member checking, triangulation, prolonged engagement, peer debriefing |
| Transferability | External validity | Thick description so the reader can judge applicability |
| Dependability | Reliability | Audit trail; detailed documentation of analytical decisions |
| Confirmability | Objectivity | Reflexivity; making researcher influence transparent |
Include a brief reflexivity statement: who you are in relation to the topic, what assumptions you brought to the analysis, and how this might have shaped your interpretation. This is not optional in most qualitative traditions.
12.3.6 Ethical considerations
Qualitative data are often intimate or identifiable. State explicitly:
- Ethics approval (committee, reference number)
- Informed consent procedure (oral or written, recorded or signed)
- Anonymisation and pseudonymisation (when and how)
- Data storage and retention policies
- Right to withdraw and how it was communicated
Example:
“We conducted semi-structured interviews with 17 young adults (aged 18–28) about their experiences with algorithmic content curation on TikTok. Participants were recruited via purposive sampling on Instagram and through university networks; inclusion criteria were daily TikTok use and willingness to discuss their feed. Interviews were conducted in Dutch via Zoom between February and April 2025, lasted on average 58 minutes (range: 41–82), and were audio-recorded with participant consent. Recordings were transcribed verbatim using Amberscript and manually corrected. Data were analysed following Braun and Clarke’s (2022) six-phase reflexive thematic analysis, with a hybrid inductive/deductive coding approach in Atlas.ti 23. Two researchers independently coded the first three interviews and discussed discrepancies before continuing individually. Quotes used in the results were translated to English and pseudonymised (R01–R17). The study was approved by the VU Amsterdam Ethics Committee (ref. 2025-CW-0142).”