AI-powered content analysis: Using ChatGPT to measure media and communication content

Methods tutorial #28835, module (political) communication research methods, Winter term 2023/2024

Marko Bachl

Freie Universität Berlin

2023-10-25

Hello again

Agenda

  1. Orga: Short presentations

  2. Refresher: Traditional content analysis

  3. Workshop: Validity, reliability, reproducibility, robustness, replicability

  4. Questions

Orga: Short presentations

Orga: Short presentations

  • Current status: Blackboard

  • Overview:

    • Current work about LLM-based zero-shot classification
    • One paper presented by two participants
    • Short presentations (15 Minutes)
    • Not a detailed description, but a summary for the class:
        1. what kind of questions and studies might be interesting
        1. which texts might be worth reading once they have decided on a study idea.

Refresher

Refresher: Content analysis workflow

(Neuendorf, 2017)
  • Computational content analysis with machine learning, fine tuning:
    • Train and evaluate classifier
    • Classify study materials: Predict labels
    • Analyze and report

Questions?

Workshop: Validity, reliability, reproducibility, robustness, replicability

Definitions

Measurement

  • Validity: Measurement validity: measures what it is supposed to measure; corresponds to some external truth

  • Reliability: Repeated measurements taken from the same data yield similar results; (by different coders: intercoder r.; same coders: intracoder r.)

Results

  • Reproducibility: Same results based on the same data and methods

  • Robustness: Same conclusions when using different methods but the same data

  • Replicability: Same conclusions based on the same methods but different data

Workshop: Instruction

  • Three groups:

    1. Validity
    2. Reliability
    3. Reproducibility, robustness, replicability

  • What does the criterion mean in an evaluation study of zero-shot classification?

  • How can it be assessed?

  • How does it relate to the other criteria?

  • 25 minutes work and discussion, 15 minutes presentation

Workshop: 25 minutes

Questions

  • What does the criterion mean in an evaluation study of zero-shot classification?

  • How can it be assessed?

  • How does it relate to the other criteria?

25:00

Workshop: Presentations

Questions?

Thank you — see you next week

Marko Bachl

References

Neuendorf, K. A. (2017). The content analysis guidebook. SAGE Publications, Inc. https://doi.org/10.4135/9781071802878