01
Research question
When an AI detector labels a published article, how closely does that label correspond to the article’s documented production workflow and reader-relevant provenance?
The study does not assume that a detector is useless. It distinguishes two claims: that a tool can recognize patterns associated with machine-produced language, and that the label presented to readers accurately describes how a particular work was authored.
02
Unit of analysis
The unit of analysis is one published post, not one writer. Every included post receives its own frozen text, workflow attestation, detector run, and review record.
A writer’s normal practice may not describe a particular article. The study separates participant-level background from post-level evidence.
03
Cohort and recruitment
Recruitment begins with a small trusted cohort: known contacts and writers who have publicly disclosed their use or non-use of AI. This reduces the central limitation that workflow attestations cannot be independently verified in full, but it does not eliminate it.
The initial target described for the project is approximately 10 to 20 writers contributing 100 to 150 posts. The study proceeds only if the recruitment gate produces a credible cohort.
04
Four connected records
- Frozen article text. A plain-text copy, timestamp, word count, and checksum preserve what was analyzed.
- Workflow attestation. Tools, stages, estimated generative share, editorial decisions, and supporting evidence are recorded for the post.
- Pangram run. Interface or version, date, label, score, flagged segments, screenshot, and notes are retained.
- Review. A reviewer records distortion score, failure mode, qualitative notes, and inclusion recommendation.
05
Analysis
Working exports use coded participant and post identifiers. Detector outputs are compared with attested workflows across study strata. Representative cases may be discussed qualitatively only within each contributor’s permission choices.
Contact records, working research records, evidence manifests, and publication-safe data are kept as separate views or exports.
06
Known limitations
- Attestation is the operational ground truth but cannot be fully verified.
- Canonical URLs may identify authors even when analysis tables use coded IDs.
- The invitation-led cohort is not representative of all Substack writers.
- Detector behavior may change across product versions or interfaces.
- Workflow categories simplify practices that may be iterative or ambiguous.
These are conditions of interpretation, not details to be hidden after results are known.