Measurement
Part of Measuring a creator studio without rewarding shortcuts
Four ways to attribute studio outcomes, and how strong each conclusion is
Compare four studio attribution methods on one outcome unit, evidence need, bias, privacy burden and the strength of conclusion each permits.
Creator studio equipment attribution methods answer different questions. A tagged visit allocates an observed event. A contribution assessment tests an explanation. A counterfactual study estimates what changed because of an intervention. Calling all three "impact" destroys the distinction.
This comparison was completed on 6 September 2026. It is non-ranked and desk-only. The common outcome unit is an eligible downstream event per eligible tutorial released from the defined England workflow during the same observation window. Each method is compared on data, allocation rule, bias, privacy burden, causal ceiling and practical decision use.
What to take away
- Tagged-source allocation counts events by a declared rule, not the event's cause.
- Respondent statements reveal routes logs miss but do not isolate the equipment's effect.
- Contribution assessment tests a theory and yields a bounded judgement, not a causal percentage.
- Counterfactual designs can support causal estimates if implementation, balance and uncertainty are handled well.
- Choose the method by the decision question and never average their outputs.
Tagged-source allocation
A controlled link or campaign code associates an event with a recorded source under a declared rule. It needs stable identifiers, a fixed window, duplicate handling and a documented route from release to event. The result is a count or proportion allocated to that source, not the event's cause.
Tagged-source allocation chain
- Stable identifiers
- Fixed window
- Duplicate handling
- Documented release-to-event route
- Count or proportion allocated
Cookies, pixels, link decoration, scripts and similar collection methods require a technology-specific assessment. The ICO's current storage and access guidance explains the UK privacy and PECR boundary. Missing consent states and cross-device activity can change coverage.
Respondent source statement
Ask an eligible respondent how they first encountered the tutorial or what influenced the action, using a pre-set question and response options. Preserve "do not know", multiple influences and no response. The unit remains an eligible event, but the evidence is a person's report at a particular time. Recall, question order and social desirability can affect it.
Tagged vs respondent evidence
Tagged-source allocation
- Evidence unit
- Recorded event
- Strength
- Count or proportion
- Main bias
- Coverage gaps
- Reconciliation
- Do not average
Respondent source statement
- Evidence unit
- Person's report
- Strength
- Declared influence
- Main bias
- Recall and social desirability
- Reconciliation
- Do not reconcile mechanically
This method can reveal routes that logs miss. It still does not isolate the equipment's effect, and it should not be reconciled mechanically with tagged allocation.
Contribution assessment
Set out the proposed chain from equipment change to production behaviour and then to the defined event. Examine whether expected steps occurred, test rival explanations and record contrary evidence. HM Treasury's analytical annex describes contribution analysis as an evidenced line of reasoning rather than definitive proof.
The output is a bounded judgement with assumptions and gaps. It can guide investigation where experimental allocation is impractical, but should not be converted into a percentage of causal credit.
Counterfactual impact design
Pre-specify the intervention, eligible units, outcome, comparison, assignment, period and analysis. A credible control or comparison estimates what would probably have happened without the equipment change. HM Treasury's QPIE record says monitoring change alone cannot establish causation and links stronger attribution to a strong counterfactual.
Randomised or suitable quasi-experimental designs can support a causal estimate if implementation, balance, interference, missing data and uncertainty are handled well. Feasibility and proportionality still matter.
The causal estimate uses the pre-specified outcome unit for intervention and comparison groups. Report it with uncertainty, departures from the plan and limits on which studios or outputs the conclusion can cover. A small or inconclusive estimate should not be recast as directional proof.
Choose by the decision
Use allocation to describe recorded routes, respondent evidence to hear declared influences, contribution work to test a theory, and counterfactual evaluation for a proportionate causal question. Report the same event definition and observation window across methods, but never average their outputs. Their evidential meanings are not interchangeable.
Record the choice before viewing results. The decision note should name the question, eligible tutorials, event, method, owner, privacy basis, known bias, review date and wording ceiling. If the tagged and self-reported routes disagree, preserve both and investigate coverage. Do not choose whichever assignment produces the larger number.
None of the four methods measures device safety, audio quality or audience welfare. Those questions require their own tests and qualified reviewers. Attribution evidence can inform a studio decision, but it cannot compensate for a failed operational or legal gate.
Before you act
- Define the same event and observation window across methods.
- Record the method choice before viewing results.
- Preserve do not know and multiple influences in respondent data.
- Do not convert contribution assessment into causal credit.
- Report causal estimates with uncertainty and plan departures.
- Do not average outputs from different attribution methods.
Common questions
What does a tagged-source allocation actually measure?
It associates an event with a recorded source under a declared rule. The result is a count or proportion allocated to that source, not the event's cause. It needs stable identifiers, a fixed window, duplicate handling and a documented route from release to event.
When is a counterfactual impact design appropriate?
It is appropriate for a proportionate causal question. Pre-specify the intervention, eligible units, outcome, comparison, assignment, period and analysis. A credible control or comparison estimates what would probably have happened without the equipment change. Randomised or suitable quasi-experimental designs can support a causal estimate if implementation, balance, interference, missing data and uncertainty are handled well.
How should disagreement between tagged and self-reported routes be handled?
Preserve both and investigate coverage. Do not choose whichever assignment produces the larger number. The article states that if the tagged and self-reported routes disagree, preserve both and investigate coverage. It also warns against reconciling them mechanically.
