Phase 10: Reconciliation Workflow and UI¶

Release 2 -- The Centerpiece Phase 10 is the most user-facing feature in Release 2. This is where a reconciler sits down and resolves disagreements to produce gold-standard answers.

Summary¶

This phase delivers the complete reconciliation experience: a structured workflow where disagreements between annotators are resolved by an independent reconciler, producing auditable, authoritative answers for every study. It includes random study assignment, anonymised side-by-side comparison, bulk approval for unanimous studies, and agreement metrics (Percent Agreement and Cohen's Kappa).

Reconciliation is the core value proposition of the annotation management initiative. Everything built in previous phases -- versioned questions, the rebuilt form, per-question auto-save, authority determination -- exists to make this workflow possible, traceable, and trustworthy.

The Problem¶

When multiple researchers annotate the same study, they will sometimes disagree. Today, SyRF offers only a boolean "reconciled" flag with no structured process for resolving disagreements. This means:

No way to identify where annotators disagree (which questions, which studies)
No method for systematically resolving disagreements
No quality metrics to report to journals and funders
No audit trail showing how the final answer was determined
The "reconciled" flag has never been used in production (verified February 2026: zero reconciliation sessions exist)

Journals and funding bodies increasingly expect systematic reviews to report inter-rater agreement statistics and describe their reconciliation process. Without this capability, SyRF cannot support publication-ready systematic reviews.

What We Are Building¶

The Reconciliation Workflow¶

flowchart TD
    A[Two or more annotators complete a study] --> B{Do all annotators agree on every answer?}
    B -->|Yes| C[Eligible for bulk approve]
    B -->|No| D[Study enters reconciliation pool]
    D --> E[System randomly assigns study to a reconciler]
    E --> F[Reconciler sees anonymised answers side-by-side]
    F --> G[Reconciler records their own answer for every question]
    G --> H[Gold-standard record created]
    C --> I[Admin bulk-approves unanimous studies]
    I --> H

    style H fill:#c8e6c9

Key Design Principles¶

Random assignment, not cherry-picking: When a study needs reconciliation, it enters a pool. The system randomly assigns it to an available reconciler -- reconcilers cannot browse or select studies. This prevents bias and ensures fair workload distribution.

Anonymised comparison: The reconciler sees candidate answers labelled "Annotator A" and "Annotator B" -- never the actual identities. This ensures the reconciler judges answers on their merits, not on who provided them.

Reconciler always records their own answer: Even when the reconciler agrees with one of the candidates, they record their own independent answer. The gold standard is always the reconciler's deliberate decision, never a pointer to someone else's work. This is consistent with best practice in systematic review methodology.

Bulk approve for unanimous studies: When all annotators agree on every answer for a study, the project administrator can approve it in bulk. The system creates gold-standard records on the reconciler's behalf -- no manual review needed when there are no disagreements. This saves significant time for projects with high inter-rater agreement.

The Reconciliation Form¶

For each study assigned to a reconciler:

Disagreed questions are highlighted, with each candidate's answer displayed side-by-side
Agreed questions are grouped separately (candidates gave the same answer)
For every question, the reconciler enters their own answer
Optional rationale field explains why the reconciler made their decision (administrators can require this per stage)
Cross-stage context: if a question was reconciled in a previous stage, that answer is shown as read-only context

Agreement Metrics¶

Two metrics are computed automatically:

Metric	What It Measures	Applicable To
Percent Agreement	The proportion of questions where all annotators gave the same answer	All question types (yes/no, multiple choice, text, numbers)
Cohen's Kappa	A statistical measure that accounts for chance agreement -- more rigorous than simple percent agreement	Categorical questions only (yes/no, multiple choice)

These metrics are computed per question, per study, and per stage. They are visible on the reconciliation dashboard and exportable as CSV -- ready for inclusion in systematic review publications.

Reconciliation Dashboard¶

The dashboard gives project administrators a high-level view of reconciliation progress:

Pool size: How many studies still need reconciliation
Bulk approve eligible: How many studies can be approved without manual review (all annotators agreed)
Agreement summary: Distribution of agreement levels across the pool
Progress tracking: How many studies have been reconciled vs. remaining
Per-question metrics: Agreement rates broken down by individual questions -- useful for identifying questions that are commonly misunderstood

What Changes for Each Role¶

For Reconcilers (New Role)¶

Step	What They Do
1. Open reconciliation page	Navigate to the stage's reconciliation view
2. Receive assignment	System assigns a study -- no browsing
3. Review comparison	See anonymised answers side-by-side
4. Record own answers	Enter their own answer for every question
5. Optionally add rationale	Explain why they chose their answer
6. Submit	Creates the gold-standard record
7. Next study	System assigns the next study from the pool

For Project Administrators¶

View reconciliation dashboard with pool management
Bulk approve unanimous studies
Configure reconciliation settings per stage (MinAnnotators, cross-stage visibility, rationale requirements)
Review agreement metrics for quality monitoring

For Data Consumers¶

Exported data includes reconciliation status per question per study
Agreement metrics exportable as CSV for publication
Full audit trail: which question version was asked, what was answered, how it was reconciled

How It Connects¶

Connection	Detail
Phase 5 (Annotation Form v2)	The reconciliation form reuses the annotation form's rendering engine for side-by-side comparison
Phase 8 (Project Groups)	Only users with the Reconcile permission can be assigned studies
Phase 9 (Reconciliation Model)	Uses the authority determination rules to populate the reconciliation pool
Phase 11 (Release 2 Migration)	Multi-annotator studies enter the reconciliation pool during migration
Phase 15 (Screening Annotations)	Screening reconciliation follows the same random-assignment pattern

For the platform architecture overview, see platform-architecture.md. For the full reconciliation technical specification, see README.md.

Phase 8 creates the permission structure. Phase 9 builds the reconciliation data model. Phase 10 delivers the reconciliation workflow. Phase 11 migrates existing data.