Skip to content

Reconciliation

Structured workflow for resolving annotation disagreements between multiple annotators to produce a single authoritative "gold standard" answer set per study.

Overview

When a systematic review project assigns multiple annotators to the same studies, those annotators will sometimes disagree. Reconciliation is the process by which these disagreements are resolved to produce a single, auditable, authoritative answer for every question on every study.

The reconciliation feature produces a versioned gold-standard record per study -- the ReconciliationSession -- that accumulates across stages. Each time a stage's annotations are reconciled, the system creates a new ReconciliationSessionVersion (RSV) that carries forward all previously reconciled answers and adds the newly reconciled ones. The latest RSV is always the complete, self-sufficient authoritative answer set for that study.

This feature is the core value proposition of the annotation management initiative. Question versioning, the annotation form, and auto-save all exist to make reconciliation possible, traceable, and trustworthy.

Problem Statement

SyRF's current reconciliation support is minimal: a Reconciled: boolean flag on individual annotations and a Reconciliation: boolean flag on sessions. There is no structured workflow, no conflict detection, no agreement metrics, no assignment mechanism, and no separate authority record. The Reconciled flag approach has fundamental problems:

  1. Duplication -- In most scenarios (single annotator, candidates agree, reconciler agrees with a candidate), the system would need to create reconciled annotations that are identical copies of existing annotations.
  2. Authority ambiguity -- A boolean flag cannot distinguish "auto-promoted because it was the only annotation" from "a reconciler reviewed this and approved it."
  3. No audit trail -- There is no record of who reconciled, when, why they chose a particular answer, or what the candidates' answers were.
  4. No metrics -- No percent agreement, no Cohen's kappa, no aggregate statistics.
  5. No workflow -- No way to assign reconciliation tasks, track progress, or enforce independent review.

Production database verification (Feb 2026) confirmed zero reconciliation sessions and zero reconciled annotations exist. The feature was partially implemented in the backend but never exposed in the UI and never used. This means the reconciliation model can be built on a clean slate.

Solution

Replace the Reconciled flag with a ReconciliationSession -- a versioned gold-standard record per study. Authority is determined automatically based on completed annotation session counts. Studies needing human review enter a reconciliation pool and are randomly assigned to reconcilers, who create their own independent annotations after reviewing blinded candidate answers side-by-side.

This approach:

  • Produces a complete, self-sufficient gold standard per study (latest RSV)
  • Provides full audit trail (version history shows how the gold standard evolved)
  • Computes agreement metrics automatically (Percent Agreement, Cohen's Kappa)
  • Prevents bias through random assignment and candidate blinding
  • Handles cross-stage question overlap naturally through RSV versioning

Scope

In Scope

  • ReconciliationSession entity with append-only ReconciliationSessionVersions
  • Authority determination rules (auto-promote vs reconciliation pool)
  • Random assignment of studies to reconcilers from the unresolved pool
  • Blinded candidate presentation (Annotator A, Annotator B)
  • Reconciler creates own annotation for every question (even when agreeing)
  • Bulk approve for studies where all candidates agree
  • Cross-stage RSV scope (required questions = current stage QSV + prior RSV questions)
  • Cross-stage visibility configuration (Blind, ShowOwnPrior, ShowReconciled)
  • Agreement metrics (Percent Agreement, Cohen's Kappa)
  • Reconciliation dashboard with pool management
  • Per-answer optional rationale (admin-configurable to require per stage)
  • Screening reconciliation ordering constraint (screening reconciliation must complete before annotation reconciliation)

Out of Scope / Future

  • Reconciliation bypass -- Every multi-annotator study goes through reconciliation (or bulk approve if unanimous). Bulk approve already handles the efficiency concern without removing reconciler oversight. Bypass may be considered as a future enhancement if demand arises.
  • Claim-based assignment -- Replaced by random assignment to prevent cherry-picking (D9).
  • Automatic conflict resolution (majority vote) -- Undermines reconciler judgement; methodologically unsound.
  • Decimal comparison tolerance -- Exact match for now; tolerance can be added later.
  • Reconciliation re-reconciliation workflow -- When a stage needs re-reconciliation (e.g., new candidate data), handling of previously carried-forward answers is deferred.
  • Cross-project aggregation -- Depends on sharing and reconciliation being stable.
  • Integrated reconciler workbench -- A unified UI combining screening and annotation reconciliation is a future UX improvement.

Key Concepts

Concept Definition
ReconciliationSession A per-study entity that holds the versioned gold standard. Each study has at most one ReconciliationSession.
ReconciliationSessionVersion (RSV) An immutable snapshot of the complete authoritative answer set for a study. Each stage's reconciliation adds a new RSV. The latest RSV is the current gold standard.
Authority determination The automatic process by which the system classifies a study as auto-promoted (single annotator) or requiring reconciliation (multiple annotators).
Auto-promotion When a study has exactly one completed annotation session and MinAnnotators=1, the system automatically creates an RSV with SingleAnnotator resolution. No human reconciliation needed.
Reconciliation pool The set of studies with 2+ completed annotation sessions that have not yet been reconciled for a given stage.
Reconciler A project member with the Reconcile permission for a stage. Reconcilers resolve disagreements by creating their own independent annotations.
Candidate annotations The annotations created by individual annotators during their annotation sessions. Identity: (StudyId, AnnotatorId, QuestionId).
Reconciliation annotations Special annotations with annotatorId: null and isReconciliation: true. Identity: (StudyId, QuestionId). Shared across stages and reconcilers. Authorship is tracked per-AV via committedBy.
Blinded comparison During reconciliation, candidate answers are presented anonymously (e.g., "Annotator A", "Annotator B") so the reconciler judges answers on their merits, not on who provided them.
Agreement metrics Quantitative measures of inter-rater agreement: Percent Agreement (all question types) and Cohen's Kappa (categorical questions only).
Resolution How a particular answer was determined: SingleAnnotator, CandidateAgreement, or ManualReconciliation.

Authority Determination

Authority is determined per stage when a study reaches sufficient annotation coverage. The outcome is a new version of the study's ReconciliationSession.

Single-Annotator Studies (Auto-Promote)

When a study has exactly one completed annotation session and the stage's MinAnnotators is set to 1:

  • The system auto-creates a ReconciliationSessionVersion.
  • Resolution for each question is set to SingleAnnotator.
  • The RSV's AnnotationAVMap points to the candidate's existing AnnotationVersionIds (no new annotations created).
  • The candidate's answers become the gold standard without human reconciliation.

This is the expected path for projects where each study is annotated by a single person.

Multi-Annotator Studies (Pool Entry)

When a study has 2 or more completed annotation sessions (regardless of the MinAnnotators setting):

  • The study enters the reconciliation pool for that stage automatically.
  • A reconciler must review and resolve the study before it has an authoritative answer set.

MinAnnotators as Readiness Threshold

MinAnnotators controls when a study is eligible for authority determination, not what kind of determination occurs. The actual completed session count determines the pathway:

  • Exactly 1 completed session meeting MinAnnotators = auto-promote.
  • 2+ completed sessions = reconciliation always required, even if MinAnnotators was 1.

How can a study have more sessions than MinAnnotators? Admins assign annotators to stages, not individual studies. Multiple annotators working on the same stage will independently annotate the same studies. The system does not prevent this -- it is expected and handled by the reconciliation pathway.

Decision Flow

CompletedSessions == 0
  --> Not ready. No action.

CompletedSessions == 1, MinAnnotators == 1
  --> Auto-promote (SingleAnnotator).
  --> System creates RSV with candidate's AVs as gold standard.

CompletedSessions == 1, MinAnnotators > 1
  --> Not ready. Needs more annotators.

CompletedSessions >= 2
  --> Reconciliation required (regardless of MinAnnotators value).
  --> Study enters the reconciliation pool for this stage.

User Workflows

Reconciler Workflow

The reconciler workflow follows a structured, bias-resistant process:

  1. Open reconciliation dashboard -- The reconciler navigates to the reconciliation page for a stage.
  2. Receive random assignment -- The system assigns a study from the unresolved pool. The reconciler cannot browse, search, or select studies. This prevents cherry-picking and ensures fair workload distribution.
  3. Review blinded candidates -- For each question in the stage's Question Set Version:
  4. Candidate answers are displayed side-by-side, labelled "Annotator A", "Annotator B", etc.
  5. The reconciler never knows which annotator provided which answer.
  6. Questions where all candidates agree are visually grouped separately from questions where they disagree.
  7. See cross-stage context -- For questions that have been reconciled in previous stages (present in a prior RSV), the existing gold-standard answer is displayed as read-only context (subject to the stage's cross-stage visibility setting).
  8. Create own annotations -- For each required question, the reconciler enters their own answer. Even when agreeing with a candidate, the reconciler records their own deliberate answer. Optionally, the reconciler adds a rationale explaining their decision.
  9. Submit -- The system creates:
  10. New AnnotationVersions (AVs) on the study's reconciliation annotations (one per question, with annotatorId: null, committedBy: reconcilerId, stageId: current stage).
  11. A new ReconciliationSessionVersion carrying forward all prior answers and adding the newly reconciled ones.
  12. Resolution metadata per question set to ManualReconciliation.
  13. Next study -- The system presents the next randomly-assigned study from the pool. The pool shrinks as studies are resolved.

Bulk Approve (PA Workflow)

For studies where all candidates agree on every answer, the project administrator can approve them in bulk without manual review:

  1. Open reconciliation dashboard -- The PA sees a breakdown: X studies need manual reconciliation, Y studies are eligible for bulk approve.
  2. Review bulk-approve candidates -- Click "Bulk Approve" to see the list of unanimous studies.
  3. Confirm -- Click "Approve All". For each study, the system:
  4. Creates reconciliation annotations on the reconciler's behalf (copying the agreed answer, attributed to the reconciler).
  5. Creates a ReconciliationSessionVersion with CandidateAgreement resolution for each question.
  6. Track progress -- The dashboard updates to reflect the reduced pool.

Cross-Stage Scope

The reconciliation scope for each study is not limited to the current stage's questions. It encompasses:

  • Required questions: All questions in the current stage's active QSV.
  • Carried-forward questions: All questions already present in the study's prior RSV (from reconciliation of earlier stages).
  • Optional questions: All other project questions not in either category above -- available for the reconciler to answer if they choose.

This means each stage's reconciliation is another version of the same session, so that eventually all questions belonging to all stages will have reconciled answers and the full context of previous answers is available to reconcilers.

For questions that appear in multiple stages (structurally forced by the parent integrity rule on QSVs):

  • Stage A reconciler answers question Q1, creating an AV on Q1's reconciliation annotation. RSV v1 maps this.
  • Stage B reconciler sees v1's answer for Q1 as context, alongside Stage B's candidate annotations for Q1.
  • Stage B reconciler provides their own answer for Q1 (it is required because Q1 is in Stage B's QSV).
  • A new AV is created on Q1's reconciliation annotation with committedBy: Stage B reconciler, stageId: Stage B.
  • RSV v2 maps the updated AV for Q1.

Cross-stage disagreement is resolved through the natural act of reconciliation -- no separate conflict detection or resolution workflow is needed. The later-stage reconciler sees the previous answer, considers the candidates, and makes a deliberate decision.

Cross-Stage Visibility Configuration

When a question has been reconciled in a previous stage, the admin configures how that answer is shown in subsequent stages:

Setting Annotators Reconcilers
Blind See nothing from other stages See nothing from other stages
ShowOwnPrior See only their own prior answers from other stages See only their own prior answers
ShowReconciled Remain blinded (no cross-stage answers) See the reconciled answer from prior stages as context

Default: ShowReconciled for reconcilers (maximum context for informed decisions), Blind for annotators (preserves independent assessment). This is configured per stage by the project admin.

Reconciliation Annotations

Reconciliation annotations differ from candidate annotations in a fundamental way: they are shared across stages and reconcilers.

Aspect Candidate Annotation Reconciliation Annotation
Identity (StudyId, AnnotatorId, QuestionId) -- personal (StudyId, QuestionId) with annotatorId: null -- shared
Who contributes One annotator Any reconciler, via AVs
Shared across stages Yes (same annotator's answer) Yes (the gold standard)
Shared across reconcilers No Yes
annotatorId Set (the annotator) null (authorship tracked per-AV via committedBy)
isReconciliation false true

For each (StudyId, QuestionId) pair, there is at most one reconciliation Annotation. Its latest AV (pointed to by the RSV's AnnotationAVMap) is the authoritative gold-standard answer. Each AV on a reconciliation annotation records:

  • committedBy -- which reconciler created this version of the answer
  • stageId -- which stage's reconciliation produced this AV

This design unifies candidate and reconciliation answer storage. Both are AnnotationVersions on Annotation entities -- they just differ in identity scope and ownership.

Agreement Metrics

Percent Agreement

A simple, intuitive metric applicable to all question types (boolean, select, checklist, text, numeric, autocomplete).

Definition: The proportion of questions (or studies) where all annotators gave the same answer.

Calculation:

PercentAgreement = (agreed_items / total_items) x 100

Agreement rules by question type:

Question Type Agreement Rule
boolean Exact match
integer Exact match
decimal Exact match (no tolerance for MVP)
string Exact match (case-insensitive, trimmed)
boolean[] Set equality (order-independent)
string[] Set equality (order-independent)
integer[] Set equality (order-independent)

Cohen's Kappa

A statistical measure that accounts for chance agreement. Only meaningful for categorical questions (boolean, select, checklist). Returns null for free-text and numeric questions.

Definition:

kappa = (Po - Pe) / (1 - Pe)

where:
  Po = observed agreement proportion
  Pe = expected agreement by chance

Cohen's Kappa is more informative than Percent Agreement because it adjusts for the probability that annotators would agree by chance alone. A kappa of 1.0 indicates perfect agreement; 0.0 indicates agreement no better than chance.

Edge cases are reported clearly (e.g., when one category dominates and Pe approaches 1.0, kappa can be misleadingly low despite high observed agreement).

Metric Granularity

Level Computation
Per question, per study Do all annotators agree on this question for this study?
Per question, across stage Percent agreement and kappa across all studies with 2+ annotators
Per study Percent of questions in agreement for this study
Stage aggregate Average percent agreement across all studies

When Metrics Are Computed

  • On session completion: When a study reaches 2+ completed sessions, per-study metrics are computed.
  • On reconciliation resolution: Updated to reflect resolved state.
  • On demand: Admin can trigger recomputation for a stage.
  • Background: MassTransit consumer handles computation asynchronously.

Screening Reconciliation Ordering

When a project uses both screening and annotation stages, annotation reconciliation cannot proceed until screening reconciliation has determined that the study is included. Excluded studies are never reconciled for annotations -- this saves work and avoids meaningless reconciliation.

Candidates complete screening + annotations
  --> Screening Annotation Reconciliation
    --> Reconciler resolves screening conflicts
      --> FinalScreeningOutcome
        --> Exclude: Study excluded. No annotation reconciliation needed.
        --> Include: Study included. Check annotation sessions per stage.
          --> CompletedSessions >= MinAnnotators?
            --> Yes, 1 session, MinAnnotators==1: Auto-promote (SingleAnnotator)
            --> Yes, 2+ sessions: Study enters annotation reconciliation pool
            --> No: Not ready. Awaiting more annotation sessions.

Data Model

ReconciliationSession

The ReconciliationSession is a per-study entity that acts as the gold standard. It is a materialised decision record, not a traditional DDD aggregate -- it has no lifecycle or state transitions. Each version is an immutable fact.

ReconciliationSession (per Study)
  CurrentVersionNumber: int
  Versions: List<ReconciliationSessionVersion>    (append-only)

ReconciliationSessionVersion (Immutable)
  Id: Guid                            (globally unique)
  VersionNumber: int
  ReconciledStageId: Guid             (which stage's reconciliation produced this version)
  ReconcilerId: Guid                  (who reconciled)
  QSVId: Guid                         (the stage's active QSV -- audit)
  AnnotationAVMap: Map<AnnotationId, AVId>   (reconciliation Annotation --> authoritative AV)
  ResolutionMetadata: Map<QuestionId, ResolutionMeta>  (per-question resolution tracking)
  SubmittedAt: DateTime
  ChangeReason: string?

ResolutionMeta (per-question metadata)
  Resolution: Resolution              (how this answer was determined)
  Rationale: string?                  (optional per-answer rationale)

Resolution Enum

Value Meaning AV committed by
SingleAnnotator Only one annotator completed; auto-promoted System (on behalf of the candidate)
CandidateAgreement Multiple annotators agreed; reconciler bulk-approved Reconciler (creating own AV with agreed answer)
ManualReconciliation Annotators disagreed; reconciler manually resolved Reconciler (creating own AV with their answer)

How RSV Versioning Works

When a reconciler reconciles Stage B for a study that already has RSV v1 (from Stage A):

  1. Load v1's AnnotationAVMap (the current gold standard).
  2. Stage B's QSV questions are required -- reconciler must provide answers.
  3. All other project questions are optional -- previous answers shown as context.
  4. Reconciler submits. New AVs are created on reconciliation Annotations for Stage B questions, with committedBy: reconcilerId and stageId: Stage B.
  5. Carried-forward entries from v1 retain their existing AVs (no new AV created for unchanged answers).
  6. Write v2 with the complete merged AnnotationAVMap.

The latest version's AnnotationAVMap is always a complete snapshot of the gold standard -- no need to traverse version history.

Key Properties

  • Complete snapshot: Each version contains ALL authoritative answers (newly reconciled + carried forward). The latest version is self-sufficient.
  • Audit trail: Version history shows exactly how the gold standard evolved -- which stage, which reconciler, what changed at each step.
  • Per-answer attribution: Each reconciliation AV records committedBy and stageId, preserving full provenance even when answers are carried forward across versions.
  • Concurrent stage reconciliation: Handled by optimistic concurrency. If two reconcilers submit for the same study simultaneously, the second writer detects a version mismatch, reloads, merges (stage questions are mostly disjoint), and retries. This is rare in practice.

Key Rules

  • One ReconciliationSession per study for the entire project.
  • Once a reconciliation AV exists for a question, no further candidate sessions are accepted for that study/question in the originating stage.
  • Resolution metadata (Resolution, Rationale) lives on ResolutionMeta in the RSV, never on the annotation itself. Annotations are pure data; the ReconciliationSession carries authority and resolution tracking.
  • The Rationale field defaults to optional. Admins can configure it as required per stage (MVP), with potential per-question configurability in the future.

API

Pool Assignment

GET /api/projects/{projectId}/stages/{stageId}/reconciliation/next
  --> 200 OK { study, candidateAnnotations (blinded), priorRSV (if exists) }
  --> 204 No Content (pool empty)

System randomly selects an unresolved study from the pool and returns it with blinded candidate data. The reconciler does not choose.

Reconciliation Submission

POST /api/projects/{projectId}/stages/{stageId}/reconciliation/submit
Body: {
  studyId,
  answers: [
    { questionId, answer, rationale? }
  ]
}
  --> 200 OK { rsvId, nextStudy? }

Creates reconciliation annotations (AVs), writes a new RSV, and optionally returns the next study.

Bulk Approve

POST /api/projects/{projectId}/stages/{stageId}/reconciliation/bulk-approve
Body: {
  studyIds: [...]    // Studies where all candidates agree
}
  --> 200 OK { approvedCount, rsvIds }

For each study, creates reconciliation annotations on the reconciler's behalf and writes RSVs with CandidateAgreement resolution.

Metrics

GET /api/projects/{projectId}/stages/{stageId}/agreement-metrics
  --> 200 OK {
    overall: { percentAgreement, cohensKappa, totalStudies, studiesWithMultipleAnnotators, studiesInPerfectAgreement },
    perQuestion: [ { questionId, questionText, percentAgreement, cohensKappa, studiesCompared } ]
  }

GET /api/projects/{projectId}/stages/{stageId}/agreement-metrics/export
Accept: text/csv
  --> 200 OK (CSV file)

Dashboard

GET /api/projects/{projectId}/stages/{stageId}/reconciliation/dashboard
  --> 200 OK {
    poolSize,
    bulkApproveEligible,
    manualReconciliationRequired,
    resolved,
    agreementSummary: { distribution of agreement levels },
    perQuestionAgreement: [...]
  }

Permissions

Action Required Role
View dashboard Project Member
Receive assignment (reconcile) Reconciler or Admin
Submit reconciliation Reconciler or Admin
Bulk approve Reconciler or Admin
View metrics Project Member
Export metrics Project Member
Configure reconciliation settings Admin

Migration Strategy

Migration is additive only. No existing data is deleted or moved.

Phase 6: Reconciliation Model

  • ReconciliationSession entity created on studies during authority determination.
  • Authority determination rules implemented.

Phase 7: Data Migration Backfill

  • For every study with exactly 1 completed session per stage where MinAnnotators == 1:
  • Create a ReconciliationSessionVersion with Resolution = SingleAnnotator.
  • RSV's AnnotationAVMap points to the candidate's (now v1) AnnotationVersionIds.
  • Studies with 0 completed sessions: no action (not ready).
  • Studies with 2+ completed sessions: enter the reconciliation pool (no auto-backfill -- reconciler must review).

Phase 8: Reconciliation Workflow

  • Full reconciliation UI, random assignment, blinded comparison, bulk approve, metrics, dashboard.
  • Feature flag controls visibility: enabled per project.

Backward Compatibility

  • The Reconciled boolean flag is preserved on existing annotations but not used for authority determination. It is replaced by the ReconciliationSession model.
  • Existing annotation forms continue to work throughout migration.
  • API consumers see the same structures with additional optional fields.
  • Rollback is straightforward: $unset new fields and remove ReconciliationSession documents.

Success Criteria

Requirements mapped to RECON-01 through RECON-17:

  1. RECON-01: ReconciliationSession exists as the authority mechanism -- a versioned gold standard per study.
  2. RECON-02: ReconciliationSessionVersion contains a complete answer snapshot (newly reconciled + carried forward from prior versions).
  3. RECON-03: Single-annotator studies are auto-promoted when 1 completed session meets MinAnnotators=1. System creates RSV with SingleAnnotator resolution.
  4. RECON-04: Studies with 2+ completed sessions automatically enter the reconciliation pool for that stage.
  5. RECON-05: MinAnnotators acts as a readiness threshold only. It does not determine the type of authority determination -- the completed session count does.
  6. RECON-06: Reconciler is randomly assigned a study from the unresolved pool. No browsing, searching, or claiming.
  7. RECON-07: Candidate blinding is a system invariant enforced at the API level. Annotators never see other candidates' answers during annotation or after completing their session.
  8. RECON-08: Reconciler always creates their own annotation when reconciliation occurs, even when agreeing with a candidate's answer.
  9. RECON-09: Bulk approve creates reconciliation annotations on the reconciler's behalf for studies where all candidates agree. Resolution is set to CandidateAgreement.
  10. RECON-10: Per-study reconciliation scope includes the current stage's QSV questions (required) plus questions already in the study's prior RSV (context/carried forward).
  11. RECON-11: Cross-stage visibility is admin-configurable per stage with three options: Blind, ShowOwnPrior, ShowReconciled.
  12. RECON-12: Candidate annotations are presented anonymously during reconciliation (Annotator A, Annotator B). The reconciler does not know which annotator gave which answer.
  13. RECON-13: Percent Agreement metric is computed per question, per study, and per stage for all question types.
  14. RECON-14: Cohen's Kappa metric is computed for categorical questions (boolean, select, checklist). Returns null for free-text and numeric questions. Handles edge cases clearly.
  15. RECON-15: Reconciliation dashboard displays pool size, agreement summary, bulk approve vs manual split, and progress tracking.
  16. RECON-16: Reconciliation form displays side-by-side candidate comparison with required questions (current stage QSV) and optional questions (prior RSV, other project questions).
  17. RECON-17: Per-answer rationale field is optional by default. Admins can configure it as required per stage.
Document Purpose
Design Decisions Authoritative reference -- D5 (ReconciliationSession), D7 (reconciler creates own annotation), D8 (MinAnnotators threshold), D9 (random assignment), D10 (candidate blinding), D15-D18 (RSV model), D19-D28 (authority and metrics), D45 (reconciliation annotations with null annotatorId), D50 (cross-stage RSV scope)
Annotation Versioning Design D45 (reconciliation annotation identity), D49-D50 (reconciliation annotations shared across stages and reconcilers)
Product Overview PO-facing description of reconciliation workflows and phased delivery
Annotation Form v2 Form rendering for the reconciliation comparison view
Question Management Question versioning that reconciliation depends on for meaningful comparison
Screening Annotations Screening reconciliation -- separate process with aligned patterns (shared pool management, random assignment, bulk approve)
Requirements RECON-01 through RECON-17 requirement definitions and phase traceability

Implementation Appendices

⚠️ Precedence note: The appendices below were merged from the closed PR #2325 branch (created 2026-02-09) to preserve implementation detail not carried forward during spec restructuring. The product-level spec above is authoritative for design intent. Where appendix content contradicts the spec above or the Design Decisions (D1–D50), the higher-precedence document wins. In particular:

  • The Consensus Model Decision below uses the original ReconciliationRecord with claim/release workflow. The authoritative model uses random assignment (D9), candidate blinding (D10), and the Reconciliation Session entity (D5) — as described in the spec above.
  • The MassTransit contracts are illustrative starting points. Event names and payloads should align with the Reconciliation Session model.
  • The Angular UI components describe target UX (dashboard, detail view, metrics) that remains valid regardless of data model changes.
  • The acceptance criteria should be used as a starting point for Phase 9–10 planning, updated to reflect the current design decisions.

Angular UI Components

Reconciliation Dashboard

New route: /projects/{projectId}/stages/{stageId}/reconciliation

  • ReconciliationDashboardComponent: Main dashboard
  • Summary cards: total, unresolved, in progress, resolved
  • Agreement metrics panel (charts)
  • Study table with sorting, filtering, pagination
  • Bulk actions: assign to reconciler

Reconciliation Detail View

New route: /projects/{projectId}/reconciliation/{id}

  • ReconciliationDetailComponent: Side-by-side conflict resolution
  • Left panel: Question list with conflict indicators
  • Center: Side-by-side annotator answers
  • Right: Consensus entry form
  • Bottom: Rationale notes
  • Keyboard shortcuts: Tab (next question), ½ (select annotator), N (new answer)

Agreement Metrics View

  • AgreementMetricsComponent: Charts and tables
  • Bar chart: per-question agreement
  • Histogram: study-level agreement distribution
  • Table: detailed per-question metrics
  • Export button

Backend Implementation Notes

MassTransit Consumers

// Detect conflicts for a study after annotation completion
public class ComputeConflictsConsumer : IConsumer<ComputeConflicts>
{
    // Triggered by: SessionCompleted event (when session count target reached)
    // Publishes: ReconciliationTaskCreated (if conflicts found)
}

// Compute aggregate metrics for a stage
public class ComputeAgreementMetricsConsumer : IConsumer<ComputeAgreementMetrics>
{
    // Triggered by: ReconciliationResolved event, or on-demand
    // Updates: cached metrics for the stage
}

MassTransit Contracts

public record ComputeConflicts(Guid ProjectId, Guid StageId, Guid StudyId, int StageConfigVersion);
public record ReconciliationTaskCreated(Guid ProjectId, Guid ReconciliationId, int ConflictCount);
public record ReconciliationResolved(Guid ProjectId, Guid ReconciliationId, Guid ResolvedBy);
public record ComputeAgreementMetrics(Guid ProjectId, Guid StageId);

Permissions

Action Required Role
View dashboard Project Member
Claim task Reconciler or Admin
Resolve task Reconciler or Admin
Assign reconciler Admin
Trigger conflict detection Admin
View metrics Project Member
Export metrics Project Member

Consensus Model Decision

Decision: Consensus answers are stored as a separate layer, not overwrites of original annotations.

Rationale:

  • Original annotations remain immutable for audit purposes
  • Can reconstruct the full history: original → conflict → consensus
  • Supports different consensus strategies (majority vote, expert override, averaged)
  • Enables metrics comparison between original and consensus

Implementation: Consensus answers are stored as annotations in Study.ExtractionInfo.Annotations with:

  • Reconciled = true (existing flag, already supported)
  • AnnotatorId set to the reconciler's ID
  • QuestionVersionNumber matching the version from the ReconciliationRecord
  • The ReconciliationRecord itself (in ExtractionInfo.ReconciliationRecords) tracks the ConsensusAnnotationId linking back to the consensus annotation

Acceptance Criteria Summary

  • Conflicts automatically detected when annotation count target reached
  • Dashboard shows all reconciliation tasks with filtering and pagination
  • Reconcilers can claim, resolve, and release tasks
  • Side-by-side comparison with quick-select actions
  • Rationale recorded per decision
  • Agreement metrics: percent agreement + Cohen's kappa
  • Metrics available per-question, per-study, and aggregate per-stage
  • CSV export for metrics
  • Full audit trail (append-only)
  • RBAC enforced: only reconcilers/admins can resolve
  • Original annotations remain immutable after reconciliation
  • Keyboard navigation in resolution UI