Annotation Management & Reconciliation — Product Specification¶

Audience: Product owners, project leads, and stakeholders. This document describes what we are building, why, who it affects, and how it will be delivered. For engineering details, see the technical specs and design decisions.

Executive Summary¶

SyRF's annotation system is being upgraded to produce reconciled, authoritative annotation data per study. This means that for every study in a systematic review, there will be a single, auditable, gold-standard answer set that records what was decided and how.

This requires four interconnected capabilities delivered across nine phases:

Question Management Platform — Versioned questions with a draft-to-activation lifecycle, so projects can evolve their annotation protocol without breaking past data
Annotation Form v2 — A rebuilt, performant annotation form that auto-saves per question and tracks exactly which question version each answer was collected against
Annotation Reconciliation — A structured workflow where disagreements between annotators are resolved by an independent reconciler, producing a versioned gold-standard record per study
Data Export — Enriched exports including reconciliation status, version references, and agreement metrics

Every other feature — versioning, question sets, form performance, auto-save — exists to make reconciliation possible, traceable, and trustworthy.

The Problem¶

What happens today¶

When a project administrator edits an annotation question in SyRF — changing the wording, adding an option, or adjusting the answer type — that change happens silently and retroactively. Every past annotation collected against the old version of the question now appears to have been answered against the new wording.

SyRF does snapshot the question text onto each annotation at the time it is created, but this snapshot is limited — it captures only the question text string, not the full question definition (available options, control type, help text, conditional logic, or parent-child relationships).

Because of these risks, SyRF currently locks annotation questions entirely once they have been answered. This is a deliberate safety measure, but it creates a different problem: projects are stuck with their initial question design, even when improvements are clearly needed.

Three practical consequences¶

Incomplete audit trail — Systematic reviews require traceability. While the question text is preserved per annotation, the full question context (options, answer type, conditional rules) is not. Reviewers, journals, and regulators cannot reconstruct exactly what the annotator's form looked like when they recorded their answer.
Inability to evolve questions — A team may realise midway through annotation that a question needs refining — a dropdown option is missing, wording is ambiguous, or a new question is needed. Today, answered questions are locked. Teams must work around this by creating entirely new questions or accepting suboptimal wording for the remainder of the project.
No structured reconciliation — When two annotators disagree, the current system offers only a boolean flag ("reconciled: yes/no"). There is no structured workflow, no agreement metrics, and no clear record of how the disagreement was resolved. Reconciliation is ad-hoc and impossible to audit.

What this means for systematic reviews¶

Credible systematic reviews require: - Traceability: the ability to reconstruct what question was asked, what the annotator saw, and what they answered - Reproducibility: evidence that disagreements were systematically resolved, not arbitrarily - Quality metrics: inter-rater agreement statistics (percent agreement, Cohen's kappa) that journals and funders expect

SyRF currently cannot provide any of these. This initiative addresses all three.

What We Are Building¶

1. Question Management Platform¶

Draft Questions and Activation¶

Questions now follow a two-phase lifecycle:

Draft phase: Project administrators create and edit "Draft Questions" (DQs) using the existing question management UI (~90% complete behind a feature flag). Drafts are fully mutable — text, options, help text, answer filters, parent-child relationships can all be changed freely.
Activation: When the administrator enables a stage, all draft questions assigned to that stage are "activated" — they become permanent Annotation Questions (AQs) with an initial version (v1). From this point forward, the question's structural identity (parent, data type, group-as-single) is fixed. Content properties (text, options, help text, answer filters) can still be changed, but each change creates a new version.

Question Versioning¶

What changes for users: Every time a project admin edits an activated question, SyRF creates a new version (v1, v2, v3...). Previous versions are preserved. When an annotator answers a question, SyRF records which version they were answering.

What users see:

A version badge on each question card (e.g., "v3")
Editing opens a "Save as New Version" flow — admins understand they are creating a new version, not overwriting history
A version history panel shows all past versions, who changed them, and when
Side-by-side diff view highlights exactly what changed between any two versions
Structural properties are fixed — the question's parent, data type (boolean, select, etc.), and group-as-single setting cannot change after activation. This prevents silently breaking the question's role in the hierarchy or changing its answer format

What can be versioned (content properties):

Question text (the wording annotators see)
Answer options (dropdown choices, checkbox labels)
Help text (guidance shown alongside the question)
Answer option filters (which options from a parent question's answer restrict this question's options)

What cannot change after activation (structural properties):

Parent question (hierarchical position)
Data type (boolean, select, checklist, text, numeric, autocomplete)
Group-as-single setting

Question Sets¶

Each stage in a project has a Question Set — a versioned, ordered list of specific question versions that annotators see when they work in that stage.

What users see:

The Assign tab shows version-aware assignment — when assigning questions to a stage, the admin selects which version of each question to include
A configuration history shows how the stage's question set has evolved over time
The Preview tab renders exactly what annotators will see with the current question set
Historical annotations reference the question set version that was active when they were collected

Why it matters: Projects can evolve their annotation protocol without fear. Adding questions, removing questions, or upgrading to a newer question version mid-project is safe because SyRF tracks which configuration was active at each point in time.

Project Question Library¶

A flat manifest of all question versions available to the project. This is the "pool" from which stage question sets are composed. Future versions will enable sharing questions across projects and organisations.

2. Annotation Form v2¶

SUPERSEDED: This section describes per-question auto-save but does not specify the server-side mechanics. Auto-save is implemented via a pendingAnswer field on each Annotation document — a mutable server-side buffer updated with each auto-save, cleared on Save/Complete/Revert. See annotation-versioning-design.md Section 2.5 (D46) and design-decisions.md Section 11.

The annotation form is rebuilt to resolve a known performance issue (#1125) where projects with many questions experience slow rendering and frame drops.

What changes for annotators:

Today	After
Entire form renders at once (slow with 50+ questions)	Virtual scroll — only visible questions render (smooth with 100+)
Full form must be saved at once	Per-question auto-save — each answer saves individually as you work
Submit creates one record	Submit creates an immutable session snapshot linking to specific answer versions
Navigating away loses all unsaved work	Unsaved-changes warning before navigation
No version tracking	Each answer records which question version it was collected against

Why it matters: Annotators get a faster, more reliable experience. The system gets precise per-answer versioning that reconciliation depends on.

3. Annotation Reconciliation¶

When multiple annotators have answered the same questions for the same study, SyRF detects where they disagree and assigns the study to a reconciler for resolution.

How it works¶

                                    ┌──────────────────────────┐
                                    │   Annotator A completes  │
                                    │   study annotation       │
                                    └──────────┬───────────────┘
                                               │
                                               ▼
                                    ┌──────────────────────────┐
                                    │   Annotator B completes  │
                                    │   study annotation       │
                                    └──────────┬───────────────┘
                                               │
                            ┌──────────────────┴────────────────────┐
                            │                                       │
                    All answers agree?                       Answers disagree
                            │                                       │
                            ▼                                       ▼
                ┌───────────────────────┐             ┌─────────────────────────┐
                │   Eligible for        │             │   Study enters the      │
                │   bulk approve        │             │   reconciliation pool   │
                └───────────────────────┘             └───────────┬─────────────┘
                                                                  │
                                                                  ▼
                                                      ┌─────────────────────────┐
                                                      │   System randomly       │
                                                      │   assigns study to a    │
                                                      │   reconciler            │
                                                      └───────────┬─────────────┘
                                                                  │
                                                                  ▼
                                                      ┌─────────────────────────┐
                                                      │   Reconciler sees       │
                                                      │   anonymised answers    │
                                                      │   (Annotator A vs B)    │
                                                      │   side by side          │
                                                      └───────────┬─────────────┘
                                                                  │
                                                                  ▼
                                                      ┌─────────────────────────┐
                                                      │   Reconciler creates    │
                                                      │   their OWN answer for  │
                                                      │   each question         │
                                                      └───────────┬─────────────┘
                                                                  │
                                                                  ▼
                                                      ┌─────────────────────────┐
                                                      │   Gold-standard record  │
                                                      │   created (RSV)         │
                                                      └─────────────────────────┘

Key design principles¶

Random assignment, not claiming: When a study needs reconciliation, it enters a pool. The system randomly assigns it to an available reconciler — reconcilers cannot browse or cherry-pick studies. This prevents bias and ensures fair distribution.

Candidate blinding: The reconciler sees candidate answers labelled "Annotator A" and "Annotator B" — never the actual identities. This ensures the reconciler judges answers on their merits, not on who provided them. Blinding is a system invariant enforced at the API level, not a UI preference.

Reconciler always creates their own answer: Even when the reconciler agrees with a candidate, they record their own answer. The gold standard is always the reconciler's deliberate decision, never a pointer to someone else's work. This is consistent with best practice in systematic review methodology.

Bulk approve for unanimous studies: When all candidates agree on every answer, the project administrator can bulk-approve these studies. The system creates reconciliation records on the reconciler's behalf — no manual review needed for studies with no disagreements.

Per-study reconciliation scope: Each study's reconciliation covers the current stage's question set plus any questions already in this study's prior reconciliation record. This means reconciliation accumulates over time — a study reconciled in an earlier stage carries those answers forward.

Authority determination¶

The system automatically classifies each study based on how many annotators have completed it:

Scenario	What happens
1 annotator completed, MinAnnotators = 1	Auto-promoted — the single annotator's answers become the gold standard (no reconciliation needed)
2+ annotators completed	Study enters the reconciliation pool automatically

MinAnnotators is a readiness threshold: it controls when a study is ready for reconciliation, not whether reconciliation is needed. A study with 2 completed sessions always requires reconciliation regardless of the MinAnnotators setting.

Agreement metrics¶

Two metrics are computed automatically:

Percent Agreement — The fraction of questions where all annotators gave the same answer. Intuitive, applicable to all question types (including text and numeric). Computed per question, per study, and per stage.
Cohen's Kappa — A statistical measure that accounts for chance agreement. Only meaningful for categorical questions (boolean, select, checklist). Returns null for free-text and numeric questions. Reports edge cases clearly (e.g., when one category dominates).

These metrics are visible on the reconciliation dashboard and exportable as CSV.

Reconciliation dashboard¶

The reconciliation dashboard gives project administrators a high-level view:

Pool size — How many studies need reconciliation
Agreement summary — Distribution of agreement levels across the pool
Bulk approve eligible — How many studies can be approved without manual review
Progress tracking — How many studies have been reconciled vs. remaining
Per-question metrics — Agreement rates broken down by individual questions

4. Data Export¶

Existing CSV and Google Sheets exports are enriched with:

New export field	What it shows
Reconciliation status	Whether each question per study has been reconciled or not
AQVersionId	Which question version the annotation was collected against
QSVId	Which question set version was active in the stage
Agreement metrics	Percent Agreement and Cohen's Kappa per question and per study (standalone CSV)

How These Features Relate¶

These capabilities form a dependency chain — each builds on the one before:

Question Management Platform
    │
    │  Versioned questions with immutable snapshots
    │  Question sets tie versions to stages
    │
    ▼
Annotation Form v2
    │
    │  Per-question auto-save with version tracking
    │  Immutable session snapshots on submit
    │
    ▼
Reconciliation
    │
    │  Compare answers to the SAME question version
    │  Produce gold-standard record per study
    │
    ▼
Data Export
    │
    │  Version references + reconciliation status
    │  Agreement metrics for reporting
    │
    ▼
Audit Trail
    Full traceability from question creation
    through annotation to reconciliation decision

Question versioning provides the foundation — without it, reconciliation cannot meaningfully compare answers because it cannot be sure annotators were answering the same thing. The form captures per-answer version references. Reconciliation consumes both to produce authoritative answers. Export surfaces everything for downstream analysis.

What Changes for Each User Role¶

Project Administrators¶

Today	After
Answered questions are locked; no editing possible	Create new versions when editing; view version history
Questions go live immediately when created	Draft → Activate lifecycle; preview before going live
Basic question assignment via legacy UI	Version-aware assignment with drag-and-drop and ordering
No preview of annotation form	Preview tab shows exactly what annotators will see
Reconciliation = boolean flag	Dashboard with pool management, metrics, bulk approve
No quality metrics	Percent Agreement and Cohen's Kappa per question
No control over reconciliation assignment	Configure stages: MinAnnotators, cross-stage visibility, rationale requirements

Annotators¶

Today	After
Full form renders at once (slow with many questions)	Virtual scroll — smooth with 100+ questions
Must save entire form at once	Per-question auto-save — work is preserved as you go
No awareness of question versions	Transparent — versioning tracked silently
See all annotations on a study	See only their own answers (blinding enforced)

Reconcilers (new role)¶

Reconcilers are project members assigned to resolve disagreements. This is a new workflow:

Step	What they do
1. Receive assignment	System randomly assigns a study — no browsing
2. Review candidates	See anonymised answers side-by-side (Annotator A vs B)
3. Create own answers	Record their own answer for each question, even if they agree with a candidate
4. Add rationale	Optionally explain why (admin can require this per stage)
5. Submit	Creates immutable gold-standard record

Data Consumers / Analysts¶

Today	After
Export annotations without version context	Export includes question version and question set version
No reconciliation data in exports	Reconciliation status per question per study
No agreement metrics	Percent Agreement and Cohen's Kappa exportable as CSV
Cannot trace what question was asked when	Full audit trail: question → version → annotation → reconciliation

User Workflow Walkthroughs¶

Workflow 1: Admin creates and activates questions¶

Context: A project administrator is setting up a new annotation stage.

Open Question Management — Navigate to the project's Question Management page (Design tab)
Create draft questions — Add questions with types (boolean, select, checklist, etc.), set parent-child relationships, write help text, configure answer options
Iterate freely — Change wording, reorder, delete — drafts are fully mutable
Switch to Assign tab — Select the target stage, drag questions from the unassigned pool to the assigned list
Preview — Switch to Preview tab, select the stage, see exactly what annotators will see
Enable stage — When satisfied, enable the stage. SyRF automatically:
Converts all assigned drafts into versioned Annotation Questions (v1)
Creates the stage's first Question Set Version (a snapshot of which questions at which versions in what order)
Opens the stage for annotation

Key point: Drafts are cheap — iterate as much as needed before activation. Once activated, changes create versions (which is safe, just auditable).

Workflow 2: Admin edits a question after it has been answered¶

Context: An admin notices that a dropdown option is missing from an active question that annotators have already answered.

Open the question in the Design tab — a version badge shows "v1"
Click Edit — a versioning wizard appears explaining: "This question has been answered. Editing will create v2."
Make the change — add the missing dropdown option
Save as New Version — SyRF creates v2 of the question
Update the stage's question set — the admin assigns v2 to replace v1 in the stage configuration, creating a new Question Set Version
Decide about existing sessions — if annotators have in-progress sessions using v1, the admin chooses:
Pin — existing sessions continue using v1; only new sessions get v2
Invalidate — in-progress sessions are flagged for re-review
Require re-annotation — affected sessions must be re-done with v2

Key point: All past annotations collected against v1 remain linked to v1. The audit trail is preserved.

Workflow 3: Reconciler resolves disagreements¶

Context: Two annotators have completed the same study with different answers on some questions.

Open Reconciliation — The reconciler navigates to the reconciliation page
Receive assignment — The system assigns a study from the pool (the reconciler does not choose)
See side-by-side comparison — For each question:
Left column: "Annotator A" answer
Right column: "Annotator B" answer
Questions where candidates agree are shown in a separate "agreed" section
Questions where they disagree are highlighted
Create own answer — For each question, the reconciler enters their own answer
Even for agreed questions, the reconciler confirms (or overrides) the answer
Optional: add a rationale explaining the decision
Submit — Creates an immutable Reconciliation Session Version — this is the gold standard for this study
Next study — System assigns another study from the pool

Key point: The reconciler never sees who provided which answer. They create their own independent judgement.

Workflow 4: Admin bulk-approves unanimous studies¶

Context: Many studies have two annotators who agreed on everything.

Open Reconciliation Dashboard — See the pool breakdown: X studies need manual reconciliation, Y studies are eligible for bulk approve
Review bulk-approve candidates — Click "Bulk Approve" to see the list of unanimous studies
Confirm — Click "Approve All" — the system creates reconciliation records using the agreed answers
Track progress — Dashboard updates to show fewer studies in the pool

Phased Delivery¶

The initiative is delivered across nine phases. Each phase has explicit success criteria that must be verified before proceeding.

Phase Overview¶

Phase	What Ships	User Impact
1. Spec Resolution	Resolved contradictions, verified tech stack	None (internal)
2. Collection Infrastructure	New database collections, concurrency protection	None (internal)
3. Question Lifecycle	Draft questions, activation, versioning, question sets	Admin can create and version questions
4. Annotation Form v2	Rebuilt form with auto-save, virtual scroll	Annotators get faster, more reliable form
5. Question Management UI	Version history, diffs, badges, admin decisions	Admin can manage versions and stage transitions
6. Reconciliation Model	Authority determination, auto-promote	Single-annotator studies auto-promoted
7. Data Migration	Existing data backfilled, feature flags	Existing projects gain versioning retroactively
8. Reconciliation Workflow	Full reconciliation UI, metrics, dashboard	Reconcilers can resolve disagreements
9. Data Export	Enriched exports with versions and metrics	Analysts get full traceability in exports

Execution Order and Parallelism¶

Phase 1 ──► Phase 2 ──► Phase 3 ──┬──► Phase 4 (Form v2)              ──┐
                                   ├──► Phase 5 (QM UI)    ──┐          │
                                   └──► Phase 6 (Recon Model) ┴► Phase 7 ┴► Phase 8 ──► Phase 9

Key parallelism: After Phase 3 completes, Phases 4, 5, and 6 can all proceed simultaneously — they share Phase 3 as a dependency but are independent of each other. This is the most significant acceleration opportunity in the schedule.

Phase Details¶

Phase 1: Spec Resolution & API Verification¶

Goal: All implementation specs are internally consistent and the technology choices are confirmed.

What happens: The existing specification documents contain 12 identified contradictions (e.g., the reconciliation spec describes a "claim" workflow, but our design mandates random assignment). This phase resolves all contradictions and confirms that Angular 21's signal forms API exists with the expected signatures.

Delivers: A single authoritative implementation reference that developers can follow without ambiguity.

Phase 2: Collection Infrastructure¶

Goal: The database foundations are in place for versioned questions.

What happens: Two new MongoDB collections are created:

pmAnnotationQuestion — stores versioned annotation questions (separate from the project document, preventing document bloat)
pmQuestionSet — stores immutable question set snapshots

Both collections include scope and ownership fields that future cross-project sharing will use. Additionally, optimistic concurrency protection is added to study documents to prevent data corruption when multiple users annotate simultaneously.

Delivers: Database infrastructure ready for the question lifecycle.

Phase 3: Question Lifecycle¶

Goal: Administrators can create draft questions, activate them, and manage versions.

What happens: The complete draft-to-activation lifecycle is built:

Create and edit Draft Questions in the QM Design tab
Assign drafts to a disabled stage
Preview before activation
Enable stage to activate (drafts become versioned AQs, first Question Set Version created)
Edit activated questions to create new versions
Parent integrity enforcement (if a child question is in a question set, all its ancestors must be too)

Delivers: The core question management platform. Administrators can create, version, and manage questions.

Phase 4: Annotation Form v2¶

Goal: Annotators use the rebuilt form with per-question auto-save.

What happens: The annotation form is rebuilt with:

All question types (boolean, select, checklist, text, numeric, autocomplete)
Virtual scroll for large question sets
Per-question auto-save (no more losing work)
Immutable session snapshot on submit
Unsaved-changes warning
Candidate blinding at the API level

Delivers: A faster, more reliable annotation experience. Per-answer version tracking.

Phase 5: Question Management UI¶

Goal: Administrators can view version history and manage version transitions.

What happens: The question management UI gains version-awareness:

Version history panel with diff viewer
Version badges on question cards
Versioning wizard for editing activated questions
Admin decision framework for handling in-progress sessions when question sets change

Delivers: Full administrative control over question versions.

Parallelism: Runs alongside Phase 4 and Phase 6.

Phase 6: Reconciliation Model & Authority¶

Goal: The system automatically classifies studies as needing reconciliation.

What happens: The reconciliation data model is created:

ReconciliationSession entity on each study (versioned)
Authority determination rules:
1 annotator + MinAnnotators=1 → auto-promoted (gold standard created automatically)
2+ annotators → enters reconciliation pool
MinAnnotators configured as readiness threshold per stage

Delivers: Automatic study classification. Single-annotator studies get gold-standard records without manual work.

Parallelism: Runs alongside Phases 4 and 5.

Phase 7: Data Migration & Rollout¶

Goal: Existing production data is backfilled into the versioned model.

What happens:

All existing questions become v1 AQVersions
Initial Question Set Versions created for each stage
Single-annotator studies auto-promoted
Multi-annotator studies enter the reconciliation pool
Feature flags enable ordered rollout (versioning → form v2 → reconciliation)
Rollback capability: any new fields can be removed with a simple database operation

Delivers: Existing projects gain versioning and classification retroactively. No data deleted.

Depends on: Phases 3 and 6 (needs both the question model and the reconciliation model).

Phase 8: Reconciliation Workflow & UI¶

Goal: Reconcilers can resolve disagreements through the full workflow.

What happens: The complete reconciliation experience:

Random pool assignment
Candidate blinding (Annotator A, Annotator B)
Side-by-side comparison form
Reconciler creates own annotations
Bulk approve for unanimous studies
Cross-stage visibility (admin-configurable)
Per-answer rationale (optional or required per stage)
Agreement metrics computation (Percent Agreement, Cohen's Kappa)
Reconciliation dashboard with pool management

Delivers: The complete reconciliation workflow — the core value of this initiative.

Phase 9: Data Export¶

Goal: Exports include reconciliation data and agreement metrics.

What happens:

CSV/Google Sheets exports gain reconciliation status and version references
Standalone agreement metrics CSV export

Delivers: Full traceability in exported data for downstream analysis and reporting.

Migration and Data Safety¶

Existing data is safe¶

The migration is deliberately minimal and additive:

No data is deleted or moved — existing annotations, questions, and projects are untouched
New fields are added alongside existing ones
All existing questions become "version 1" — which is accurate, since they have never been versioned before
If anything goes wrong, new fields can be removed and the system reverts to its current behaviour

Backward compatibility¶

The current production question management UI continues to work throughout migration
The existing annotation form continues to work (the rebuilt form rolls out behind a feature flag)
API consumers see the same data structures with additional optional fields

Rollout strategy¶

Features roll out in a controlled order using feature flags:

Question versioning (first) — Admin-facing, no annotator impact
Annotation Form v2 (second) — Behind feature flag, opt-in per project
Reconciliation (third) — Requires both versioning and form v2

Each can be rolled back independently by unsetting feature flags and removing new database fields.

Key Risks¶

Risk	Impact	Mitigation
Document size growth — Embedding version history in project documents could exceed MongoDB's 16MB limit	High	Questions and question sets stored in separate collections (pmAnnotationQuestion, pmQuestionSet), not embedded. Monitoring alerts at 4MB, 8MB, 12MB.
Shared staging/production database — Staging and production share the same MongoDB instance	High	Additive-only migration. No destructive operations. Migration tested in isolated environment first.
Angular 21 signal forms stability — New API may change	Medium	API verified in Phase 1. Adapter pattern insulates from changes.
Performance with 100+ questions — Form must remain responsive	Medium	Virtual scroll renders only visible items. Benchmarked at each phase.
User confusion with versioning — Admins may not understand version lifecycle	Medium	Draft → Activate lifecycle separates design from deployment. Version badges and history panel make versions visible.
Reconciler availability — Studies may queue if too few reconcilers	Low	Bulk approve handles unanimous studies automatically. Dashboard shows pool health.

Decisions Requiring Product Input¶

The following items need product-owner input. Some are from the original specification, updated to reflect the current design:

Confirmed decisions (for awareness)¶

These decisions have been made during architecture planning. They are listed here for visibility:

Random assignment for reconciliation — Reconcilers cannot browse or cherry-pick studies. The system assigns randomly. This prevents bias and matches the screening reconciliation pattern.
Reconciler always creates own annotation — Even when agreeing with a candidate, the reconciler records their own answer. The gold standard is always the reconciler's deliberate decision.
Candidate blinding is a system invariant — Annotators never see each other's answers. This is enforced at the API level, not the UI level.
No bypass for reconciliation — Every multi-annotator study goes through reconciliation (or bulk approve if unanimous). There is no "skip reconciliation" option. This ensures quality is never silently undermined.
MinAnnotators is a readiness threshold — It controls when a study is ready for reconciliation, not whether reconciliation is needed. 2+ annotators always requires reconciliation.

Open decisions needing PO input¶

Legacy annotation audit — Should we run an audit to identify annotations that reference questions that no longer exist? This would clean up historical data before migration.
Reconciliation access control — Who should be able to act as a reconciler?
(a) Any project member with annotation permission
(b) Designated reconcilers only (new role)
Cross-stage visibility default — When a reconciler is working on Stage 2, should they see the study's Stage 1 reconciled answers by default? Options:
Blind — See nothing from other stages (strictest)
ShowOwnPrior — See their own prior annotations from other stages
ShowReconciled — See the reconciled answers from prior stages
Rationale requirement — Should reconcilers be required to explain their decisions?
(a) Always required (strongest audit trail)
(b) Required only when overriding both candidates
(d) Admin-configurable per stage
Export format changes — Adding version references to data exports may affect downstream tools. Should we survey current data export consumers before making changes?
Minimum viable metrics — Ship both Percent Agreement and Cohen's Kappa from day one, or start with Percent Agreement only (simpler to explain)?

Deferred design decisions (documented, built later)¶

These have been designed but will be implemented in a future version:

AQV update impact handling — When a question version changes and annotators have already answered the old version, there are three options:
Option A: Mark affected annotations as "answered against outdated version" (informational only)
Option B: Invalidate affected answers and require re-annotation
Option C: Allow both old and new versions to coexist in the stage

The wizard UI for this is designed but will be built after the core versioning ships.

Question reorganization wizards — Copy, Copy & Disable, and Copy & Delete workflows for restructuring question hierarchies are designed but deferred to a future version.

Future Capabilities (v2 and beyond)¶

The architecture is being built to support these future capabilities, but they are not in scope for v1:

Screening Reconciliation (v1.x)¶

Structured screening annotations with hierarchical exclusion reasons
Screening profiles (named, reusable inclusion/exclusion criteria)
Screening reconciliation with random pool assignment (same pattern as annotation reconciliation)
Screening reconciliation must complete before annotation reconciliation begins

Questions scoped at four levels: System, Organisation, Researcher, Project
Import a question set from another project by reference (no copy needed)
Fork a shared question to customise it (copy-on-write)
Community catalogue of published question sets
Organisation-level question libraries

Advanced Question Management (v2)¶

AQV update impact wizard (Options A/B/C for affected annotations)
Question reorganization wizards (Copy, Copy & Disable, Copy & Delete)
System questions remodelled as a CAMARADES-curated question set

Document	Audience	Purpose
Engineering Master Plan	Engineering	Technical overview of all features
Design Decisions	Engineering	Authoritative — 36 design decisions, takes precedence over other specs
Question Management Spec	Engineering	Detailed versioning and stage config design
Reconciliation Spec	Engineering	Workflow, metrics, and data model
Annotation Form v2 Spec	Engineering	Performance, signal forms, virtual scroll
Data Model & Migration	Engineering	Schema changes and migration strategy (see design decisions)
Requirements	Engineering	54 v1 requirements with phase traceability
Roadmap	Engineering	9-phase delivery plan

Annotation Management & Reconciliation — Product Specification¶

Executive Summary¶

The Problem¶

What happens today¶

Three practical consequences¶

What this means for systematic reviews¶

What We Are Building¶

1. Question Management Platform¶

Draft Questions and Activation¶

Question Versioning¶

Question Sets¶

Project Question Library¶

2. Annotation Form v2¶

3. Annotation Reconciliation¶

How it works¶

Key design principles¶

Authority determination¶

Agreement metrics¶

Reconciliation dashboard¶

4. Data Export¶

How These Features Relate¶

What Changes for Each User Role¶

Project Administrators¶

Annotators¶

Reconcilers (new role)¶

Data Consumers / Analysts¶

User Workflow Walkthroughs¶

Workflow 1: Admin creates and activates questions¶

Workflow 2: Admin edits a question after it has been answered¶

Workflow 3: Reconciler resolves disagreements¶

Workflow 4: Admin bulk-approves unanimous studies¶

Phased Delivery¶

Phase Overview¶

Execution Order and Parallelism¶

Phase Details¶

Phase 1: Spec Resolution & API Verification¶

Phase 2: Collection Infrastructure¶

Phase 3: Question Lifecycle¶

Phase 4: Annotation Form v2¶

Phase 5: Question Management UI¶

Phase 6: Reconciliation Model & Authority¶

Phase 7: Data Migration & Rollout¶

Phase 8: Reconciliation Workflow & UI¶

Phase 9: Data Export¶

Migration and Data Safety¶

Existing data is safe¶

Backward compatibility¶

Rollout strategy¶

Key Risks¶

Decisions Requiring Product Input¶

Confirmed decisions (for awareness)¶

Open decisions needing PO input¶

Deferred design decisions (documented, built later)¶

Future Capabilities (v2 and beyond)¶

Screening Reconciliation (v1.x)¶

Cross-Project Question Sharing (v2)¶

Advanced Question Management (v2)¶

Related Documentation¶