Skip to content

Screening Annotations: Capturing Reasons for Exclusion

Overview

Feature Name: Screening Annotations Target Users: Reviewers, Project Administrators Business Value: Enable PRISMA-compliant reporting, improve review transparency, and promote methodological rigour in exclusion decisions

The Problem

SyRF's screening infrastructure has no built-in mechanism to record why a study was excluded. Reviewers can mark a study as "Exclude" but cannot capture the justification as part of that screening decision.

The Current Workaround: Annotations as Pseudo-Screening

Teams have found a workaround: they use annotation questions (the existing annotation infrastructure) to track exclusion reasons. For example, they create annotation questions like "Reason for Exclusion" with options such as "Wrong Population", "Wrong Intervention", etc.

The problem: This annotation data is completely decoupled from the screening decision:

  • The exclusion reason lives in the annotation system
  • The screening decision lives in the screening system
  • There is no formal link between them
  • Depending on stage configuration and project settings, the context of why a study was excluded is not tied to the actual screening decision

The Consequence: Teams Abandon Screening

Because of this disconnect, many teams abandon the screening infrastructure entirely and instead: 1. Use annotation questions to capture both the decision AND the reason 2. Treat annotation responses as their de facto screening outcome 3. Lose the benefits of SyRF's screening-specific features (reconciliation, conflict handling, stage filtering)

This is a significant platform adoption problem - teams are working around core functionality rather than using it as designed.

Why This Matters

  1. PRISMA Compliance Gap: The PRISMA 2020 guidelines require reporting exclusions broken down by reason. When reasons are captured separately from screening decisions, generating compliant reports requires manual data reconciliation.

  2. Data Integrity: Without a formal link, there's no guarantee that an exclusion reason corresponds to an actual screening exclusion. A reviewer might annotate a reason but forget to mark the screening decision, or vice versa.

  3. Lost Platform Value: Teams using annotations-as-screening miss out on screening-specific features like dual screening, automated reconciliation, and stage-based study pools.

  4. Methodological Rigour: When exclusion reasons are formally structured and required at the point of decision, reviewers are encouraged to apply exclusion criteria more consistently and thoughtfully. This aligns with SyRF's core mission: empowering researchers to follow best-practice methodology. A structured exclusion form acts as a checklist that prompts reviewers to explicitly identify which criterion the study fails, rather than making ad-hoc or undocumented decisions.

The Solution: Screening Annotations

Formally couple exclusion reasons to screening decisions - when configured by the project admin.

The goal is NOT to replace the existing annotation infrastructure, but to provide an optional, tightly-integrated path for teams who want their exclusion reasons directly linked to screening decisions.

Key Concept

A Screening Annotation Definition is a configurable questionnaire that captures structured reasons when a reviewer excludes a study. It is:

  • Linked to a Screening Profile (the inclusion/exclusion criteria)
  • Triggered by the screening action (not a separate annotation step)
  • Stored atomically with the screening decision (guaranteed data integrity)
  • Optional - admins choose whether to require reasons for their screening definitions

This acts as an "exit survey" for excluded studies, but one that is formally part of the screening record.

Why Structured Data Matters

Approach Example PRISMA Usable? Analysis Power
Free text "Wrong species, not relevant" No None
Flat tags "Wrong Population" Limited Basic counts
Structured questionnaire Primary reason → Wrong Population → Sub-reason: "Non-mammalian species" Yes Full breakdown + drill-down

Our approach enables:

  • Automatic PRISMA diagram generation with reason breakdowns
  • Detailed analysis of why studies fail criteria
  • Consistent categorisation across reviewers
  • Methodological discipline: The structured form prompts reviewers to systematically apply exclusion criteria, reducing subjective or inconsistent decision-making

User Workflows

Administrator: Creating an Exclusion Form

Where: Project Settings → Screening Annotation Definitions (new section)

Workflow: 1. Create a new Screening Annotation Definition (e.g., "Full-Text Exclusion Reasons") 2. Add questions using familiar question types: - Single-choice (radio buttons) - Multiple-choice (checkboxes) - Dropdown lists - Free-text input 3. Configure conditional/hierarchical logic: - Example: If "Primary Reason" = "Wrong Population" → show sub-question "Describe the population issue" 4. Link the definition to one or more Screening Profiles

Example Form Structure: 

Q1: Primary Reason for Exclusion [Single choice, Required]
   - Wrong Population
   - Wrong Intervention
   - Wrong Outcome
   - Wrong Study Design
   - Duplicate
   - Full text unavailable
   - Other

   Q1.a) [Conditional - if Q1 = "Wrong Population"]
         Population Issue Details [Free text]

   Q1.b) [Conditional - if Q1 = "Other"]
         Please specify [Free text, Required]

Reviewer: Excluding a Study

The reviewer experience depends on whether the admin has configured a Screening Annotation Definition for the active Screening Profile.

When Screening Annotations ARE configured:

Workflow: 1. Reviewer reviews study and decides to exclude 2. Clicks "Exclude" button 3. Instead of immediate completion: An inline form or modal appears showing the linked Screening Annotation questions 4. Reviewer completes the required questions 5. Clicks "Submit" → Both the exclusion decision AND the structured reasons are saved atomically

The form is mandatory when configured - this guarantees 100% of exclusions have recorded reasons. In high-throughput screening, optional fields are routinely skipped. By coupling justification to the decision, we guarantee data completeness for PRISMA reporting.

No partial saves: The screening annotation form must be completed in full before submission. Reviewers cannot save progress and return later - this ensures atomic, complete records and prevents orphaned partial data.

When Screening Annotations are NOT configured:

The existing behaviour remains unchanged - clicking "Exclude" immediately records the exclusion with no additional prompts. Teams who don't need formal exclusion reasons (or who prefer to capture them via the existing annotation infrastructure) are unaffected.

Why This Is Better Than the Current Workaround

Aspect Current (Annotations as Pseudo-Screening) Proposed (Screening Annotations)
Data coupling Separate systems, no formal link Single atomic record
Timing Annotation may happen before/after/never relative to screening Captured at moment of decision
Reconciliation Must reconcile annotations AND screening separately Reasons follow screening reconciliation
Stage filtering Annotations don't drive stage pools Screening outcomes (with reasons) drive pools
PRISMA reporting Manual join of two data sources Direct aggregation from screening data
Methodological rigour Reasons optional, easily skipped Structured form enforces systematic criteria application

Reconciliation Terminology

The screening pipeline involves three distinct reconciliation concepts. This document is precise about which is meant in each context.

Three Reconciliation Types

Type Mechanism Entity Created Phase
Screening decision reconciliation Automatic (agreement mode) None — updates Screening Outcome status Current (Phase 1)
Screening annotation reconciliation Manual — reconciler reviews candidate screenings Reconciled Screening (decision + screening annotation) Phase 2 (this feature)
Data extraction reconciliation Manual — reconciler reviews candidate annotation sessions Reconciled Annotation Session (authoritative data extraction answers) Future

Acts vs Workflow

The last two types above are distinct reconciliation acts — they always produce different entities with different purposes. However, the reconciliation workflow (user experience) can evolve:

  • MVP: Separate workflows — screening annotation reconciliation is the only manual reconciliation implemented. Data extraction reconciliation is a future feature with its own design process.
  • Future: An integrated reconciler workbench could present both acts in a single session (shared assignment queues, unified study view), while preserving the distinction between entity types. This integration would be a UX improvement that does not merge the underlying data models.

When this document says "reconciliation" without a qualifier in a workflow context (e.g., "reconciliation pool", "reconciler role"), it refers to screening annotation reconciliation unless stated otherwise. The specific act or entity is always named explicitly where precision matters.

Screening Annotation Reconciliation

Important Distinction: Screening Decision vs Screening Annotation Reconciliation

These are two separate processes that operate differently:

Aspect Screening Decision Reconciliation (Automatic) Screening Annotation Reconciliation (Manual — New)
Trigger Reviewers disagree on Include/Exclude Reviewers both Exclude but give different reasons, or study enters reconciliation pool
Resolution Automatic tie-breaker (agreement mode) Manual reconciler decision
Role required Any reviewer who hasn't screened Formal Reconciler permission
Human oversight Implicit (majority wins) Explicit (reconciler owns final decision)
Entity created None (updates Screening Outcome) Reconciled Screening (decision + annotation)

Current Screening Decision Reconciliation (Background)

For reference, the current screening decision reconciliation works as follows:

  • Default agreement mode: If 2 reviewers disagree (Include vs Exclude), the study is randomly presented to a 3rd reviewer
  • Any reviewer who hasn't already screened the study can be the tie-breaker
  • The majority decision becomes the final Screening Outcome
  • This is automatic - no special role required

Screening Annotation Reconciliation (New Process)

Screening Annotation reconciliation is fundamentally different and requires explicit human ownership:

The Reconciler Role

  • A stage-level permission configured in the stage's security settings
  • Permissions are granted via project groups (members are assigned to groups, groups are granted stage permissions) or direct member assignment at the stage level
  • This follows SyRF's existing permission model (see StageActivity.Reconcile in StagePermission.cs)
  • Only users with this permission can reconcile screening annotations for that stage
  • Multiple reconcilers can be assigned (see considerations below)
  • The reconciler's identity is recorded on the final reconciled annotation

Why Human Ownership Matters

The final exclusion reason that appears in PRISMA reporting must have clear attribution. Even when annotations agree, a human reconciler "signs off" on the final value. This ensures:

  • Accountability for the reported reasons
  • Audit trail showing who approved the final data
  • Opportunity to catch errors even in "agreement" cases

Reconciler Workflow

When a study has multiple screening annotations (from dual screening where both reviewers excluded):

  1. Reconciler views the study with both reviewers' screening annotations displayed
  2. Comparison view shows each reviewer's answers side-by-side
  3. Agreement indicators highlight where answers match vs differ
  4. Reconciler makes decisions:
  5. For agreed answers: Can accept as-is (one click)
  6. For disagreed answers: Must choose or enter a reconciled value
  7. Submit creates the final Reconciled Screening Annotation attributed to the reconciler

Auto-Reconciliation Option

Reconcilers can choose to auto-reconcile screening annotations that are in agreement:

  • Still attributed to the reconciler - they own the decision
  • Flagged as auto-generated - metadata indicates "auto-reconciled due to agreement"
  • Bulk action available - "Auto-reconcile all studies with full agreement"
  • Reconciler can review before committing - preview which studies would be affected

This saves time while maintaining human ownership of the final data.

Admin Configuration: What Gets Reconciled?

Not all screening annotation questions are equally suitable for reconciliation. Admins should be able to configure:

Option 1: Primary Question Only

  • Only reconcile the main "Reason for Exclusion" question (typically single/multiple choice)
  • Choice-based questions are more likely to have literal agreement
  • Faster reconciliation, simpler workflow
  • Secondary questions (free text details) are stored but not formally reconciled

Option 2: All Questions

  • Reconcile every question in the Screening Annotation Definition
  • More complete data, but more reconciler effort
  • Free-text questions may have semantic agreement but literal differences (different phrasing, spelling, etc.)
  • Much less scope for auto-reconciliation

Recommendation: Make this configurable per Screening Annotation Definition, with "Primary Question Only" as the default.

Design Considerations

Multiple Reconcilers

When multiple users have the Reconciler permission:

  • No locking needed for different studies - reconcilers work on separate studies
  • Same study: First reconciler to submit "wins" - subsequent attempts show "already reconciled"
  • Workload distribution: Consider a "claim" mechanism or assignment queue (future enhancement)
  • Consistency: All reconcilers should follow the same criteria - consider reconciler guidelines/training

Reconciled Annotation Status and Independence

Key principle: The reconciler's screening annotation is a separate, independent annotation with special status:

  1. Independent judgment: The reconciler interprets what the reviewers meant and records their own authoritative annotation - it is not a merge or selection from reviewer answers
  2. Special "reconciled" status: The reconciled annotation is stored with a distinct status that differentiates it from candidate screening annotations. Only the reconciled annotation is used for PRISMA reporting and final outcomes
  3. Candidate annotations preserved: The original reviewer (candidate) annotations remain in the system for audit purposes but are superseded by the reconciled annotation

This means:

  • Free-text handling is simplified - the reconciler writes their own interpretation at their discretion
  • No "agreement matching" needed for free text - reconcilers decide based on semantic meaning
  • Attribution is clear - the reconciled annotation is the reconciler's own work

Why Multiple Choice is Strongly Preferred

While free text is permitted, multiple choice questions are strongly preferred for screening annotations:

Aspect Multiple Choice Free Text
Auto-reconciliation ✅ Enables automatic reconciliation when reviewers agree ❌ Cannot auto-reconcile (semantic vs literal agreement)
Systematic approach ✅ Enforces consistent categorisation ⚠️ Allows inconsistent phrasing
PRISMA reporting ✅ Direct aggregation ⚠️ Requires manual categorisation
Reconciler effort ✅ Quick verification ⚠️ Must interpret and re-write

Recommendation: Design Screening Annotation Definitions with multiple choice as the primary question type. Use free text sparingly for "Other - please specify" or supplementary details only.

Agreement and Candidate Screenings in Agreement

Before discussing reconciliation, we need clear definitions:

Agreement Threshold

The agreement threshold is defined by the project's agreement mode configuration. A study "meets the agreement threshold" when:

  • The required number of candidate screenings has been completed (e.g., 2 reviewers)
  • The required agreement ratio is satisfied (e.g., majority, unanimous)

Agreed Decision

When the agreement threshold is met, the agreed decision is the majority decision among candidate screenings. For example:

  • 2 Include, 0 Exclude → Agreed decision = Include
  • 1 Include, 2 Exclude → Agreed decision = Exclude
  • 2 Include, 1 Exclude → Agreed decision = Include (majority)

Candidate Screenings in Agreement

Candidate screenings in agreement are those candidate screenings whose decision matches the agreed decision.

Example Agreed Decision Candidate Screenings in Agreement
Reviewer A: Include, Reviewer B: Include Include A and B
Reviewer A: Exclude, Reviewer B: Exclude, Reviewer C: Include Exclude A and B (not C)
Reviewer A: Include, Reviewer B: Exclude, Reviewer C: Include Include A and C (not B)

Only the candidate screenings in agreement (and their associated screening annotations) contribute to the Final Screening Outcome when reconciliation is not performed.

Reconciliation Pool Configuration

The reconciliation pool is the set of studies that require manual reconciler review for a given screening profile. Admins configure which studies are added to this pool.

Default Rule: Reconcile When Annotations Exist

By default, a study is added to the reconciliation pool if:

Condition Rationale
Candidates disagree on decision Decision conflict requires authoritative resolution
Candidate screenings in agreement have screening annotations Annotations need reconciler review to create authoritative PRISMA data

By default, a study is NOT added to the reconciliation pool only if:

Condition Rationale
Candidates agree AND no screening annotations exist Nothing to reconcile - no decision conflict, no annotations

Key insight: The default is conservative. Any study with screening annotations goes to the reconciliation pool, ensuring reconciler oversight of all exclusion reasons.

Admin-Configured Bypass Criteria

Admins can configure criteria that allow studies to bypass the reconciliation pool and have their Final Screening Outcome derived directly from candidate agreement.

The admin specifies which screening annotation questions must be in perfect agreement for a study to bypass reconciliation:

Config Bypass Criteria Use Case
No bypass (default) All studies with annotations require reconciliation Maximum oversight
Primary question only Primary exclusion reason must match exactly Balanced approach
Specified questions Listed questions must all match exactly Custom control
All questions Every annotation answer must match exactly Strictest auto-bypass

Example: If admin configures "Primary question only" bypass:

Scenario Primary Q Match? Bypass? Result
Both "Wrong Population" ✅ Yes ✅ Bypass Final Outcome from candidate agreement
"Wrong Population" vs "Wrong Intervention" ❌ No ❌ Pool Requires reconciler
Both "Wrong Population", different sub-reasons ✅ Yes ✅ Bypass Final Outcome includes agreed primary, truncates at disagreement

Annotation Tree Truncation (Agreement-Derived Annotations)

When the Final Screening Outcome annotation is derived from candidate screenings in agreement (not from a reconciled screening), the annotation only includes answers up to the point where any branch disagrees.

Example - Hierarchical annotation structure:

Q1: Primary Reason for Exclusion
├── Q1.a: [Conditional] Population Issue Details
└── Q1.b: [Conditional] Please specify other

Reviewer A answers:
  Q1: "Wrong Population"
  Q1.a: "Non-mammalian species"

Reviewer B answers:
  Q1: "Wrong Population"
  Q1.a: "Wrong age range"

Result in Final Screening Outcome:

Annotation:
  Q1: "Wrong Population"    ✓ Included (agreed)
  Q1.a: [omitted]           ✗ Truncated (disagreed)

The Final Screening Outcome captures the agreed exclusion reason ("Wrong Population") for PRISMA reporting, while acknowledging that deeper detail was not agreed upon.

Truncation rules:

  • Include answers where ALL candidate screenings in agreement have identical values
  • Omit answers where any disagreement exists among candidates in agreement
  • Truncation applies at each level of the annotation tree independently

Alternative Configuration: All Studies Require Reconciliation

Admins can configure that ALL studies are added to the reconciliation pool, regardless of agreement:

  • Reconciler reviews every study, including agreed Includes
  • Maximum oversight - every decision has explicit reconciler sign-off
  • Use case: High-stakes reviews, regulatory submissions, training scenarios

The Final Screening Outcome as Authoritative Source

The Final Screening Outcome document is THE authoritative source for both the screening decision and screening annotation. All downstream consumers (stage study pools, PRISMA reporting, exports) use this document exclusively.

Source Final Screening Outcome Contains
Reconciled Screening exists Decision and Annotation copied from Reconciled Screening
No Reconciled Screening (candidate agreement) Decision from agreed decision; Annotation from candidate screenings in agreement (truncated at disagreement)

This design ensures a single, consistent source of truth regardless of how the outcome was determined.

Unified Reconciliation: Decision and Annotation

Rather than treating screening decision reconciliation and screening annotation reconciliation as separate processes, we unify them into a single Reconciled Screening concept. This aligns with systematic review best practice where an adjudicator (the reconciler) has full authority to make the final, authoritative decision.

The Reconciled Screening Model

When a reconciler reviews a study, they create a Reconciled Screening that consists of:

Component Description When Created
Reconciled Screening Decision The authoritative Include/Exclude decision Always
Reconciled Screening Annotation The authoritative exclusion reason(s) Only when decision is Exclude

This is a single, unified process - not two separate workflows.

Data Model and Precedence

Study
├── Candidate Screenings[]                    # From reviewers (primary data, exportable for audit)
│   ├── Decision: Include | Exclude
│   ├── ReviewerId
│   └── Candidate Screening Annotation        # If applicable 
│       └── Answers to screening annotation questions
│
├── Reconciled Screening (0..1)               # From reconciler (when reconciliation performed)
│   ├── Decision: Include | Exclude
│   ├── ReconcilerId                          # Always required - reconciler owns the decision
│   ├── WasAutoReconciled: boolean            # True if auto-reconciled due to agreement
│   ├── Reconciled Screening Annotation       # If applicable
│   │   └── Answers to screening annotation questions
│   └── ReconciliationNotes (optional)        # Justification for decision
│
└── Final Screening Outcome                   # THE AUTHORITATIVE SOURCE for operational use
    ├── Decision: Include | Exclude
    ├── Annotation                            # If applicable (truncated if from candidate agreement)
    │   └── Answers to screening annotation questions
    └── Source: Reconciled | CandidateAgreement

Final Screening Outcome is THE authoritative source for operational use:

Stage study pools, PRISMA reporting, and standard exports use the Final Screening Outcome exclusively. This ensures consistent, authoritative data for all operational workflows.

Candidate Screenings remain accessible:

The primary data (candidate screenings and their annotations) is preserved and exportable for audit, research, and inter-rater reliability analysis. This supports transparency and methodological scrutiny.

Reconciled Screening cardinality (0..1):

  • 0 reconciled screenings: Reconciliation was not performed (per configured reconciliation pool policy, or pending)
  • 1 reconciled screening: A reconciler has explicitly reviewed the study (including via bulk auto-reconciliation)

WasAutoReconciled flag:

  • true: The reconciled screening was created via bulk auto-reconciliation (reviewers agreed, reconciler approved in bulk)
  • false: The reconciled screening was created through manual reconciler review
  • This flag is for audit purposes - both types are equally authoritative

Final Screening Outcome population:

Scenario Decision Source Annotation Source
Reconciled Screening exists Reconciled Screening Decision Reconciled Screening Annotation (complete)
No Reconciled Screening Agreed decision from candidates in agreement Candidate annotations in agreement (truncated at disagreement)

When derived from candidate agreement, the annotation includes only answers where ALL candidate screenings in agreement had identical values. Disagreed branches are truncated (see "Annotation Tree Truncation" above).

Key uses of Final Screening Outcome:

  • Stage study pool filtering: This is the property used when filtering studies for other stage study pools. For example, a "Full-Text Screening" stage might define its study pool as "studies where Title/Abstract Screening outcome = Include"
  • Reporting queries: "Find all excluded studies", "count by screening outcome"
  • PRISMA diagram generation: Aggregating exclusion counts by reason

Source field tracks how the Final Screening Outcome was populated:

  • Reconciled: Final outcome derived from Reconciled Screening (reconciler-owned)
  • CandidateAgreement: Final outcome derived from candidate agreement (no reconciliation performed)

Why This Approach?

This unified model:

  • Grants full authority: Reconcilers can confirm OR change the screening decision
  • Maintains audit trail: Original candidate decisions and annotations are preserved
  • Simplifies workflow: One reconciliation process covers both decision and reasons
  • Aligns with best practice: Matches PRISMA 2020 and Cochrane adjudication methodology
  • Handles all scenarios: Works whether reviewers agreed, disagreed, or both made errors

Reconciler Scenarios

Scenario Reconciler Action Result
Reviewers agree: both Exclude, same reason Confirm decision, auto-reconcile annotation Reconciled = Exclude with agreed reason
Reviewers agree: both Exclude, different reasons Confirm decision, write reconciled annotation Reconciled = Exclude with reconciler's reason
Reviewers disagree: Include vs Exclude Make authoritative decision Reconciled = reconciler's choice
Reviewers agree: both Exclude, but wrong Change decision to Include Reconciled = Include (no annotation needed)
Reviewers agree: both Include, but wrong Change decision to Exclude Reconciled = Exclude with reconciler's annotation

Impact on Existing Agreement Modes

The current agreement-based screening (where 2 agreeing reviewers = automatic final outcome) continues to work for teams who don't use reconciliation. The Reconciled Screening is optional - it only exists when a reconciler explicitly reviews a study.

Configuration Behaviour
No reconciler assigned Current agreement modes apply (automatic)
Reconciler assigned, hasn't reviewed Candidate agreement applies (pending reconciliation)
Reconciler has reviewed Reconciled Screening Decision is Final Outcome

Reporting Considerations

For PRISMA and statistics:

  • Final counts use the Reconciled Screening Decision (or candidate agreement if no reconciliation)
  • Audit reports can show: original candidate decisions → reconciled decision → any changes made
  • Reconciliation rate: Track how often reconcilers change vs confirm candidate decisions
  • Error detection: Track how often reconcilers identify candidate errors (quality metric)

Integration with Regular Annotations

Stages can have both Screening (with Screening Annotations) and regular Annotations enabled. Key questions:

Data Separation

  • Recommendation: Keep Screening Annotations separate from regular Annotations in the study document
  • Different purposes: Screening Annotations capture decision justifications; regular Annotations are data extraction
  • Different reconciliation workflows and timing
  • Clearer audit trail and reporting

UX Considerations (Default Behaviour)

The default behaviour assumes screening annotations are configured for Exclude decisions:

Scenario Default UX
Reviewer excludes Show Screening Annotation form immediately (part of exclusion action)
Reviewer includes Proceed to regular Annotation questions (if configured)
Reconciler view Separate tabs/sections for Screening Annotation reconciliation vs Annotation reconciliation

Configurable alternatives:

  • Screening annotations for Include: Form shown on Include action (e.g., capturing eligibility confirmation)
  • Screening annotations for both decisions: Form shown on both Include and Exclude
  • All screenings require reconciliation: Every study goes to reconciliation pool regardless of agreement

The system should support these configurations; the UX adapts accordingly based on the Screening Annotation Definition settings.

Reconciliation Workflow: Separate or Integrated?

The two manual reconciliation acts (screening annotation reconciliation → Reconciled Screening; data extraction reconciliation → Reconciled Annotation Session) are always distinct at the data model level. The question is whether the workflow (user experience) should present them separately or together.

Three options to consider:

  1. Separate workflows (MVP approach)
  2. Screening annotation reconciliation happens first (produces Reconciled Screenings)
  3. Data extraction reconciliation happens separately (produces Reconciled Annotation Sessions)
  4. Clearer workflow, easier to implement

  5. Each act has its own pool, assignment queue, and UI

  6. Integrated workflow

  7. Single reconciler workbench covers both acts in one session
  8. Reconciler sees screening annotations and data extraction side-by-side for the same study
  9. More efficient for reconcilers, but more complex UX

  10. The two entity types remain distinct — integration is purely at the UI/assignment level

  11. Admin configurable

  12. Let project admins choose based on their workflow
  13. Maximum flexibility, but adds complexity

Decision for MVP: Separate workflows (Option 1). This is primarily because data extraction reconciliation is not yet implemented. When it is being designed and specified, deeper consideration of an integrated workflow should take place — with the understanding that integration is a UX concern that does not merge the underlying reconciliation acts or their entity types.

PRISMA Integration

The primary driver for this feature is enabling automated PRISMA 2020 flow diagram generation.

How Screening Annotations Feed PRISMA

The PRISMA box "Reports excluded (with reasons)" requires: - Total count of excluded studies at full-text stage - Breakdown by exclusion reason

With Screening Annotations:

  1. System counts all studies with "Exclude" outcome from the Full-Text Screening Profile
  2. System aggregates the answers from the linked Screening Annotation Definition
  3. Generates breakdown: "Wrong Population (n=45), Wrong Intervention (n=23), Wrong Study Design (n=12)..."

Result: One-click PRISMA diagram export with accurate, auditable reason breakdowns.

Upstream PRISMA Data (Future Dependency)

Screening Annotations address the "Reports excluded (with reasons)" box in PRISMA 2020. Full PRISMA flow diagram generation also requires upstream data not covered by this feature:

  • Identification phase: Import-level source_type tagging (Database/Register, Citation, Website, Other) and duplicate counts captured during deduplication
  • Screening phase: Pre-screen removal counts (e.g., ineligiblePreScreen status) and "reports not retrieved" tracking (FullTextUnavailable)

These are separate feature requirements that should be addressed when PRISMA diagram generation is designed in full.

Competitive Advantage

Platform Exclusion Reasons Structure Level
Covidence Simple tag selection Flat list
Rayyan Labels/tags Flat list
SyRF (proposed) Hierarchical questionnaire Conditional logic, sub-questions

Our approach captures richer, more detailed data that enables:

  • More granular PRISMA reporting
  • Better analysis of screening patterns
  • Identification of criteria that cause most exclusions

Future consideration: Evaluate what UX features other platforms provide that SyRF currently lacks. Consider whether adopting such features (in some form) aligns with SyRF's methodological ethos. Also ensure simple use cases remain simple - avoid locking users into overly complex workflows when a straightforward approach would suffice.

Scope & Boundaries

In Scope (MVP)

  • Create/edit Screening Annotation Definitions with hierarchical questions
  • Link definitions to Screening Profiles (optional per definition)
  • Mandatory annotation form on exclusion when configured (no partial saves)
  • Store annotations atomically with exclusion decision
  • Basic reporting: count exclusions by top-level reason
  • PRISMA data export with reason breakdown
  • Reconciler role/permission for Screening Annotation reconciliation
  • Reconciler UI: side-by-side comparison, agreement indicators
  • Auto-reconcile option for annotations in agreement (attributed to reconciler)
  • Separate Screening Annotation data from regular Annotations in study document

Out of Scope (MVP)

  • Annotations for "Include" decisions: Accommodated but not default. The system supports configuring screening annotations for Include decisions (or both Include and Exclude), but the default mode is annotations for Exclude decisions only. Admin guidance should explain how to configure alternative modes when needed (e.g., when capturing reasons for inclusion is methodologically required).
  • Bulk editing of historical annotations
  • Custom validation rules beyond required/optional
  • Migration tool for teams currently using annotation workaround (see Key Questions)
  • Integrated reconciliation workflow (unified workbench covering both screening annotation reconciliation and data extraction reconciliation acts in a single session)
  • Reconciler assignment/claim queue for workload distribution

Coexistence with Existing Annotation Infrastructure

This feature does NOT replace the existing annotation system. Teams can:

  1. Use Screening Annotations only - formal exclusion reasons tied to screening
  2. Use existing annotations only - continue current workaround if preferred
  3. Use both - Screening Annotations for exclusion reasons, existing annotations for other data capture

The existing annotation infrastructure remains fully functional and unchanged.

Success Metrics

Metric Target
Exclusion reason capture rate (when configured) 100% (mandatory by design)
PRISMA diagrams generated without manual data entry 90%+ of projects using feature
Time to produce PRISMA exclusion breakdown < 1 minute (vs hours manually)
Teams using screening infrastructure (vs annotation workaround) Increase adoption
New projects adopting Screening Annotations Track uptake

Key Questions for Product Owner

General

  1. Default reasons template: Should we provide a pre-built "standard exclusion reasons" template that admins can use as a starting point? Yes (not MVP but should be on the list of upcoming features because it would improve UX)

  2. Retroactive capture: For ongoing projects, should we allow (optional) backfilling of reasons for already-excluded studies? yes but not MVP (consider as future enhancement)

  3. Reviewer flexibility: Should reviewers be able to add "Other" free-text reasons, or strictly use admin-defined options only? yes, but at the project admin's discretion (configurable per question) e.g. using the autocomplete annotaiton question control. Or the project admin could also explicitly add a seperate other questions.

  4. Reporting granularity: For PRISMA, do we report only top-level reasons, or include sub-reason breakdowns? MVP top-level, potential future enhancement for sub-reasons.

Migration

  1. Migration from annotation workaround: Many teams currently use annotation questions to track exclusion reasons. Should we provide:
  2. A migration tool to convert existing annotation data into Screening Annotations?
  3. Guidance/documentation for teams to transition their workflow?
  4. Or simply let teams start fresh with new projects?

Migration is not included for MVP. Could consider in app migration or also an screening annotation import function (there already is screening decision import functionality implemented). Guidance at MVP release will be that it is not supported.

  1. Teams who abandoned screening: For teams who stopped using screening infrastructure entirely, what's the path back? Should we provide tooling to:
  2. Retroactively generate screening decisions from annotation responses?
  3. Or recommend they continue with their current approach for existing projects? At MVP stage the recommendation would be to export their anontation data that includes the screening decision and re-import it using the screening decision import function.

Reconciliation (see detailed section above)

  1. ~Reconciliation scope~: DECIDED - Should admins configure what gets reconciled?
  2. Primary question only (recommended default) - easier agreement matching

  3. All questions - more complete but harder to auto-reconcile free text Agreed default is primary question only, with option to configure all questions.

  4. Free text handling: DECIDED - The reconciler creates their own independent annotation based on their interpretation of the candidate annotations. Free text is not "reconciled" in the merge sense; the reconciler writes their own version. Multiple choice is strongly preferred as it enables auto-reconciliation.

  5. Auto-reconciliation controls: What controls should reconcilers have for bulk auto-reconciliation?

  6. Preview before committing? Not MVP
  7. Undo/rollback capability? Yes for MVP add a simple button to remove all auto-reconciled reconciled screenings by the reconciler who presses it.(with warning that this is irreversible). Reconcilers can also always manually update their reconciled screenings afterwards.

  8. Audit log of auto-reconciled studies? The autoreconciled property on the reconciled screening covers audit logging.

  9. ~Integration with regular annotation reconciliation~: DECIDED - For stages with both Screening and Annotations, should reconciliation be:

    • Separate processes (recommended for MVP)
    • Integrated into single workflow
    • Configurable by admin Yes seperate process until annotation reconciliation is designed and specified.(see above)

  10. Reconciler authority over screening decision: DECIDED - Unified Reconciliation model adopted. The reconciler creates a Reconciled Screening consisting of both the decision (Include/Exclude) and annotation (if Exclude). This has special status and takes precedence over candidate screenings. Original candidate data is preserved for audit. See "Unified Reconciliation: Decision and Annotation" section above.

Reconciliation Pool Configuration (see detailed section above)

  1. Reconciliation pool default policy: DECIDED - By default, all studies with screening annotations are added to the reconciliation pool. Studies bypass the pool only when candidates agree AND no screening annotations exist. See "Default Rule: Reconcile When Annotations Exist" above.

  2. Authoritative annotation from candidate agreement: DECIDED - When reconciliation is bypassed, the Final Screening Outcome Annotation is derived from candidate screenings in agreement using annotation tree truncation: include only answers where ALL candidates in agreement have identical values; omit (truncate) branches where any disagreement exists. See "Annotation Tree Truncation" above.

  3. Bypass criteria configuration: Admins can configure which questions must be in "perfect agreement" for a study to bypass reconciliation. Remaining questions:

    • What should be the default bypass criteria? (No bypass / Primary question only / All questions) No bypass should be default (see above for details).
    • Should this be per-screening-profile or per-stage? per-profile definitely NOT per-stage.
    • Should free-text questions be automatically excluded from agreement matching? No

  4. Screening annotations for Include decisions: Should admins be able to configure screening annotations for Include decisions (not just Exclude)?

    • Use case: Capturing "reason for inclusion" or confirming eligibility criteria
    • Would this be a separate Screening Annotation Definition, or the same one with conditional questions? Yes, they should be as MVP however the default should be exclude only. The settings to configure this should be not on the main path but in an advanced settings area to avoid complicating the main use case.

  5. Reconciliation pool visibility: DECIDED - Should reviewers/reconcilers see which studies are in the reconciliation pool vs which were skipped?

    Reviewers: Candidate reviewers must be kept blind to other reviewers' decisions, including whether a study is in the reconciliation pool. This is a fundamental methodological principle for systematic reviews that must be upheld. By default, reviewers are presented with studies in random order and have no access to studies they have not yet reviewed. However, project admins may grant a reviewer project group access to the study listing of unreviewed studies if needed for their workflow.

    Reconcilers: By default, reconcilers cannot choose or see which studies are in (or not in) the screening reconciliation pool - they work through studies ready for reconciliation at random. However, project admins may grant a reconciler project group fuller access to study status, including the ability to see pool membership and select which studies to reconcile, if that level of control is required for the project's workflow.

  6. Truncated annotation reporting: When Final Screening Outcome has truncated annotations (due to disagreement at deeper levels):

    • How should PRISMA reports indicate that sub-reasons were not agreed upon? No, this wouldn't be indicated in PRISMA reports as they typically only show top-level reasons (MVP). However, detailed reports could include a note indicating that sub-reasons were truncated due to disagreement.(non MVP)
    • Should exports distinguish between "no answer provided" vs "answer truncated due to disagreement"?

  7. Agreement with more than 2 reviewers: When there are 3+ candidate screenings in agreement with annotations:

    • Does truncation require ALL to have identical values, or just the majority?
    • Example: 3 reviewers exclude, 2 say "Wrong Population" / "Non-mammalian", 1 says "Wrong Population" / "Wrong age" - what gets truncated? All to have identical values.