Skip to content

Phase 3: Collection Infrastructure

Implementation plan reference (frozen). Originally captured as GSD-style plans in .planning/phases/03-collection-infrastructure/ on PR #2398. The phase was researched and planned but not yet executed. Migrated here so the design rationale, code skeletons, and pitfalls survive the cleanup of .planning/.

When this work is picked up: Use this as a design reference. Execute through whatever workflow is current at the time — the original GSD execution loop is no longer in active use.

Phase goal: Two new MongoDB collections (pmAnnotationQuestion and pmQuestionSet) and foundational patterns (scope fields, optimistic concurrency) exist and are ready for domain entities.

Requirements addressed: ARCH-03, ARCH-04, ARCH-05, ARCH-06 (see requirements tracker)

Depends on: Phase 2 (PRISMA constraints inform field design)

Plan structure: Three sub-plans, all in Wave 1 (parallel-safe with care around shared file edits in IPmUnitOfWork.cs / MongoPmUnitOfWork.cs).

Plan Subject Files
03-01 pmAnnotationQuestion collection New entity + repository + UoW wiring
03-02 pmQuestionSet collection New entity + repository + UoW wiring
03-03 Study optimistic concurrency fix New SaveWithConcurrencyCheckAsync extension + ConcurrencyException

Research Summary

Researched: 2026-02-18 Confidence: HIGH (all patterns verified by reading source)

Phase 3 creates two new MongoDB collections and fixes the optimistic concurrency mechanism for Study writes. The codebase has a mature, well-established pattern for new collections — extending AggregateRoot<Guid>, implementing MongoRepositoryBase<T, Guid>, registering in IPmUnitOfWork / MongoPmUnitOfWork, and relying on Lamar scan conventions for DI discovery. No new libraries or frameworks are needed.

The optimistic concurrency fix is the highest-risk item. The current MongoExtensions.SaveAsync() uses IsUpsert = true on all ReplaceOne operations — a version mismatch silently creates a duplicate document rather than rejecting the write. For Study documents, this must be changed to IsUpsert = false with explicit MatchedCount == 0 checking and a new ConcurrencyException.

Primary recommendation: Follow the existing repository pattern exactly for both new collections. Add a NEW concurrency-checked save method (don't modify the existing SaveAsync) to keep blast radius zero outside Study.

Standard Stack

All infrastructure already exists in the codebase. No new packages.

Library Purpose
MongoDB.Driver 3.x (already in codebase) IMongoCollection, FilterDefinition, ReplaceOneAsync, BulkWriteAsync
SyRF.SharedKernel (internal) Entity<TId>, AggregateRoot<TId>, Audit, base interfaces
SyRF.Mongo.Common (internal) MongoRepositoryBase<T,TId>, MongoContext, MongoExtensions, MongoUnitOfWorkBase
Lamar DI scan conventions for auto-wiring
MongoDB.Bson BsonClassMap, BsonExtraElements for schema evolution 
src/libs/project-management/
├── SyRF.ProjectManagement.Core/
│   ├── Model/
│   │   ├── AnnotationQuestionAggregate/   # NEW (03-01)
│   │   │   └── AnnotationQuestion.cs      # AggregateRoot<Guid>
│   │   └── QuestionSetAggregate/          # NEW (03-02)
│   │       └── QuestionSet.cs             # AggregateRoot<Guid>
│   └── Interfaces/
│       ├── IAnnotationQuestionRepository.cs  # NEW (03-01)
│       ├── IQuestionSetRepository.cs         # NEW (03-02)
│       └── IPmUnitOfWork.cs                  # MODIFIED — add new repository properties
├── SyRF.ProjectManagement.Mongo.Data/
│   ├── Repositories/
│   │   ├── AnnotationQuestionRepository.cs   # NEW (03-01)
│   │   └── QuestionSetRepository.cs          # NEW (03-02)
│   └── MongoPmUnitOfWork.cs                  # MODIFIED — wire new repositories

src/libs/mongo/SyRF.Mongo.Common/
├── ConcurrencyException.cs                   # NEW (03-03)
└── MongoExtensions.cs                        # MODIFIED — add SaveWithConcurrencyCheckAsync

Established Patterns to Follow

Pattern 1 — New Collection Entity: Extend AggregateRoot<Guid>. The class must be in a namespace starting with SyRF.ProjectManagement so MongoContext.GetBoundedContextCode() returns the pm prefix.

namespace SyRF.ProjectManagement.Core.Model.AnnotationQuestionAggregate;

public class AnnotationQuestion : AggregateRoot<Guid>
{
    // ARCH-03 fields
    public Guid ProjectId { get; set; }       // Scope
    public Guid? OwnerId { get; set; }         // Owner
    public Guid? DerivedFromId { get; set; }   // Lineage

    public AnnotationQuestion(Guid id) : base(id) { DefaultSchemaVersion = 1; }
    public AnnotationQuestion() : base(Guid.NewGuid()) { DefaultSchemaVersion = 1; }
}

Pattern 2 — Repository Interface + Implementation: Interface extends ICrudRepository<T, Guid>; implementation extends MongoRepositoryBase<T, Guid>.

public interface IAnnotationQuestionRepository : ICrudRepository<AnnotationQuestion, Guid> { }

public class AnnotationQuestionRepository
    : MongoRepositoryBase<AnnotationQuestion, Guid>, IAnnotationQuestionRepository
{
    public AnnotationQuestionRepository(
        MongoContext context,
        RepositoryCache<Guid, AnnotationQuestion> cache,
        ILogger<AnnotationQuestionRepository> logger,
        Func<IPmUnitOfWork> unitOfWork
    ) : base(context, cache, logger, unitOfWork) { }

    public override async Task InitialiseIndexesAsync()
    {
        await CreateAscendingIndexAsync(aq => aq.ProjectId);
        await CreateAscendingIndexAsync(aq => aq.ProjectId, aq => aq.OwnerId);
    }

    public override void CreateMappings()
    {
        if (!BsonClassMap.IsClassMapRegistered(typeof(AnnotationQuestion)))
        {
            BsonClassMap.RegisterClassMap<AnnotationQuestion>(cm =>
            {
                cm.AutoMap();
                cm.SetIgnoreExtraElements(true);
            });
        }
    }
}

Pattern 3 — Wire into UnitOfWork: Add to both IPmUnitOfWork and MongoPmUnitOfWork. Use AddRepository(...) in the constructor and a typed accessor property.

Pattern 4 — Collection Naming (automatic): Class AnnotationQuestion in namespace SyRF.ProjectManagement.Core.Model.AnnotationQuestionAggregate becomes collection pmAnnotationQuestion. Class QuestionSet becomes pmQuestionSet.

Pattern 5 — DI Auto-Discovery: Lamar's WithDefaultConventions() auto-maps IXxxRepository to XxxRepository by name. No manual DI registration needed if naming convention is followed.

Anti-Patterns to Avoid

  • Embedding AggregateRoot in another document: AnnotationQuestion and QuestionSet must be standalone collections, not embedded in Project. Use Entity<Guid> for embedded objects, AggregateRoot<Guid> for collections.
  • Manual DI registration: Follow Lamar naming conventions — manual For<>().Use<>() calls are anti-pattern here.
  • Wrong namespace: Entity classes must be in SyRF.ProjectManagement.* namespace to get the pm prefix.
  • Skipping BsonClassMap registration: Always implement CreateMappings() for forward-compatible deserialization with SetIgnoreExtraElements(true).

Key Pitfalls

Pitfall 1 — IsUpsert=true creates duplicates on version mismatch (CRITICAL)

The current MongoExtensions.SaveAsync() uses IsUpsert = true on ReplaceOneAsync. When two concurrent saves target the same document:

  1. Save A succeeds (version filter matches).
  2. Save B's version filter no longer matches.
  3. Because IsUpsert = true, MongoDB inserts a new document with a different _id instead of rejecting.

Result: Duplicate Study documents with the same logical identity but different _id values. MatchedCount = 0 and ModifiedCount = 0 in ReplaceOneResult with an implicit insert.

Fix scope (Plan 03-03): Add a new SaveWithConcurrencyCheckAsync method that uses IsUpsert = false and throws ConcurrencyException when MatchedCount == 0 for existing entities. Keep the existing SaveAsync untouched to avoid blast radius.

Pitfall 2 — Naming collision with existing embedded AnnotationQuestion

There is already an AnnotationQuestion class at SyRF.ProjectManagement.Core.Model.ProjectAggregate.AnnotationQuestion (extends Entity<Guid>, embedded in Project). The new standalone class shares the name.

Fix: Place the new entity in AnnotationQuestionAggregate namespace. Use namespace aliases in MongoPmUnitOfWork.cs where both are referenced:

using AnnotationQuestionEntity = SyRF.ProjectManagement.Core.Model.AnnotationQuestionAggregate.AnnotationQuestion;

Pitfall 3 — Missing BsonClassMap registration

Complex types with enums or schema evolution may not serialize correctly without explicit BsonClassMap registration. Always implement CreateMappings().

Pitfall 4 — Forgetting to add to MongoPmUnitOfWork

DI auto-discovers the interface→implementation mapping, but the UnitOfWork's internal _crudRepositoryDictionary needs explicit AddRepository() calls. Checklist for every new repository:

  1. Add constructor parameter
  2. Call AddRepository(...) in constructor body
  3. Add typed property accessor on MongoPmUnitOfWork
  4. Add property declaration on IPmUnitOfWork

Pitfall 5 — ARCH-03 indexes must support PRISMA scope queries

Index ProjectId first in compound indexes. PRISMA queries will filter by project then aggregate across questions.

Plan 03-01: pmAnnotationQuestion collection

Files modified:

  • src/libs/project-management/SyRF.ProjectManagement.Core/Model/AnnotationQuestionAggregate/AnnotationQuestion.cs (NEW)
  • src/libs/project-management/SyRF.ProjectManagement.Core/Interfaces/IAnnotationQuestionRepository.cs (NEW)
  • src/libs/project-management/SyRF.ProjectManagement.Core/Interfaces/IPmUnitOfWork.cs (MODIFIED)
  • src/libs/project-management/SyRF.ProjectManagement.Mongo.Data/Repositories/AnnotationQuestionRepository.cs (NEW)
  • src/libs/project-management/SyRF.ProjectManagement.Mongo.Data/MongoPmUnitOfWork.cs (MODIFIED)

Objective

Create the standalone pmAnnotationQuestion MongoDB collection with its entity, repository, indexes, and UnitOfWork wiring.

Purpose: Foundation for the question versioning system. Unlike the existing embedded AnnotationQuestion (which is Entity<Guid> inside Project), this standalone collection uses AggregateRoot<Guid> and supports independent versioning, scope fields (ARCH-03), and PRISMA-compatible cross-project queries.

Task 1: Create AnnotationQuestion entity and repository interface

Create the standalone entity in a new namespace to avoid collision with the existing embedded AnnotationQuestion:

  • Namespace: SyRF.ProjectManagement.Core.Model.AnnotationQuestionAggregate
  • Class extends AggregateRoot<Guid>
  • ARCH-03 fields: ProjectId, OwnerId?, DerivedFromId?
  • Standard constructor pattern + parameterless constructor for MongoDB deserialization
  • No domain-specific fields yet — those come in Phase 4 (Question Lifecycle). This phase creates infrastructure only.

Repository interface extends ICrudRepository<AnnotationQuestion, Guid> with no extra methods (Phase 4 adds domain-specific queries).

Verify: dotnet build src/libs/project-management/SyRF.ProjectManagement.Core/SyRF.ProjectManagement.Core.csproj — must compile.

Task 2: Create repository implementation and wire into UnitOfWork

Repository implementation follows DataExportJobRepository.cs pattern exactly. Key indexes:

public override async Task InitialiseIndexesAsync()
{
    // ARCH-03 primary scope index — essential for project-level queries
    await CreateAscendingIndexAsync(aq => aq.ProjectId);
    // Compound index for owner-scoped queries within a project
    await CreateAscendingIndexAsync(aq => aq.ProjectId, aq => aq.OwnerId);
    // Lineage queries for version chains
    await CreateAscendingIndexAsync(aq => aq.DerivedFromId);
}

IPmUnitOfWork change: Add IAnnotationQuestionRepository AnnotationQuestions { get; }.

MongoPmUnitOfWork changes:

  1. Use a type alias to disambiguate from the embedded entity: 
    using AnnotationQuestionEntity = SyRF.ProjectManagement.Core.Model.AnnotationQuestionAggregate.AnnotationQuestion;
  2. Add constructor parameter IAnnotationQuestionRepository annotationQuestionRepository
  3. Call AddRepository(annotationQuestionRepository) in constructor body
  4. Add typed accessor: 
    public IAnnotationQuestionRepository AnnotationQuestions =>
    (IAnnotationQuestionRepository) GetRepository<AnnotationQuestionEntity, Guid>();

Verify: dotnet build src/services/project-management/project-management.slnf — must compile cleanly.

Success criteria

  • pmAnnotationQuestion collection infrastructure fully wired
  • ARCH-03 fields present: ProjectId, OwnerId, DerivedFromId
  • Scope-based indexes: ProjectId, ProjectId+OwnerId, DerivedFromId
  • BsonClassMap registered with SetIgnoreExtraElements(true)
  • No namespace collision with existing embedded AnnotationQuestion

Plan 03-02: pmQuestionSet collection

Files modified:

  • src/libs/project-management/SyRF.ProjectManagement.Core/Model/QuestionSetAggregate/QuestionSet.cs (NEW)
  • src/libs/project-management/SyRF.ProjectManagement.Core/Interfaces/IQuestionSetRepository.cs (NEW)
  • src/libs/project-management/SyRF.ProjectManagement.Core/Interfaces/IPmUnitOfWork.cs (MODIFIED — alongside 03-01's additions)
  • src/libs/project-management/SyRF.ProjectManagement.Mongo.Data/Repositories/QuestionSetRepository.cs (NEW)
  • src/libs/project-management/SyRF.ProjectManagement.Mongo.Data/MongoPmUnitOfWork.cs (MODIFIED — alongside 03-01's additions)

Wave note: Plans 03-01 and 03-02 both modify IPmUnitOfWork.cs and MongoPmUnitOfWork.cs. Read the current file state before editing — additions from one plan should preserve additions from the other.

Objective

Create the standalone pmQuestionSet MongoDB collection.

Purpose: QuestionSets group AnnotationQuestions into reusable, versioned sets that can be assigned to stages. Each QuestionSet has scope fields (ARCH-03) for cross-project sharing and a Published flag for lifecycle management. The Published index enables efficient querying of active sets (ARCH-06).

Task 1: Create QuestionSet entity and repository interface

  • Namespace: SyRF.ProjectManagement.Core.Model.QuestionSetAggregate
  • Class extends AggregateRoot<Guid>
  • ARCH-03 fields: ProjectId, OwnerId?, DerivedFromId?
  • Additional infrastructure field: public bool Published { get; set; } — used for indexed queries during stage activation
  • Standard constructor patterns
  • No domain-specific fields yet (question membership, version info come in Phase 4)

No naming collision concerns — QuestionSet is unique across the codebase.

Verify: Core project compiles cleanly.

Task 2: Create repository implementation and wire into UnitOfWork

Indexes (note the published-flag compound index required for ARCH-06):

public override async Task InitialiseIndexesAsync()
{
    await CreateAscendingIndexAsync(qs => qs.ProjectId);
    // ARCH-06: efficient query for "active question sets in a project"
    await CreateAscendingIndexAsync(qs => qs.ProjectId, qs => qs.Published);
    await CreateAscendingIndexAsync(qs => qs.ProjectId, qs => qs.OwnerId);
    await CreateAscendingIndexAsync(qs => qs.DerivedFromId);
}

UnitOfWork changes mirror plan 03-01: interface property, constructor parameter, AddRepository(...), typed accessor. No type alias needed since QuestionSet is unique.

Verify: dotnet build src/services/project-management/project-management.slnf — must compile cleanly.

Success criteria

  • pmQuestionSet collection infrastructure fully wired
  • ARCH-03 fields present: ProjectId, OwnerId, DerivedFromId
  • Published flag for lifecycle management
  • Indexes: ProjectId, ProjectId+Published (ARCH-06), ProjectId+OwnerId, DerivedFromId
  • BsonClassMap registered

Plan 03-03: Study optimistic concurrency

Files modified:

  • src/libs/mongo/SyRF.Mongo.Common/ConcurrencyException.cs (NEW)
  • src/libs/mongo/SyRF.Mongo.Common/MongoExtensions.cs (MODIFIED — additions only)

Objective

Add optimistic concurrency protection to Study document writes by creating a new SaveWithConcurrencyCheckAsync extension method that uses IsUpsert = false for existing entities and throws ConcurrencyException on version mismatch.

Critical design rationale: Add NEW methods rather than modifying the existing SaveAsync/Save. The existing methods are used by every entity in the system; changing their IsUpsert semantics globally is too risky. The new method is opt-in. Wiring it into Study save paths happens in Phase 4+.

Task 1: Create ConcurrencyException and SaveWithConcurrencyCheckAsync

ConcurrencyException.cs

namespace SyRF.Mongo.Common;

public class ConcurrencyException : Exception
{
    public ConcurrencyException(string message) : base(message) { }
    public ConcurrencyException(string message, Exception innerException)
        : base(message, innerException) { }
}

Add to MongoExtensions.cs (after the existing SaveAsync)

public static async Task<ReplaceOneResult> SaveWithConcurrencyCheckAsync<TAggregateRoot, TId>(
    this IMongoCollection<TAggregateRoot> collection,
    TAggregateRoot aggregateRoot,
    int schemaVersion,
    string appVersion,
    Guid? userId = null,
    IClientSessionHandle? session = null)
    where TAggregateRoot : IAggregateRoot<TId>
    where TId : notnull, IEquatable<TId>, IComparable<TId>
{
    var currentVersion = aggregateRoot.Version;
    aggregateRoot.OnSaving(schemaVersion, appVersion, userId);
    var filter = GetFilter<TAggregateRoot, TId>(aggregateRoot.Id, currentVersion);

    // For new entities (version 0 before OnSaving incremented): use upsert
    // For existing entities (version > 0): strict replace, throw on mismatch
    var isNewEntity = currentVersion == 0;

    var result = session == null
        ? await collection.ReplaceOneAsync(filter, aggregateRoot,
            new ReplaceOptions { IsUpsert = isNewEntity })
        : await collection.ReplaceOneAsync(session, filter, aggregateRoot,
            new ReplaceOptions { IsUpsert = isNewEntity });

    if (!isNewEntity && result.MatchedCount == 0)
    {
        throw new ConcurrencyException(
            $"Concurrency conflict: {typeof(TAggregateRoot).Name} with id {aggregateRoot.Id} " +
            $"was modified by another operation (expected version {currentVersion})");
    }

    return result;
}

A sync variant (SaveWithConcurrencyCheck) follows the same pattern using collection.ReplaceOne(...).

Bulk-write helper

public static ReplaceOneModel<TAggregateRoot> CreateConcurrencyCheckedReplaceModel<TAggregateRoot, TId>(
    TAggregateRoot aggregateRoot, int schemaVersion, string appVersion, Guid? userId = null)
    where TAggregateRoot : IAggregateRoot<TId>
    where TId : notnull, IEquatable<TId>, IComparable<TId>
{
    var currentVersion = aggregateRoot.Version;
    var filter = GetFilter<TAggregateRoot, TId>(aggregateRoot.Id, currentVersion);
    aggregateRoot.OnSaving(schemaVersion, appVersion, userId);
    var isNewEntity = currentVersion == 0;
    return new ReplaceOneModel<TAggregateRoot>(filter, aggregateRoot) { IsUpsert = isNewEntity };
}

Important: BulkWrite does NOT throw on individual match failures — callers building batches with this helper must compare BulkWriteResult.MatchedCount against the expected count and surface a ConcurrencyException themselves. Study bulk-write adoption is deferred to Phase 4+.

Verification

  1. dotnet build src/libs/mongo/SyRF.Mongo.Common/SyRF.Mongo.Common.csproj — compiles
  2. dotnet build src/services/project-management/project-management.slnf — no downstream breakage
  3. Existing SaveAsync and Save are completely unchanged — diff shows only additions
  4. New entity path (version == 0) uses IsUpsert = true — initial creation still works
  5. Existing entity path (version > 0) uses IsUpsert = false + MatchedCount == 0 check — concurrent modifications throw

Success criteria

  • ConcurrencyException class exists in SyRF.Mongo.Common namespace
  • SaveWithConcurrencyCheckAsync extension method available for opt-in use
  • Sync variant SaveWithConcurrencyCheck available
  • CreateConcurrencyCheckedReplaceModel helper available for bulk write scenarios
  • Existing SaveAsync/Save methods completely untouched — zero blast radius

Open Design Questions

These were flagged during research but deferred to the implementation pass.

1. Naming strategy for standalone AnnotationQuestion

Recommendation: Use the same name AnnotationQuestion in the new namespace AnnotationQuestionAggregate (consistent with how Study lives in StudyAggregate). Both can coexist via fully-qualified names / type aliases during migration.

2. Scope of optimistic concurrency fix

Recommendation: Study-specific (via opt-in to the new method) for Phase 3. A global fix can be a separate, well-tested phase. Avoids any blast radius on currently-working save paths.

3. ARCH-03 field design — Scope, OwnerId, DerivedFrom

Recommendation: Guid ProjectId for Scope, Guid? OwnerId for ownership, Guid? DerivedFromId for lineage. Formalise based on the requirements when Phase 4 work begins.

4. New entity's relationship to existing embedded AnnotationQuestion data

Recommendation: Phase 3 creates empty collections only. The schema should be compatible with eventual migration from the embedded format, but do not build migration tooling in this phase. Migration is its own phase (Phase 7 — Release 1 Data Migration).

Reference Code Locations

Key existing files used as patterns during research:

  • src/libs/kernel/SyRF.SharedKernel/BaseClasses/AggregateRoot.cs — base entity pattern
  • src/libs/kernel/SyRF.SharedKernel/BaseClasses/Audit.cs — version tracking
  • src/libs/kernel/SyRF.SharedKernel/Interfaces/ICrudRepository.cs — repository interface
  • src/libs/mongo/SyRF.Mongo.Common/MongoRepositoryBase.cs — repository base class
  • src/libs/mongo/SyRF.Mongo.Common/MongoContext.cs — collection naming, bounded context prefixes
  • src/libs/mongo/SyRF.Mongo.Common/MongoExtensions.cs — SaveAsync and version filtering (the IsUpsert = true site)
  • src/libs/mongo/SyRF.Mongo.Common/MongoLamarRegistry.cs — DI scan conventions
  • src/libs/project-management/SyRF.ProjectManagement.Core/Interfaces/IPmUnitOfWork.cs
  • src/libs/project-management/SyRF.ProjectManagement.Mongo.Data/MongoPmUnitOfWork.cs
  • src/libs/project-management/SyRF.ProjectManagement.Mongo.Data/Repositories/StudyRepository.cs — complex repository example
  • src/libs/project-management/SyRF.ProjectManagement.Mongo.Data/Repositories/DataExportJobRepository.cs — simple repository pattern
  • src/libs/project-management/SyRF.ProjectManagement.Core/Model/ProjectAggregate/AnnotationQuestion.cs — existing embedded entity (collision source)
  • src/libs/project-management/SyRF.ProjectManagement.Core/Model/StudyAggregate/Study.cs — concurrency target