Offline-First Field Inspection App for Environmental Health with CRDT-Based Data Sync

Architecture Blueprint & Data Orchestration for Offline-First Field Inspection Systems

The foundational architecture for an offline-first field inspection application targeting environmental health domains must resolve three core engineering tensions: local data sovereignty during connectivity loss, conflict-free merging upon reconnection, and deterministic state convergence across distributed peers. The dominant paradigm achieving this without a central coordinating server during offline periods is Conflict-Free Replicated Data Types (CRDTs), specifically operation-based or state-based variants that guarantee eventual consistency without requiring consensus.

Core Data Synchronization Model: CRDT Selection Matrix

The choice between operation-based and state-based CRDTs determines the entire sync protocol, bandwidth consumption, and storage footprint. For field inspection apps where payloads include multimedia attachments and structured form data, the following table clarifies the trade-offs:

| CRDT Property | State-Based (CvRDT) | Operation-Based (CmRDT) | |---|---|---| | Sync payload size | Entire state (scales with data volume) | Only operations (smaller, O(log n)) | | Conflict resolution | Idempotent merge function required | Requires total-order broadcast or causal context | | Bandwidth efficiency | Poor for large inspection forms | Good for incremental field updates | | Partial sync support | Must transmit full state | Can sync individual operations | | Suitable for | Small metadata, counters, labels | Large inspection records, nested fields | | Failure tolerance | High (no operational log needed) | Requires reliable delivery layer |

For an environmental health inspection app managing inspections with 50–200 fields per form, photographs, GPS coordinates, and timestamps, a hybrid approach yields optimal results: use state-based CRDTs for metadata (inspection status flags, assignment state) and operation-based CRDTs for the inspection record body. The synchronization layer must implement a delta-state exchange protocol to avoid resending the full state during every sync cycle.

CRDT Data Structure for Inspection Records

The inspection record constitutes the central unit of work. Each record must be modeled as a LWW-Register (Last-Writer-Wins Register) embedded within a Map CRDT for field-level resolution. The following data model defines the core structure:

{
  "inspection_id": "uuid-v7",
  "site_id": "uuid-v7",
  "assigned_inspector": "string (email hash)",
  "status": {
    "type": "lww-register",
    "value": "draft | in_progress | completed | reviewed | signed_off"
  },
  "fields": {
    "type": "map-crdt",
    "entries": {
      "water_sample_turbidity": {
        "type": "lww-register",
        "vector_clock": { "node-xyz": 3 },
        "value": 12.4
      },
      "inspection_notes": {
        "type": "lww-register",
        "vector_clock": { "node-abc": 5 },
        "value": "Slight discoloration observed in downstream sample."
      }
    }
  },
  "media_attachments": {
    "type": "grow-only-set",
    "items": [
      "s3://inspections/uuid/photo_001.jpeg",
      "s3://inspections/uuid/photo_002.heic"
    ]
  },
  "geo_points": {
    "type": "grow-only-set",
    "items": [
      { "lat": 51.5074, "lng": -0.1278, "timestamp": 1716969600 },
      { "lat": 51.5075, "lng": -0.1279, "timestamp": 1716973200 }
    ]
  }
}

The vector clock approach at each field level ensures that when two inspectors modify the same inspection simultaneously offline, the merge operation applies the later chronological update based on wall-clock time combined with node priority. This design choice intentionally sacrifices strong consistency for high availability and partition tolerance—dictated by the AP (Availability and Partition Tolerance) trade-off from the CAP theorem.

Sync Protocol Design: WebRTC with Causal Broadcast

The synchronization layer must operate over unreliable network conditions. WebRTC data channels provide the ideal transport for peer-to-peer CRDT sync, as they offer low-latency, secure, and NAT-traversable connections without requiring a persistent central server. For scenarios where direct P2P connections fail (symmetric NAT, firewall restrictions), a TURN relay server acts as a fallback, though the system should prefer selective forwarding.

Connection Establishment & Stream Negotiation

Each client node maintains a PeerConnectionManager that orchestrates the following lifecycle:

1. Discovery Phase: When a device regains connectivity, it registers with the discovery service (a lightweight WebSocket or HTTP endpoint) announcing its presence and the set of inspection IDs it holds.
2. Peer Matching: The central coordinator matches peers that hold overlapping inspections requiring sync. In a purely serverless design, each peer advertises its inspection ledger via DHT (Distributed Hash Table) or mDNS-local network discovery.
3. SDP Exchange: ICE candidates and SDP offers/answers flow through the signaling channel (WebSocket or embedded signaling via the P2P data channel bootstrap).
4. CRDT State Dump: Upon establishing a DataChannel, peers exchange their latest vector clocks for each inspection record. Only the delta—fields with divergent clocks—is transmitted.

Sync Protocol Message Format

Every sync message must carry causal metadata to enable the receiver to apply operations in the correct order and detect concurrent edits:

message CRDTSyncMessage {
  string inspection_id = 1;
  uint64 message_sequence = 2; // monotonic per node
  string node_id = 3;
  map<string, uint64> vector_clock = 4; // field -> timestamp
  oneof payload {
    StateSnapshot full_state = 5;
    OperationBatch operations = 6;
    SyncAck acknowledgment = 7;
  }
}

The receiving peer applies operations only if the sender’s vector clock for each field is strictly greater than the local clock. If concurrent edits are detected (both nodes modified the same field independently), the LWW strategy selects the highest wall-clock timestamp, and on ties, the lexicographically smaller node ID breaks the conflict. This deterministic tie-breaking ensures all nodes converge to the same final state without requiring external arbitration.

Media Synchronization Strategy

Environmental health inspections generate substantial photographic evidence—often 5–20 high-resolution images per inspection. Synchronizing these over unreliable mobile networks demands a separate prioritization layer. The system implements a progressive media sync approach:

• Thumbnail-first: Immediately after capture, the app generates a 256x256 thumbnail and attaches it to the CRDT field metadata. Inspectors reviewing the inspection can see thumbnails even before full resolution images sync.
• Bandwidth-Aware Queuing: The sync manager monitors available bandwidth (estimated via WebRTC statistics API) and prioritizes thumbnail sync over full-resolution images. Only when the critical inspection data (text fields, GPS, results) has been fully synchronized does the system allocate bandwidth to media uploads.
• Content-Addressable Storage: Each media file is SHA-256 hashed. Peers first exchange the hash, and only if the receiving peer lacks the content does actual transfer occur. This prevents redundant downloads when multiple inspectors hold the same image (for example, a water sample photo taken by two devices).

Failure Mode: Stale Vector Clock

A known failure mode in operation-based CRDTs with LWW registers is the stale clock problem. If a device has been offline for weeks and then syncs with a peer that has advanced its vector clock far beyond, the stale device may incorrectly apply its older operations, believing them to be concurrent. To mitigate this, each sync session begins with a clock validation step:

def validate_clock(peer_clock, local_clock):
    # If peer's clock for field X is more than 30 days ahead of local clock,
    # treat local operations as stale and drop them
    for field, peer_ts in peer_clock.items():
        local_ts = local_clock.get(field, 0)
        if (peer_ts - local_ts) > MAX_CLOCK_DRIFT_SECONDS:
            # Force discard of local pending operations for this field
            local_pending[field] = []
            local_clock[field] = peer_ts
    return local_clock

This threshold-based invalidation prevents unbounded state divergence while remaining safe for the inspection domain: a 30-day-old unsynced edit is almost certainly irrelevant compared to the current inspection state.

Database Layer: Embedded SQLite with CRDT Interface

The local persistence layer must store CRDT state in a way that supports fast querying, indexing, and partial sync. SQLite, running in WAL (Write-Ahead Logging) mode, provides the transactional guarantees needed. The schema separates CRDT metadata from application data:

CREATE TABLE inspections (
    inspection_id TEXT PRIMARY KEY,
    site_id TEXT NOT NULL,
    assigned_inspector TEXT,
    vector_clock TEXT NOT NULL, -- serialized JSON map
    status TEXT NOT NULL DEFAULT 'draft',
    data_json TEXT NOT NULL, -- serialized map-CRDT state
    created_at INTEGER NOT NULL,
    updated_at INTEGER NOT NULL
);

CREATE TABLE crdt_operations (
    operation_id INTEGER PRIMARY KEY AUTOINCREMENT,
    inspection_id TEXT NOT NULL,
    node_id TEXT NOT NULL,
    field_path TEXT NOT NULL,
    operation_type TEXT NOT NULL, -- 'assign', 'delete', 'merge'
    value_blob BLOB,
    logical_timestamp INTEGER NOT NULL,
    applied_at INTEGER NOT NULL DEFAULT (strftime('%s','now'))
);

CREATE TABLE media_sync_queue (
    media_hash TEXT PRIMARY KEY,
    inspection_id TEXT NOT NULL,
    file_path TEXT NOT NULL,
    file_size INTEGER NOT NULL,
    sync_status TEXT NOT NULL DEFAULT 'pending',
    priority INTEGER NOT NULL DEFAULT 5
);

CREATE INDEX idx_inspections_site ON inspections(site_id);
CREATE INDEX idx_crdt_inspection_time ON crdt_operations(inspection_id, logical_timestamp);
CREATE INDEX idx_media_sync_priority ON media_sync_queue(priority, sync_status);

The data_json column stores the serialized map-CRDT state in a compact binary format (Protocol Buffers or MessagePack), enabling the app to reconstruct the inspection record without replaying the entire operation log. The crdt_operations table provides the granular operation log needed for partial sync and replay-based merging.

Offline Storage Estimations & Garbage Collection

A field inspector might conduct 3–8 inspections per day offline, each with 10–30 photos (average 3 MB per photo after compression). Over a two-week offline period (common in remote environmental health inspections), the device must manage:

| Data Type | Per Inspection | Per Day (5 inspections) | 14-Day Offline Total | |---|---|---|---| | CRDT metadata (text) | 50 KB | 250 KB | 3.5 MB | | Thumbnails (256px) | 500 KB | 2.5 MB | 35 MB | | Full-resolution photos | 30 MB | 150 MB | 2.1 GB | | Total | 30.55 MB | 152.75 MB | 2.14 GB |

The system implements a least-recently-used (LRU) eviction on full-resolution photos when storage exceeds 80% of available space (typically 4 GB allocation on modern devices). The app always retains CRDT metadata and thumbnails, which are sufficient for completing the inspection form offline. Full-resolution photos are re-downloaded from peers or S3 upon synchronization.

Security Model for Peer-to-Peer State Exchange

Offline-first systems that synchronize peer-to-peer must address the unique security threat of state injection attacks—a malicious peer could craft a vector clock claiming a high timestamp and inject arbitrary data. The defense layers are:

1. End-to-End Encryption: Each inspection record is encrypted with a symmetric key derived from the inspection ID and a server-secret salt, distributed during initial assignment. Peers never exchange plaintext CRDT state.
2. Peer Authentication: Before accepting CRDT operations, each peer verifies the sender’s X.509 certificate issued by the organization’s PKI. The certificate embeds the inspector’s role and organization ID.
3. Operation Signing: Every CRDT operation includes an HMAC-SHA256 signature over (inspection_id, field_path, value, logical_timestamp) using the inspector’s private key. Replaying old operations is detected by a monotonically increasing operation counter per peer.
4. Trusted Timestamp Oracle: While offline, the device cannot contact an NTP server. To mitigate timestamp manipulation, the app uses a monotonic hardware clock combined with a drift detection algorithm. If a peer’s claimed timestamp is more than 1 hour in the future relative to the local hardware clock, the operation is rejected.

Network Failure Scenarios & Recovery

The system must handle a spectrum of network failures. The following table defines the behavior for each failure mode:

| Failure Scenario | System Behavior | Data Loss Risk | Recovery Action | |---|---|---|---| | Complete offline (0 bars) | All operations stored locally in WAL; CRDT state maintained in SQLite | None | On reconnect, initiate full sync with peer discovery | | Partial connectivity (packet loss >30%) | Transmit CRDT metadata only; defer media sync; use binary exponential backoff for retry | None (media queued) | Resume transfer when packet loss drops below 15% | | Peer disconnects mid-sync | Detect via WebRTC DTLS timeout (default 30s); mark sync as incomplete; resume from last acknowledged operation sequence | Minimal (duplicate operations are idempotent) | On peer re-discovery, resend unacknowledged operations | | Symmetric NAT (no P2P possible) | Fall back to TURN relay; operations are forwarded through central relay but still end-to-end encrypted | None | TURN relays are ephemeral; no data persists on relay | | Power loss mid-write | SQLite WAL ensures atomicity; on restart, replay incomplete transactions; CRDT state may have one missing operation that gets resolved on next sync | Statistically negligible | Peer reconciliation on next sync fills any gap |

Configuration Template: Sync Manager Initialisation

The following YAML configuration defines the sync parameters that should be tuned based on the expected inspection environment (urban vs remote wilderness):

sync_manager:
  mode: hybrid_crdt
  state_based_metadata: true        # Use CvRDT for inspection metadata
  operation_based_body: true        # Use CmRDT for inspection fields
  
  # Peer Discovery
  discovery:
    type: hybrid                    # WebSocket signaling + local mDNS
    signaling_server: "wss://signaling.intelligent-ps.store/v1"
    mDNS_service: "_inspector-sync._tcp.local"
    peer_timeout_seconds: 120
    
  # CRDT Parameters
  crdt:
    max_clock_drift_seconds: 2592000  # 30 days
    tie_breaker: "lexicographic_smallest"
    lww_prefer_ties: "node_id"        # Lower node ID wins tie
    
  # Sync Priorities
  priorities:
    metadata: 1                       # Highest
    text_fields: 2
    thumbnails: 3
    full_resolution_photos: 4         # Lowest
    
  # Flow Control
  backoff:
    initial_interval_ms: 1000
    multiplier: 2.0
    max_interval_ms: 60000            # 1 minute
    max_retries: 10

  # Security
  encryption:
    algorithm: "AES-256-GCM"
    key_derivation: "HKDF-SHA256"
    signature_algorithm: "ECDSA-P256"
    
  # Storage Management
  storage:
    max_local_inspections: 200
    max_full_photo_gb: 3.0
    eviction_threshold_percent: 80

This configuration ensures that the sync manager adapts to bandwidth constraints while preserving data integrity across prolonged offline periods—a critical requirement for environmental health inspections in remote regions where connectivity is intermittent and power supplies may be unreliable.

Standardisation Bodies & Protocol Dependencies

The architecture relies on established standards to ensure interoperability and auditability:

• WebRTC 1.0 (W3C Recommendation): DataChannel API for P2P transfer, ICE for NAT traversal, DTLS for encryption.
• CRDT formalization (Baidu’s YATA or Automerge’s RGA): Operation-based CRDT algorithm selected for its O(n) merge complexity and support for nested Map/LWW composition.
• SHA-256 (FIPS 180-4): Content addressing for deduplication.
• SQLite WAL mode: Atomic commit even under power loss.
• Protocol Buffers 3: Wire-efficient serialization for CRDT sync messages; smaller than JSON by 3–5x for typical inspection payloads.

The combination of these technologies, deployed through a careful architecture of bounded approximations, vector clocks, and peer-to-peer WebRTC channels, creates an inspection platform that operates reliably in the most challenging connectivity environments. The entire system can be deployed through the Intelligent-Ps SaaS Solutions suite (https://www.intelligent-ps.store/), which provides pre-built orchestration layers for CRDT management, sync configuration, and peer discovery—reducing the implementation complexity from a year-long engineering effort to a configurable deployment.