Zero-ETL Multi-Modal Vector Fabric for Real-Time Fraud Detection in Government Benefits

Architecture Blueprint & Data Orchestration: The Zero-ETL Vector Fabric Paradigm

Modern fraud detection in government benefits systems faces a unique computational trilemma: the need for real-time inference, the complexity of multi-modal data (structured financial records, unstructured text from case notes, image-based identity documents, and temporal sequence logs), and the stringent governance requirements of public sector deployments. The traditional Extract, Transform, Load (ETL) pipeline, which pre-processes and normalises data into a single relational schema before analysis, introduces latency that renders real-time fraud detection impossible at scale. The solution is a fundamental architectural shift towards a Zero-ETL Multi-Modal Vector Fabric.

This architecture eliminates the need for centralised data transformation. Instead, it operates on the principle of federated vectorisation, where data remains in its native silo—or is streamed through lightweight connectors—and is immediately embedded into a high-dimensional vector space. The core engineering challenge is not simply storing vectors, but orchestrating a semantic alignment layer that can correlate an image of a forged signature with a text narrative describing an anomaly and the numerical pattern of a disbursement outlier, all within milliseconds.

Core Systems Design: The Vector Alignment Engine

The central component of this fabric is the Vector Alignment Engine (VAE). This is not a standard vector database; it is a distributed computational layer that sits upstream of the storage layer. Its function is to perform cross-modal embedding alignment during ingestion, not at query time.

System Inputs, Transformation, and Failure Modes

| Component | Input Data Type | Primary Function | Critical Failure Mode | Mitigation Strategy | | :--- | :--- | :--- | :--- | :--- | | Document Vectoriser | PDF/JPEG/PNG (Claims, IDs, Proofs) | OCR + Layout-Aware Embedding (e.g., LayoutLMv3) | Adversarial image corruption leading to false positive embedding | Implement perceptual hash pre-filtering; ingest raw bytes as fallback. | | Temporal Sequence Encoder | Log streams, transaction timestamps | Time2Vec + Transformer encoding for behavioural patterns | Concept drift in spending patterns post-policy change | Deploy online learning with sliding window retraining; trigger alert on embedding drift. | | Structured Embedder | SQL tables, CSV feeds, API payloads | Tabulard (Transformer for tabular data) encoding of numeric/categorical features | Missing data imputation introducing systematic bias | Use separate missing-value embedding token; flag for manual review. | | Cross-Modal Fuser | Output vectors from above | Attentive fusion mechanism (cross-attention layers) | Catastrophic forgetting during model update | Use elastic weight consolidation (EWC) for continual learning. |

The VAE ingests data via idempotent event streams. Each benefit claim triggers a parallel embedding process. The Document Vectoriser processes the identity proof and medical forms. The Temporal Sequence Encoder ingests the claimant’s interaction history with the portal. The Structured Embedder encodes the household income tier, region, and benefit type. These are not merged into a single feature vector at ingestion. Instead, they remain as separate high-dimensional embeddings (typically 768 or 1024 dimensions) stored in a distributed vector index alongside a shared alignment key (the unique claim ID).

The query pattern for real-time fraud detection is a hybrid vector search. A new claim arrives. It is vectorised using the same three encoders. The system then performs a multi-vector search: it finds the top-K nearest neighbours for each of the three embedding types. A cross-modal affinity score is calculated using a learned aggregation function that weights the similarity match across all three spaces. A high cohesion score (e.g., all three embeddings match a known fraud cluster) triggers an immediate block. A low cohesion score (e.g., document vector matches a known fake ID, but temporal vector matches legitimate behaviour) triggers a second-tier heuristic check.

Comparative Engineering Stacks: Vector Database Selection

The choice of vector database for this fabric is non-trivial. Government systems typically require ACID compliance for audit trails, role-based access control (RBAC), and data residency guarantees. The following table compares the three primary candidates for a production-grade Zero-ETL fabric:

| Feature | Pinecone (Serverless) | Weaviate (Self-Hosted) | Qdrant (Hybrid Deployment) | | :--- | :--- | :--- | :--- | | Deployment Model | Fully managed, cloud-only | Self-hosted (Kubernetes) or managed cloud | Self-hosted or managed cloud | | ACID Compliance | Eventual consistency; no cross-collection transactions | Object-level ACID within a single node | Strong consistency for single-shard operations; multi-shard eventual | | Multi-Modal Support | Single vector per item; requires application-side fusion | Native multi-vector support via cross-references and properties | Supports multiple named vectors per point (optimal for this use case) | | Filtering & RBAC | Metadata filtering; RBAC at workspace level | GraphQL-like filtering; fine-grained RBAC via custom modules | Payload filtering with built-in RBAC scoped to collections | | Geo-Distribution | Single region; global index with latency overhead | Multi-region via federation; requires manual sync | Native sharding and replication across regions | | Compliance Ready | SOC 2, HIPAA (on enterprise) | SOC 2, HIPAA (self-managed compliance burden) | SOC 2, GDPR built-in; FedRAMP In Progress |

Engineering Verdict: For a government benefits system requiring data residency within a specific jurisdiction (e.g., a state or province), a hybrid deployment of Qdrant with named vectors for each modality offers the optimal balance of latency, compliance, and multi-modal native support. The ability to store document_embedding, temporal_embedding, and structured_embedding as separate named vectors under a single payload ID eliminates the need for application-level fusion logic at query time.

API Specifications & Query Protocol

The interface to this fabric must be a stateless, idempotent API capable of handling 10,000+ simultaneous fraud checks per second during peak benefit disbursement cycles (e.g., the first of the month). The API is structured around a gRPC protocol with a REST gateway for legacy system integration.

Primary RPC Definition (Proto3):

syntax = "proto3";

package fraudfabric.v1;

service ZeroETLInference {
  // Ingest and vectorise a new benefit claim in real-time.
  rpc IngestClaim(IngestRequest) returns (IngestResponse);

  // Perform a multi-modal similarity search to detect fraud.
  rpc CheckFraud(FraudCheckRequest) returns (FraudCheckResponse);

  // Batch ingestion for bulk historical data (ETL fallback only).
  rpc BatchIngest(BatchIngestRequest) returns (BatchIngestResponse);
}

message IngestRequest {
  string claim_id = 1;
  bytes document_blob = 2;         // PDF or image
  string document_type = 3;        // e.g., "identity_card", "medical_note"
  repeated TemporalEvent events = 4; // sequence of interactions
  map<string, string> structured_fields = 5; // e.g., {"income": "45000", "region": "TX"}
  google.protobuf.Timestamp ingestion_time = 6;
}

message FraudCheckRequest {
  string claim_id = 1;                // Check an existing ingested claim
  float affinity_threshold = 2;       // Default 0.85
  int32 top_k_neighbours = 3;         // Default 5
}

message FraudCheckResponse {
  float fraud_risk_score = 1;         // 0.0 to 1.0
  string matched_cluster = 2;        // e.g., "synthetic_identity_ring_alpha"
  repeated NearestNeighbour neighbours = 3; // Top-K results for explainability
}

REST Gateway Endpoint (Fallback):

# OpenAPI 3.0 specification for REST fallback
paths:
  /v1/fabric/check:
    post:
      summary: "Perform real-time fraud check on ingested claim"
      requestBody:
        content:
          application/json:
            schema:
              type: object
              properties:
                claim_id:
                  type: string
                affinity_threshold:
                  type: number
                  format: float
                  default: 0.85
                top_k:
                  type: integer
                  default: 5
      responses:
        '200':
          description: "Fraud risk assessment"
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/FraudCheckResponse'

Configuration Templates: Orchestrating the Ingestion Pipeline

The Zero-ETL fabric requires a lightweight orchestrator for embedding model lifecycle management and vector index configuration. The following is a declarative configuration template for deploying the core ingestion pipeline using Temporal.io (for deterministic workflow execution) and Qdrant as the backend.

Ingestion Workflow Configuration (Temporal / YAML):

name: zero_etl_ingestion_workflow
namespace: gov-benefits-fraud
task_queue: ingestion-queue

activities:
  - name: vectorise_document
    task_queue: embedding-tasks
    retry:
      initial_interval: 1s
      maximum_attempts: 3
      backoff_coefficient: 2.0
  - name: encode_temporal_sequence
    task_queue: embedding-tasks
    retry:
      initial_interval: 500ms
      maximum_attempts: 5
  - name: embed_structured_fields
    task_queue: embedding-tasks
    non_retryable_errors: ["InvalidDataError"]
  - name: store_vectors_to_qdrant
    task_queue: vector-storage
    retry:
      initial_interval: 2s
      maximum_attempts: 3

index_config:
  qdrant:
    collection_name: benefits_fabric_v1
    vector_config:
      document_embedding:
        size: 768
        distance: Cosine
        hnsw_config:
          m: 16
          ef_construct: 200
      temporal_embedding:
        size: 512
        distance: Cosine
        hnsw_config:
          m: 12
          ef_construct: 150
      structured_embedding:
        size: 256
        distance: Dot
        hnsw_config:
          m: 8
          ef_construct: 100
    payload_schema:
      - field: claim_id
        type: keyword
        indexed: true
      - field: benefit_type
        type: integer
        indexed: true
      - field: ingestion_timestamp
        type: datetime
        indexed: false
    optimizers_config:
      default_segment_number: 2
      memmap_threshold_kb: 10000
      indexing_threshold: 10000 # Number of vectors before auto-optimise

Python Ingestor Mockup (Serverless Lambda / Cloud Function):

# Mockup for the primary ingestion Lambda function.
# This function is triggered by an SQS event (new claim submission).
import json
import logging
from typing import Dict, Any
from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct, VectorParams, Distance

# Assuming pre-loaded models (document_embedder, temporal_encoder, etc.)
from models import DocumentVectoriser, TemporalEncoder, StructuredEmbedder

logger = logging.getLogger()
logger.setLevel(logging.INFO)

# Global client initialisation (warm start optimization)
client = QdrantClient(host="qdrant-cluster.internal", port=6333, grpc_port=6334)
COLLECTION_NAME = "benefits_fabric_v1"

def lambda_handler(event: Dict[str, Any], context: Any) -> Dict[str, Any]:
    """
    Ingests a single claim from an SQS event.
    Performs Zero-ETL vectorisation.
    """
    for record in event['Records']:
        payload = json.loads(record['body'])
        claim_id = payload['claim_id']
        logger.info(f"Processing claim: {claim_id}")

        try:
            # 1. Vectorise document (PDF/Image)
            doc_bytes = payload.get('document_blob')
            doc_embedding = DocumentVectoriser.embed(doc_bytes)

            # 2. Encode temporal sequence (if present)
            temporal_events = payload.get('events', [])
            temporal_embedding = TemporalEncoder.encode(temporal_events)

            # 3. Embed structured fields (tablue data)
            structured_data = payload.get('structured_fields', {})
            structured_embedding = StructuredEmbedder.embed(structured_data)

            # 4. Persist to Qdrant as a single point with 3 named vectors
            point = PointStruct(
                id=hash(claim_id),  # Deterministic ID for idempotency
                vector={
                    "document_embedding": doc_embedding.tolist(),
                    "temporal_embedding": temporal_embedding.tolist(),
                    "structured_embedding": structured_embedding.tolist(),
                },
                payload={
                    "claim_id": claim_id,
                    "benefit_type": payload.get('benefit_type', 0),
                    "ingestion_timestamp": payload.get('ingestion_time')
                }
            )
            client.upsert(
                collection_name=COLLECTION_NAME,
                points=[point],
                wait=True
            )
            logger.info(f"Successfully ingested claim: {claim_id}")

        except Exception as e:
            logger.error(f"Failed to ingest claim {claim_id}: {str(e)}")
            # Push to dead-letter queue for forensic analysis
            # raise e  # Uncomment to trigger SQS redrive policy
            continue

    return {"statusCode": 200, "body": f"Processed {len(event['Records'])} records"}

Core Non-Shifting Principles: Semantic Integrity Over Feature Engineering

A fundamental engineering principle of this fabric is that it resists feature engineering. Traditional fraud detection systems rely on a fixed set of hand-crafted features (e.g., "number of claims in last 30 days", "average claim amount"). These features become brittle as fraud patterns evolve. The Zero-ETL Vector Fabric inverts this principle. It stores raw, un-transformed semantic representations. The feature extraction happens implicitly through the attention mechanisms of the embedding models.

This places a premium on embedding model governance. The system must maintain a versioned registry of every embedding model used. When a model is updated, the ingestion path changes, but historical data remains at its original embedding. The cross-modal affinity score accounts for this by performing an embedding space alignment check at query time. If the embeddings of a new claim were generated by a different model version than the historical neighbours, the system applies a linear projection (learned via a small calibration dataset) to map the new embeddings into the older space before computing similarity.

This technique, known as temporal embedding alignment, ensures that the vector fabric remains stable and auditable over decades—a requirement for government systems where claims can be re-investigated years later.

Failure Mode Deep Dive: The "Ghost Claim" Attack

A specific adversarial attack this architecture must withstand is the Ghost Claim: an attacker submits a claim with entirely synthetic data that is statistically plausible but semantically detached from any real identity. A traditional system might miss this because the numbers (income, age) fall within normal ranges. The Vector Fabric detects it through cross-modal inconsistency.

Attack Vector: The attacker generates a fake identity card using a generative model. The document vectoriser encodes it. The structured data (income, region) is fabricated but internally consistent. The temporal sequence, however, is minimal (a few automated clicks) or, crucially, mimics the exact temporal pattern of a previous legitimate user (replay attack).

Defence Mechanism: The VAE detects the ghost claim through the cohesion score. The document vector matches a known cluster of synthetic IDs (because generative models leave subtle embedding fingerprints). The structured vector matches a "normal" cluster. The temporal vector, however, has zero entropy (perfectly regular intervals). The cross-modal affinity score for the temporal vector against the structured and document vectors is anomalously low. The system flags the claim for manual review, not because any single modality is abnormal, but because the relationship between them is statistically improbable.

This is the core power of the multi-modal vector fabric: it models the relationships between modes, not just the values within modes.