Integrations - Enterprise Sync System

Domain: Integrations Location: api/nextango/apps/integrations/ Last Updated: 2026-07-06

Overview

The Integrations app manages bidirectional synchronization with Sanity CMS and external systems. It handles webhook processing with state machines, domain-specific sync adapters, reference resolution, field transformation, distributed locking, automatic retry with exponential backoff, outbound reference-shape validation, and monitoring.

Ownership follows authoring surface. A record born in Sanity Studio is Sanity-owned and syncs bidirectionally. A record born from a transaction or operational event is Django-owned; it sets _outbound_sync_enabled = False and has no Sanity schema. The Site model is one Django-owned case: its Sanity schema is dormant and de-registered from sync, and it is served over REST at /api/stores/sites/. Store does sync.

Documentation Quick Links

Core Documentation

Key Features

Webhook Processing

• HMAC-SHA256 verification - Secure webhook authentication
• State machine architecture - Pending → Processing → Completed → Failed
• Priority-based processing - Hardware (99) → Core (80) → Business (50) → Analytics (10)
• Deduplication - 10-minute deduplication window
• Idempotent processing - Safe to retry failed webhooks
• Async processing - Celery-based background processing

Sync Adapters

• UserSyncAdapter - User and role synchronization
• ProductSyncAdapter - Product, variant, inventory sync
• StoreSyncAdapter - Store management synchronization
• TransactionSyncAdapter - Transaction data sync
• PaymentSyncAdapter - Payment method sync
• ZoneSyncAdapter - Zone and par level sync
• TaskSyncAdapter - Task synchronization
• PromotionalCampaignSyncAdapter - Campaign sync
• HardwareSyncAdapter - Hardware priority sync (highest)

Field Transformation

• camelCase → snake_case - Automatic field name conversion
• Nested object transformation - Deep object conversion
• Array transformation - List and array handling
• Type coercion - Automatic type conversion
• Null handling - Safe null value processing
• Custom transformations - Domain-specific field mappings

Reference Resolution

• Business-meaningful references - Use store_code, sku, email instead of UUIDs
• Dual-shape resolution - A resolver accepts a sanity_id or a legacy Django-UUID primary key, with business-field lookups as a fallback
• Read-only generation - Reference generation reads only; the object's own sync is the single writer of sanity_id, which keeps generation off the transaction hot path
• Ensure-reference guarantee - SanityDataPreserver.ensure_reference_exists returns a faithful {_type: 'reference', _ref} for a referenced record, syncing the record first through its resolved domain handler when it has no sanity_id yet
• Relationship preservation - Maintain referential integrity

Outbound Reference Fidelity

• Ref-shape validator - Inspects every outbound document before it reaches Sanity. See validators.md.
• Blocking gate - A hard reference-shape violation (typeless ref, missing _key, raw object, bare-string-in-ref-array, or a schema-required ref absent on a full document) raises OutboundRefShapeError before the write, so a malformed document never leaves the process. Governed by SANITY_REF_SHAPE_ENFORCE (default on).
• Non-failing findings - Every finding writes a shape_warning audit row alongside the write; advisory findings never block.
• Schema-to-data parity - Each Sanity schema field on a Django-mirrored document type is classified, and a guard fails when a new schema field has no classification.

Reliability Features

• Distributed locking - Prevent duplicate processing
• Automatic retry - Exponential backoff (2^attempt minutes)
• Error tracking - Detailed error messages and stack traces
• Monitoring - Health checks and metrics
• Audit logging - Sync operation history, including quarantined inbound user documents held for identity reconciliation and shape_warning outbound reference-shape findings. See models.md.
• State recovery - Resume processing after failures

Bundled Sync System (Added December 2025)

• BundleRegistry - Centralized bundle configuration
• BundledSyncMixin - Efficient batched Sanity synchronization
• Transaction bundling - Bundle line items and payments with transactions
• 80-90% HTTP reduction - Fewer requests via batching
• Fallback strategies - Individual sync on validation errors

Delete Webhook Handling (Added December 2025)

• Sanity-Operation header detection - Reliable delete event identification
• Domain-specific delete handlers - Custom delete logic per domain
• Transaction-aware soft-delete - Preserve history for ProductVariants with transactions
• Cascade delete - Products cascade soft-delete to variants
• InventoryLevel deactivation - Deactivate before marking deleted
• User sync status checks - Prevent re-syncing soft-deleted users

API Endpoints

Webhook Receiver

Webhook Management

Monitoring & Health

Sync Lock Management

Analytics Sync

Full endpoint documentation: views.md

ComplianceCheck Endpoint (Stores Domain)

The ComplianceCheck model is handled by ComplianceCheckViewSet in the Stores domain but is registered at the global API router level and is processed by the integrations sync infrastructure (webhook processing in tasks/webhook_processing.py routes ComplianceCheck documents to the zone_sync_adapter).

ViewSet: ComplianceCheckViewSet in api/nextango/apps/stores/views.py

Lookup field: check_id (string slug, not UUID primary key)

Request shape (POST/PUT): Accepts fields from ComplianceCheckSerializer, including FKs to Zone and Employee, compliance outcome, notes, and audit timestamps.

Response shape: Serialized ComplianceCheck record with zone and employee select_related expansion.

Authentication: Inherits DEFAULT_PERMISSION_CLASSES (IsActiveUser in production, AllowAny in DEBUG mode).

Integrations role: ComplianceCheck Sanity documents are synced inbound via the zone_sync_adapter. Outbound sync (Django → Sanity) is handled via signals in stores/signals.py.

Operational Kill-Switches

Two environment variables disable sync processing for maintenance or load testing:

Set these in .env or the container environment. No restart is needed if your settings module reads them dynamically.

Architecture Highlights

Webhook Processing Pipeline

1. Ingestion       → HMAC-SHA256 verification
2. Buffering       → Redis Stream → List → Database
3. Persistence     → WebhookEvent with state machine
4. Routing         → Domain adapter selection
5. Processing      → Field transformation + reference resolution
6. Sync            → Django model create/update
7. Audit           → SyncAuditLog entry

Read more: views.md, tasks.md

Sync Adapter Pattern

Each domain has a dedicated sync adapter implementing:

• handle_create(payload, sync_context) - Create new records
• handle_update(payload, sync_context) - Update existing records
• handle_delete(payload, sync_context) - Soft delete records
• validate_payload(payload) - Domain-specific validation
• resolve_references(payload) - Reference resolution

Read more: adapters.md

Reference Resolution System

Business-meaningful references instead of UUIDs:

# Sanity sends:
{
  "store": {"storeCode": "NYC001"},
  "product": {"sku": "PROD-123"}
}

# ReferenceResolver converts to:
{
  "store_id": "uuid-of-nyc001-store",
  "product_id": "uuid-of-prod-123"
}

Read more: sync_services_reference_logic.md

SyncContext Pattern

Shared context object passed through sync pipeline:

• Tracks sync operation metadata
• Maintains resolution cache
• Stores validation results
• Accumulates audit information

Read more: sync_context.md

Priority-Based Processing

Webhooks processed by priority to ensure critical updates first:

• 99 (Hardware) - Terminal configurations, critical settings
• 80 (Core) - Users, stores, products
• 50 (Business) - Transactions, payments, zones
• 10 (Analytics) - Reports, analytics, logs

Read more: models.md

Bundled Sync Architecture (Added December 2025)

Efficient sync by batching related entities into single Sanity requests:

from nextango.apps.integrations.sync.bundle_registry import BundleRegistry

# Configuration per domain
BundleRegistry._bundles = {
    'transactions.SaleTransaction': {
        'enabled': True,
        'max_bundle_size': 50,
        'max_payload_mb': 5,
        'related_entities': [
            {'relation': 'line_items', 'max_per_bundle': 40},
            {'relation': 'payment_records', 'max_per_bundle': 9}
        ]
    }
}

# Check if bundling is enabled
BundleRegistry.is_bundling_enabled('transactions.SaleTransaction')  # True

# Runtime toggle (emergency rollback)
BundleRegistry.disable_bundling('transactions.SaleTransaction')

Benefits:

• Reduces Sanity API calls by 80-90%
• Atomic transaction + line items + payments sync
• Registry-driven configuration per domain
• Fallback to individual sync on validation errors

Getting Started

1. Understand the Models

Start with models.md to understand:

• WebhookEvent state machine (pending → processing → completed)
• SyncAuditLog (operation tracking and history)
• SyncLock (distributed locking mechanism)
• Priority levels and retry logic

2. Review Webhook Pipeline

Read views.md to understand:

• Webhook receiver implementation
• HMAC signature verification
• Payload buffering and persistence
• State transitions

3. Explore Sync Adapters

Check adapters.md to learn:

• Domain-specific sync implementations
• Adapter pattern structure
• Create/update/delete handling
• Validation and error handling

4. Study Reference Resolution

Review sync_services_reference_logic.md for:

• Business reference patterns
• Lookup strategies
• Fallback mechanisms
• Cache optimization

5. Understand Field Transformation

Read services.md to learn:

• camelCase to snake_case conversion
• Nested object transformation
• Array and list handling
• Custom transformation rules

Related Domains

• Core - Base models and validators
• Users - User sync adapter
• Stores - Store sync adapter
• Products - Product sync adapter
• Transactions - Transaction sync adapter
• Payments - Payment sync adapter
• Zones - Zone sync adapter
• Tasks - Task sync adapter
• Promotions - Campaign sync adapter

Key Architectural Decisions

1. State machine pattern - Clear webhook lifecycle management
2. Priority-based processing - Critical updates processed first
3. Business references - Use meaningful identifiers, not UUIDs
4. Adapter pattern - Domain isolation and maintainability
5. Distributed locking - Prevent duplicate processing in multi-worker setup
6. Automatic retry - Exponential backoff for transient failures
7. Async processing - Celery for non-blocking webhook handling
8. SyncContext pattern - Shared context reduces parameter passing

Common Use Cases

• Sanity CMS sync - Automatic synchronization on content changes
• Reference resolution - Convert business references to Django FKs
• Field transformation - Convert camelCase to snake_case
• Webhook retry - Automatic retry of failed webhooks
• Priority processing - Hardware updates before analytics
• Distributed sync - Multi-worker webhook processing
• Audit tracking - Complete sync operation history
• Error monitoring - Track and debug sync failures

Service Methods

ReferenceResolver

Reference Resolution:

• resolve_store_reference(store_data) - Resolve store by code
• resolve_product_reference(product_data) - Resolve product by SKU
• resolve_user_reference(user_data) - Resolve user by email
• resolve_zone_reference(zone_data) - Resolve zone by name/store
• resolve_with_fallback(field_name, strategies) - Multi-strategy resolution

Cache Management:

• cache_resolution(key, value) - Cache resolved reference
• get_cached_resolution(key) - Retrieve cached resolution
• clear_resolution_cache() - Clear resolution cache

FieldTransformationUtils

Field Transformation:

• camel_to_snake(camel_str) - Convert camelCase to snake_case
• transform_object(obj) - Transform nested object
• transform_array(arr) - Transform array elements
• transform_payload(payload) - Transform entire payload

Custom Transformations:

• apply_custom_mapping(field, value, mapping) - Apply custom field mapping
• coerce_type(value, target_type) - Type coercion
• handle_null_values(obj, strategy) - Null value handling

SyncContext

Context Management:

• create_sync_context(webhook_event) - Initialize context
• add_metadata(key, value) - Add context metadata
• get_metadata(key) - Retrieve metadata
• add_audit_entry(operation, result) - Track operation
• get_audit_trail() - Get complete audit trail

Full documentation: services.md

Next Steps

1. Start with models.md - Understand state machines
2. Then views.md - Learn webhook pipeline
3. Review adapters.md - Explore sync adapters
4. Study sync_services_reference_logic.md - Master reference resolution
5. Check sync_context.md - Understand SyncContext pattern
6. Review tasks.md - Learn async processing
7. Explore services.md - See utilities and helpers