Products - Smart Filter Service

Location: api/nextango/apps/products/smart_filter_service.py Last Updated: 2026-07-06

Overview

The Smart Filter Service provides Elasticsearch-based product filtering with dynamic availability calculation. It emulates the Gatsby filterProcessing.js logic to calculate which filter options would return results based on the current filter state.

This enables the UI to:

• Show accurate counts per filter option
• Disable or gray out options that would return 0 products
• Provide instant feedback as users select filters

The service backs ProductSmartFilterAPIView (GET /search/smart-filter/, AllowAny). The view routes between two modes: browsing requests with a store_id, no search query, and none of brand, category, productType, collection, attributes, on_sale, or low_stock active serve aggregations from a pre-generated per-store manifest file (_MANIFEST_BYPASS_FILTERS, views.py:1312-1313); everything else runs real-time Elasticsearch aggregations.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                 Frontend Filter UI                          │
│        (Brand, Category, Collection, Attributes)            │
└──────────────────────┬──────────────────────────────────────┘
                       │ GET /search/smart-filter/
┌──────────────────────▼──────────────────────────────────────┐
│             ProductSmartFilterAPIView                       │
│   (views.py: hybrid manifest/real-time routing,             │
│    capability-keyed inventory strip)                        │
└──────────────────────┬──────────────────────────────────────┘
                       │ execute_smart_search()
┌──────────────────────▼──────────────────────────────────────┐
│              SmartFilterService                             │
│                                                             │
│   1. build_search_query() - Build ES query with filters     │
│   2. add_aggregations() - Add count aggregations            │
│   3. get_available_options() - Calculate smart availability │
│   4. _flatten_products_to_variants() - Build grid rows      │
│   5. _transform_aggregations() - Build filter metadata      │
└──────────────────────┬──────────────────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────────────────┐
│                 Elasticsearch                               │
│              (ProductDocument index)                        │
└─────────────────────────────────────────────────────────────┘

Key Concepts

Smart Aggregations

Traditional aggregations show counts for the current result set. Smart aggregations calculate what options would yield results if selected.

Example:

• User filters by Brand: "Nike"
• Traditional: Shows only categories in Nike products
• Smart: Shows which categories would have Nike products (grays out empty ones)

Post-Filter Aggregations

For each filter type, the service runs a query that:

1. Excludes only that filter type
2. Includes all other active filters
3. Returns which options would yield results

This tells the UI which options to enable vs disable.

Inventory Visibility

Raw per-store quantities are capability-gated at the view boundary. Staff callers (employee or manager role) and POS-session principals receive the full row shape. Everyone else, including anonymous callers, customers, and the storefront service principal, gets stock_available stripped from each result row. Binary availability (is_in_stock) and store_name survive the strip. The strip happens at response-building time in the view; the service output and the ES query shape are unchanged.

API Reference

SmartFilterService Class

Static service class with no instance state. All methods are @staticmethod.

execute_smart_search()

Main entry point for product search with smart filtering.

Signature:

@staticmethod
def execute_smart_search(
    query: Optional[str] = None,
    filters: Optional[Dict[str, Any]] = None,
    page: int = 1,
    page_size: int = 20,
    skip_aggregations: bool = False
) -> Dict[str, Any]

Parameters:

Filter Dictionary Structure:

{
    'search': 'sneaker',              # Text search
    'store_id': 'store-uuid',         # Store context for inventory
    'brand': ['brand-uuid-1'],        # Brand IDs or sanity_ids
    'category': ['cat-uuid-1'],       # Category IDs
    'productType': ['pt-uuid-1'],     # ProductType IDs or sanity_ids
    'collection': ['coll-uuid-1'],    # Collection IDs or sanity_ids
    'attributes': {                   # Attribute filters
        'Color': ['Red', 'Blue'],
        'Size': ['Large']
    },
    'on_sale': True,                  # Only products with an on-sale variant
    'low_stock': True,                # Only products with a low-stock variant
    'collapse': True,                 # Collapse to one row per product (grid mode)
    'sort': 'price-asc'               # Sort strategy
}

Sort Options:

Returns:

{
    'products': [
        {
            'id': 'product-uuid_SKU123',
            'product_name': 'Nike Air Max',
            'product_slug': 'nike-air-max',
            'name': 'Black / Size 10',
            'product_short_description': '...',
            'sku': 'SKU123',
            'price': 149.99,
            'sale_price': 119.99,
            'effective_price': 119.99,
            'on_sale': True,
            'is_in_stock': True,
            'stock_available': 5,    # stripped for non-staff/non-POS callers
            'featured': True,
            'store_name': 'Downtown Store',
            'image_urls': ['https://...']
        }
    ],
    'total': 150,            # Total matching products (ES hit count)
    'total_variants': 45,    # Rows emitted for this page
    'aggregations': {
        'brands': [...],
        'categories': [...],
        'productTypes': [...],
        'collections': [...],
        'attributes': {'featured': [...], 'regular': [...]}
    },
    'page': 1,
    'page_size': 20,
    'filters': {...}
}

Edge cases:

• Pagination applies at the product level. In the default flat mode each product can emit several variant rows, so a page can hold more rows than page_size.
• Aggregations run on the unpaginated query, so counts reflect the full filtered result set, not the current page.
• With skip_aggregations=True the aggregations key is an empty dict.
• Elasticsearch failures are logged and re-raised. The view converts them to a 500 response.

build_search_query()

Builds the Elasticsearch query with filters applied. Accepts query, filters, exclude_filter_types (a list of filter types to leave out, used by the post-filter availability queries), and an optional attr_id_map (attribute name to Sanity id set, resolved once per request by the caller and reused across calls). Returns an elasticsearch_dsl.Search object.

Filter Processing:

1. Hidden Products: Always excluded (hidden=False)

2. Store Inventory: Double-nested query for store-specific availability:

# Only show products available at the store
variants.inventory_by_store.store_id = store_id
variants.inventory_by_store.is_available = True

3. On Sale (on_sale=True): Restricts to products with at least one on-sale variant via the top-level denormalized on_sale_variants_count > 0 field. No nested query needed.

4. Low Stock (low_stock=True): Restricts to products with a store-available variant whose live variants.inventory_by_store.stock_on_hand falls in [1, 10] (LOW_STOCK_THRESHOLD = 10). Scoped to the selected store when store_id is present. The digital-inventory sentinel sits far above the ceiling, so digital variants never match. Note that the code carries two digital sentinel values, a known code inconsistency: the serializer and index convention is 999999 (serializers.py:142), while the flattener's no-store fallback uses 999 (smart_filter_service.py:688, shown under Store Inventory Logic below).

5. Text Search:

   • Short queries (3 chars or fewer): prefix matching for autocomplete
   • Longer queries: fuzzy matching with field boosting

6. Filter Resolution: Brand, productType, and collection values are resolved against both the Django UUID and the sanity_id, then matched in ES by sanity_id. Category values resolve to the ProductType sanity_ids belonging to those categories.

7. Attributes: Each active attribute name first resolves to the set of Sanity ids sharing that name (_resolve_attribute_name_ids; a name can map to more than one id when Studio holds duplicate attribute documents). The query then matches on attribute identity and value together, not value alone, because the same value can appear under two different attributes (for example the same color name under both a "Frame Finish" and a "Frame Color" attribute). Matching double-nests: an inner query on variants.attributes requires both variants.attributes.attribute in the resolved ids and variants.attributes.value in the requested values, and every attribute name's inner query is correlated inside a single outer variants nested query, so one variant must satisfy every active attribute name rather than the constraints spreading across different variants of the same product. Names combine with AND; values within a name combine with OR. An attribute name that resolves to no id fails closed: its clause matches nothing rather than falling back to a value-only match.

add_aggregations()

Adds terms aggregations for brands, categories, productTypes, and collections (size 100 each), plus a double-nested aggregation for attribute values: an outer variants nested aggregation containing an inner variants.attributes nested aggregation that buckets on variants.attributes.value (size 200). The double nesting mirrors the structure the attribute filter itself queries. Takes the Search object and the current filter state; returns the Search with aggregations attached.

get_available_options()

Calculates which filter options would return results.

Signature:

@staticmethod
def get_available_options(
    filters: Dict[str, Any],
    attr_id_map: Optional[Dict[str, set]] = None
) -> Optional[Dict[str, set]]

attr_id_map lets a caller that already resolved attribute names to Sanity ids (for example execute_smart_search) pass the map through instead of re-resolving it for every per-filter-type query.

Returns:

{
    'brands': {'brand-sanity-id-1', 'brand-sanity-id-2'},
    'productTypes': {'pt-sanity-id-1'},
    'collections': {'coll-sanity-id-1', 'coll-sanity-id-2'},
    'categories': {'cat-sanity-id-1'},
    'attributes': {'Red', 'Blue', 'Large'}  # Flat set of available values
}

Returns None only when there are no active filters beyond store_id and sort and no store_id. With a store_id alone it still runs, so availability reflects actual store inventory.

Algorithm:

For each filter_type in [brand, category, productType, collection, attributes]:
    1. Build query with ALL filters EXCEPT this type
    2. Run aggregation for this type only
    3. Extract available sanity_ids from aggregation buckets
    4. Store in available[filter_type] set

Edge case: a failed per-type query is logged and leaves that type's set empty, which downstream marks every option of that type unavailable.

apply_availability_flags()

Applies available boolean flags to an aggregations dict loaded from a manifest, using the option sets from get_available_options(). Each brand, category, productType, and collection entry is flagged by sanity_id membership; attribute entries are flagged per color or option value. Returns the updated aggregations dict, or the input unchanged when available_options is None. Used by the view's manifest mode so stale manifests never show unavailable options as selectable.

generate_store_manifest()

Module-level function that builds (and unless dry_run=True, writes) the filter manifest for one store. It runs execute_smart_search with only the store filter at page_size=1 to capture aggregations, then writes store_<store_id>.json into manifests_dir with the store id, store name, generation timestamp, and aggregations.

Parameters: store (Store instance), manifests_dir (target directory), dry_run (skip the write).

Returns: a summary dict with store_id, file_size, and per-type option counts.

Shared by the generate_filter_manifests management command (serial CLI path) and the manifest-generation Celery chord (see Tasks).

_flatten_products_to_variants()

Transforms Elasticsearch product documents into grid rows for the frontend. Two modes, selected by the collapse flag:

Flat mode (collapse=False, default; homepage carousels): emits one row per store-available variant, with no card_kind field. Attribute and merchandising filters are not re-applied at emit time; ES already selected the products.

Collapse mode (collapse=True; the /products/ grid): per product, resolves the variants matching the active filters (store availability, attribute match, search, and the on_sale/low_stock flags), then emits one row. With more than one matching variant the row is a product card (card_kind='product') carrying a price range; with exactly one match it is that variant's row tagged card_kind='variant'. Product-level facets never split a product into multiple rows; a variant-discriminating attribute or merchandising filter narrows the card to the matching variants.

Product card shape (collapse mode, >1 match):

{
    'card_kind': 'product',
    'id': 'product-uuid',
    'product_name': 'Nike Air Max',
    'product_slug': 'nike-air-max',
    'product_short_description': '...',
    'matched_variant_count': 3,
    'price_min': 119.99,
    'price_max': 169.99,
    'on_sale': True,        # any matched variant on sale
    'featured': False,      # any matched variant featured
    'is_in_stock': True,
    'store_name': 'Downtown Store',
    'image_urls': ['https://...']
}

Behavior common to both modes:

• When the search query matches product-level fields (name, short description, product notes), all variants pass. Otherwise variants are pruned to those whose name or SKU contains the query.
• Effective price is the sale price when the variant is on sale and a sale price exists, otherwise the regular price.

Store Inventory Logic:

if store_id:
    # Find inventory for this store; skip the variant when no record
    # exists or is_available is False
    stock_available = max(0, stock_on_hand - stock_reserved)
else:
    # No store context - use the immutable stockInitial snapshot
    is_in_stock = digital_inventory or (stockInitial > 0)
    stock_available = stockInitial    # 999 for digital variants

The 999 fallback on this path differs from the 999999 serializer and index convention; see the low-stock note above for the known code inconsistency.

Edge cases:

• Collapse-mode attribute matching is value-based, mirroring the ES query: per attribute name AND, within a name OR.
• The collapse-mode low_stock check uses the store's stock_on_hand when store_id is present, otherwise stockInitial; the value must fall in [1, 10].
• If ES selected a product but no variant survives emit-time matching in collapse mode, the product is skipped and the skip is logged.

_transform_aggregations()

Transforms Elasticsearch aggregation results into UI-friendly format with availability flags. Buckets are keyed by sanity_id; each is resolved against the Django Brand, Category, ProductType, or Collection table, and buckets with no matching record are dropped.

Aggregation Output Format:

{
    'brands': [
        {
            '_id': 'django-uuid',
            'sanity_id': 'brand-sanity-id',
            'name': 'Nike',
            'count': 42,
            'available': True  # Would selecting this return results?
        }
    ],
    'categories': [...],
    'productTypes': [...],
    'collections': [...],
    'attributes': {
        'featured': [
            {
                'name': 'Color',
                'attribute_id': 'attr-uuid',
                'featured': True,
                'display_order': 1,
                'colors': [
                    {'name': 'Red', 'hex': '#FF0000', 'count': 15, 'available': True},
                    {'name': 'Blue', 'hex': '#0000FF', 'count': 8, 'available': False}
                ]
            }
        ],
        'regular': [
            {
                'name': 'Size',
                'options': [
                    {'value': 'Large', 'count': 20, 'available': True}
                ]
            }
        ]
    }
}

When available_options is None, every entry is flagged available: True.

_resolve_attributes_with_metadata()

Transforms attribute aggregation buckets into structured metadata with:

• Hex codes for color swatches
• Display order
• Featured status
• Availability flags

Queries the Attribute model (visible attributes only, ordered by display_order then name) to resolve:

• attributeColorSwatch → colors array with hex values
• attributeTextList → options array

Attributes are split into featured and regular groups by the attribute's featured flag.

Text Search

Short Query Behavior (3 chars or fewer)

Uses phrase_prefix for autocomplete, plus keyword prefix clauses on SKUs, tags, productType, brand, collections, and highlights:

# Searching "sn" returns "sneaker", "snacks", etc.
ESQ('multi_match', query='sn', type='phrase_prefix', fields=[
    'name^3', 'variant_names^2', 'shortDescription', ...
])

Long Query Behavior (more than 3 chars)

Uses fuzzy matching with field boosting:

ESQ('multi_match', query='sneakers', type='best_fields', fuzziness='AUTO', fields=[
    'name^3',              # Product name (3x boost)
    'variant_names^2',     # Variant names (2x boost)
    'variant_skus^2',      # SKUs (2x boost)
    'shortDescription',    # Description
    'organization.brand',  # Brand
    ...
])

Store-Specific Filtering

When store_id is provided:

1. Product Query: Only products with available inventory at that store
2. Variant Filtering: Only variants with is_available=True at that store
3. Quantity Calculation: stock_available = max(0, stock_on_hand - stock_reserved)

Elasticsearch Nested Query:

ESQ('nested', path='variants',
    query=ESQ('nested', path='variants.inventory_by_store',
        query=ESQ('bool', must=[
            ESQ('term', **{'variants.inventory_by_store.store_id': store_id}),
            ESQ('term', **{'variants.inventory_by_store.is_available': True})
        ])
    )
)

Usage Example

Endpoint

GET /search/smart-filter/ (AllowAny). Query parameters:

Invalid numeric parameters return 400; search-service failures return 500.

Frontend Request

// Filter by brand and category, collapsed grid, page 2
const params = new URLSearchParams({
    store_id: 'store-uuid',
    sort: 'price-asc',
    collapse_variants: 'true',
    page: '2',
    page_size: '24'
});
params.append('brand[]', 'nike-brand-uuid');
params.append('category[]', 'footwear-category-uuid');
params.append('attributes[Color][]', 'Red');

const response = await fetch(`/search/smart-filter/?\${params}`);
const { products, aggregations, total } = await response.json();

// Use aggregations.brands[].available to enable/disable filter checkboxes
// Branch rendering on products[].card_kind ('product' vs 'variant')

Performance Considerations

Hybrid Manifest Routing

Browsing requests (no search query, and none of brand, category, productType, collection, attributes, on_sale, or low_stock active) with a store_id first try a pre-generated manifest at MEDIA_ROOT/manifests/store_<store_id>.json. When the manifest exists, the view runs the product search with skip_aggregations=True, injects the manifest's aggregations, and recalculates available flags from current inventory via get_available_options() and apply_availability_flags(). Any manifest failure, a search query, or any of those filters falls back to real-time aggregations.

Aggregation Strategy

Aggregations run on the unpaginated query so counts cover all matching products, not just the current page. The aggregation query requests zero documents.

Smart Availability Cost

get_available_options() runs 5 separate queries (one per filter type). This is acceptable because:

• Each query returns only aggregations (no documents)
• Queries are simple and fast
• Results enable better UX (grayed-out options)

Manifest Mode

When skip_aggregations=True:

• Skips all aggregation queries
• Returns products only
• Used by manifest routing and manifest generation

Related Documentation

• Views - ProductSmartFilterAPIView and ProductSearchAPIView endpoints
• Search - ProductDocument Elasticsearch mapping
• Tasks - Manifest generation Celery chord
• Examples - API usage examples