Think of embedded analytics with DuckDB like SQLite, but built for crunching millions of spreadsheet rows directly inside your Python process instead of paying for a separate database server. On top of DuckDB, we use a Medallion architecture—a simple data pipeline with three steps: Raw/Messy (Bronze), Cleaned/Standardized (Silver), and Business-Ready Reports (Gold). We then validate those tables in Polars and serve 14 read-only FastAPI endpoints alongside a guarded Groq AI assistant.

System architecture

Kaggle CSVs (8,494 products + 1,094,411 reviews across 5 partitions)
  └── src/ingestion/load_sephora_kaggle_to_duckdb.py
        ├── Verified closed-file backup (.bak)
        └── raw.sephora_product_info / raw.sephora_reviews / raw.raw_products
              └── dbt-duckdb Medallion (16 relations, 119 passing data tests)
                    ├── Bronze: stg_raw_products, stg_raw_reviews
                    ├── Silver: int_product_catalog, int_review_analytics
                    └── Gold:   8 analytics & data-quality marts
                          └── Polars Validation & KPI Readers (src/analytics/)
                                └── FastAPI Serving & File-Stat LRU Cache (app/main.py)
                                      ├── 14 GET /metrics/* & /health endpoints
                                      ├── POST /assistant/ask (Groq tool-calling)
                                      └── Streamlit HTTP Dashboard (dashboard/)

In the table below, p50 latency means the typical response time for half of all requests, while p95 latency measures the slowest 5% of requests.

Atomic DuckDB ingestion with verified pre-load backups

Here is the problem we are solving: if a CSV ingestion script crashes halfway through loading a million rows, you do not want a half-written, corrupted database file. We combine a verified file backup with an atomic transaction—an all-or-nothing database update where either every row loads cleanly, or the database rewinds as if nothing happened.

This function in src/ingestion/load_sephora_kaggle_to_duckdb.py copies the closed .duckdb file to a timestamped .bak snapshot and opens a test connection to prove the backup is readable before touching the live data:

# src/ingestion/load_sephora_kaggle_to_duckdb.py
def _create_verified_backup(database_path: Path, backup_dir: Path | None) -> Path | None:
    if not database_path.exists():
        return None

    resolved_backup_dir = backup_dir or database_path.parent / "backups"
    resolved_backup_dir.mkdir(parents=True, exist_ok=True)
    # Generate a UTC timestamp so backups sort chronologically and never overwrite each other
    timestamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
    backup_path = resolved_backup_dir / f"{database_path.stem}_{timestamp}{database_path.suffix}.bak"
    # copy2 copies both the file bytes and filesystem metadata (timestamps)
    shutil.copy2(database_path, backup_path)

    # Open the backup in read-only mode to verify the file is not corrupted
    with duckdb.connect(str(backup_path), read_only=True) as connection:
        connection.execute("SELECT 1").fetchone()
    return backup_path

After the backup passes verification, all 8,494 product rows and 1,094,411 review rows load inside a single BEGIN TRANSACTION ... COMMIT block using read_csv(..., all_varchar = true, filename = true). Loading every raw column as text (all_varchar = true) stops DuckDB from crashing on weird formatting in raw CSVs—such as the unnamed first CSV header (column00)—so we can clean it safely with SQL in the next step.

Dynamic dbt target binding and Medallion quality diagnosis

Here is the problem we are solving: if you hardcode database: retail_engine in your dbt config, running tests in a staging environment (retail_engine_staging.duckdb) will still read from your production database by mistake.

Removing the static database: key from dbt_retail/model/staging/schema.yml lets dbt connect dynamically to whichever database file your active profile specifies:

# dbt_retail/model/staging/schema.yml
version: 2

sources:
  - name: raw
    schema: raw
    # Omitting 'database: retail_engine' here allows --target staging
    # to read from retail_engine_staging.duckdb instead of production.
    tables:
      - name: raw_products
      - name: sephora_product_info
      - name: sephora_reviews
      - name: sephora_ingestion_manifest

When cleaning data, beginners often run .dropna() or .drop_duplicates(), which silently deletes rows and hides upstream bugs. Instead, our Bronze and Silver models keep every row and attach TRUE/FALSE quality flags (like is_missing_rating).

This query in dbt_retail/model/marts/mart_review_summary.sql uses SQL FILTER (WHERE ...) clauses—which work like a WHERE clause attached to a single AVG() or SUM()—so missing values never drag down the averages:

-- dbt_retail/model/marts/mart_review_summary.sql
SELECT
    COUNT(*)::BIGINT AS total_reviews,
    COUNT(DISTINCT product_id)::BIGINT AS reviewed_products,
    COUNT(DISTINCT brand_key)::BIGINT AS reviewed_brands,
    -- Only average rows that actually have a valid rating
    ROUND(AVG(rating) FILTER (WHERE NOT is_missing_rating), 6) AS avg_rating,
    -- quantile_cont(..., 0.5) calculates the exact median (50th percentile)
    quantile_cont(rating, 0.5) FILTER (WHERE NOT is_missing_rating) AS median_rating,
    -- Calculate recommendation % over the 926,423 valid yes/no answers, ignoring 167,988 blanks
    ROUND(
        AVG(CASE WHEN is_recommended_bool THEN 1.0 ELSE 0.0 END)
            FILTER (WHERE NOT is_missing_is_recommended AND NOT is_invalid_is_recommended),
        6
    ) AS recommendation_rate,
    SUM(total_feedback_count) FILTER (WHERE NOT is_invalid_feedback_counts)::BIGINT AS total_feedback_votes,
    ROUND(AVG(helpfulness_ratio) FILTER (WHERE helpfulness_ratio IS NOT NULL), 6) AS avg_helpfulness_ratio,
    -- NULLIF(..., 0) prevents division-by-zero errors if total votes ever equal 0
    ROUND(
        SUM(total_pos_feedback_count) FILTER (WHERE NOT is_invalid_feedback_counts)::DOUBLE
        / NULLIF(SUM(total_feedback_count) FILTER (WHERE NOT is_invalid_feedback_counts), 0),
        6
    ) AS weighted_helpfulness_rate
FROM {{ ref('int_review_analytics') }}

Polars schema validation and FastAPI file-metadata caching

Here is the problem we are solving: before sending numbers to an API or dashboard, we want a fast safety check in Python to confirm that no boolean flags turned into NULL and no required columns disappeared.

This check in src/analytics/catalog_metrics.py loads the dbt output into a Polars DataFrame (a fast, strictly typed alternative to Pandas) and raises a ValueError if any quality flag contains nulls:

# src/analytics/catalog_metrics.py
for boolean_column in (
    "is_duplicate_candidate",
    "is_missing_brand",
    "is_missing_ingredients",
    "is_missing_rating",
    "is_missing_reviews",
    "has_any_quality_issue",
):
    # .null_count() returns how many empty/NULL cells exist in that column
    null_count = int(dataframe.get_column(boolean_column).null_count())
    if null_count > 0:
        raise ValueError(f"Column '{boolean_column}' contains {null_count} null values.")

Next, we need the API to respond in milliseconds without re-reading the 857 MB DuckDB file on every click. We use an LRU (Least Recently Used) cache keyed on file metadata—think of it like keeping a sticky note of the last answer on your desk, and only recalculating it when the database file's "last modified" timestamp (st_mtime_ns) or byte size (st_size) changes.

This code in app/main.py wires that cache to FastAPI and returns an HTTP 503 (Service Unavailable) status if the database is missing or fails validation:

# app/main.py
# @lru_cache stores the last 8 results in memory keyed by the 3 arguments below
@lru_cache(maxsize=8)
def _load_cached_review_bundle(
    database_path_str: str,
    modified_time_ns: int,
    file_size_bytes: int,
) -> ReviewAnalyticsBundle:
    # modified_time_ns and file_size_bytes only exist to bust the cache when the file changes
    del modified_time_ns, file_size_bytes
    return calculate_review_analytics_bundle(Path(database_path_str))


def get_review_bundle() -> ReviewAnalyticsBundle:
    database_path, database_stat = _get_database_signature()
    try:
        return _load_cached_review_bundle(
            str(database_path),
            database_stat.st_mtime_ns,  # Nanosecond timestamp of last file update
            database_stat.st_size,      # Exact file size in bytes
        )
    except ValueError as exc:
        logger.error("Review dataset readiness or validation error: %s", exc)
        raise HTTPException(
            status_code=503,
            detail="Analytics dataset is not ready. Run the ingestion and dbt pipeline first.",
        ) from exc

On a cold start (the first request when the cache is empty), reading from DuckDB takes ~227 ms. On subsequent warm requests, FastAPI returns the cached result in 3.665 ms p50 in-process (15.43–15.67 ms over local HTTP).

Allowlisted Groq tool-calling and commercial refusal guardrails

Here is the problem we are solving: if you let an LLM write raw SQL against your database (Text-to-SQL), it can hallucinate column names, write slow joins, or invent metrics that do not exist in your data. Instead, we use allowlisted tool-calling—giving the AI a vending-machine keypad of 12 pre-written Python functions (AnalyticsOperation) so every number comes straight from our tested Polars/DuckDB readers.

We also block out-of-scope commercial questions before calling Groq. Because public Sephora data has product prices and reviews—but no order receipts or warehouse logs—this regex filter in src/ai/assistant_service.py catches questions about revenue, sales volume, profit, or inventory and refuses them immediately:

# src/ai/assistant_service.py
# Each tuple pairs a category label with a compiled case-insensitive regex pattern (\b = word boundary)
_COMMERCIAL_OUT_OF_SCOPE_PATTERNS: tuple[tuple[str, re.Pattern[str]], ...] = (
    ("revenue", re.compile(r"\brevenue\b", re.IGNORECASE)),
    ("sales", re.compile(r"\b(total\s+sales|sales\s+volume|units?\s+sold|gross\s+sales|net\s+sales)\b", re.IGNORECASE)),
    ("profit", re.compile(r"\b(profit|profits|margin|margins|ebitda)\b", re.IGNORECASE)),
    ("inventory", re.compile(r"\b(inventory|stock\s+level|stockout|warehouse\s+stock|on[-\s]?hand)\b", re.IGNORECASE)),
    ("orders", re.compile(r"\b(order\s+count|orders\s+placed|checkout|cart\s+size|aov)\b", re.IGNORECASE)),
    ("customers_spend", re.compile(r"\b(customer\s+lifetime\s+value|ltv|cac|arpu|spend)\b", re.IGNORECASE)),
)

Engineering challenges and fixes

• Challenge — Static dbt database binding polluted production during staging tests: Having database: retail_engine in dbt_retail/model/staging/schema.yml caused staging builds (--target staging) to read from the production database.
  • Fix: Removed the static database: property from schema.yml and added scripts/build_staging_db.py to verify that retail_engine_staging.duckdb builds and passes all 119 dbt tests.
• Challenge — Groq openai/gpt-oss-20b returned empty message content on strict JSON prompts: Early synthesis prompts constrained token budgets too tightly, causing reasoning models to exhaust completion tokens before emitting final JSON.
  • Fix: Added a two-stage model invocation with deterministic fallback in GroqAnalyticsService: even if Groq rate-limits (RateLimitError), times out (APITimeoutError), or returns malformed output, _build_deterministic_answer still returns the exact pre-computed KPI payload alongside a sanitized warning field.
• Challenge — Repeated DuckDB scans on concurrent dashboard refreshes: Opening a new DuckDB connection and scanning 1,094,411 review rows on every Streamlit tab switch added unnecessary I/O overhead.
  • Fix: Keyed FastAPI's @lru_cache(maxsize=8) on (str(database_path), database_stat.st_mtime_ns, database_stat.st_size), cutting warm endpoint latency from ~227 ms cold start to 3.665 ms p50 in-process.

When a member sends a question to their financial cooperative over WhatsApp, they expect an experience that feels like texting a human officer. If an AI assistant takes 4 to 6 seconds to respond, the illusion collapses: users wonder if their message failed to deliver or if the service is down.

To deliver truly snappy conversational turns, our architecture standardizes on Groq Cloud as our unified inference engine. In this post, we explain how Groq’s LPU hardware, dual-modality support (Text + Whisper), and sub-second execution transform our production pipeline.

1. The Latency Trap in Conversational WhatsApp Apps

Most cloud LLM providers run inference on clusters optimized for batched training throughput rather than single-stream latency. A standard 70B-parameter model running on typical GPU endpoints generates 20 to 35 tokens per second. For a 200-token answer, that means six to ten seconds of generation time. Because WhatsApp does not support token streaming in chat bubbles, the webhook must wait for completion before dispatching the response.

Standard GPU Cloud:
[User Sends Query] ──────▶ [Wait 4-6 seconds...] ──────▶ [Single Answer Arrives]
(Member assumes the bot crashed or network stalled)

Groq LPU Engine:
[User Sends Query] ─▶ [~300ms Inference] ─▶ [Answer Delivered]
(Immediate conversational turn)

Groq’s custom silicon—the LPU™ (Language Processing Unit)—is purpose-built for sequential inference. By utilizing high-bandwidth SRAM and deterministic tensor execution, Groq generates tokens at speeds from 500 to 1,000 tokens per second depending on model size.

For a 200-token financial explanation, raw generation takes 200–400ms. Combined with a ~120ms Time-to-First-Token (TTFT), the entire completion finishes in approximately 300 milliseconds.

2. One Engine, Multiple Modalities.

Using Groq as a single provider across text and speech keeps our client infrastructure and latency budgets unified:

sequenceDiagram
    autonumber
    actor Member as WhatsApp Member
    participant Hook as FastAPI Webhook
    participant GroqAudio as Groq Audio (Whisper)
    participant Core as RAG & Verification
    participant GroqLLM as Groq LLM (gpt-oss-20b)

    alt Member sends Voice Note
        Member->>Hook: Audio Voice Note (.ogg)
        Hook->>GroqAudio: Transcribe with Swahili/Sheng Bias
        GroqAudio-->>Hook: Transcribed Text (~400ms)
    else Member sends Text
        Member->>Hook: Text Query
    end

    Hook->>Core: Fast Intent Filter & Local RAG Retrieval (~20ms)
    Core->>GroqLLM: Prompt + Retrieved Context
    GroqLLM-->>Core: Synthesized Answer (~280ms)
    Core->>Core: Deterministic Fact Verification (~5ms)
    Core->>Member: Verified WhatsApp Reply (Text: ~350ms, Voice: ~900ms)

A. Text & Triage: openai/gpt-oss-20b

In our core configuration (app/config/settings.py), we deploy openai/gpt-oss-20b with low reasoning effort (GROQ_REASONING_EFFORT = "low").

Operating as a Mixture-of-Experts (MoE) with 3.6 billion active parameters per forward pass, gpt-oss-20b generates at ~1,000 tokens per second on Groq LPUs. This delivers rich policy synthesis and intent classification without spending the token budget or runtime latency of 70B+ dense models. For complex offline evaluation or deep reasoning chains, the architecture can swap in openai/gpt-oss-120b (~500 tokens/sec) using the same provider adapter.

B. Voice Messaging: Fast Whisper for Audio Queries

Many cooperative members prefer sending voice notes rather than typing text on a mobile keyboard.

Incoming .ogg voice notes are forwarded to Groq’s audio endpoint running whisper-large-v3-turbo. A 10-second voice note is transcribed in approximately 350–450 milliseconds.

To support the regional linguistic reality of Kenya, our audio client (app/ai/providers/groq_audio.py) primes Whisper with a trilingual prompt bias:

# Contextual vocabulary biasing for East African SACCOs
VOICE_PROMPT_BIAS = (
    "Mazungumzo ya mwanachama wa SACCO kuhusu mikopo, hisa, akiba, na huduma. "
    "Code-switching in English, Kiswahili, and Kenyan Sheng: "
    "chama, ganji, bob, chapaa, mkopo, mikopo, bodaboda, biashara, "
    "loan balance, emergency loan, dividends, shares, repayment, M-Pesa."
)

This ensures localized financial terms like ganji or chapaa (money) and mkopo (loan) transcribe accurately instead of getting garbled by standard English models.

3. Engineering for Safety: Rate Limits & Bounded Retries

hen evaluating an inference provider, two distinct metrics matter: execution latency (how fast it returns) and financial cost (how much it bills).

A. API Pricing Economics

Self-hosting a 70B or 120B model on dedicated cloud GPUs costs $1.50 to $3.00 per hour ($1,000 to $2,000+ monthly), even when idle. Groq bills strictly per token and audio hour:

At these rates, 10,000 text inquiries cost less than $1.00, and 5,000 voice queries cost roughly $0.55.

B. Engineering for the 8,000 TPM Rate Limit

In free and developer tiers, Groq enforces an 8,000 Tokens Per Minute (TPM) limit and a 14,400 Tokens Per Day (TPD) ceiling. Running smaller, focused models like gpt-oss-20b keeps each turn around 400–500 total tokens, allowing steady conversational throughput.

Our provider (app/ai/providers/groq.py) implements three defensive patterns:

1. Differentiate Token-Per-Day (TPD) vs. Token-Per-Minute (TPM): If daily quota is exhausted, fail fast immediately. Never sleep-loop when quota will not recover for hours.

2. Respect Retry-After Headers: Parse the exact reset interval from the HTTP response headers or error body.

3. Bounded Jittered Backoff: Retry transient rate spikes without blocking webhook worker threads indefinitely.

import asyncio
import httpx
from typing import Any


class GroqClient:
    def __init__(self, api_key: str, model: str = "openai/gpt-oss-20b"):
        self.api_key = api_key
        self.model = model
        self.base_url = "https://api.groq.com/openai/v1/chat/completions"

    async def generate_safe(
        self, 
        messages: list[dict[str, str]], 
        max_retries: int = 2
    ) -> str:
        """Call Groq with strict error envelopes and bounded rate-limit handling."""
        headers = {
            "Authorization": f"Bearer {self.api_key}",
            "Content-Type": "application/json",
        }
        payload = {
            "model": self.model,
            "messages": messages,
            "max_tokens": 400,
            "temperature": 0.2,
        }

        async with httpx.AsyncClient(timeout=10.0) as client:
            for attempt in range(max_retries + 1):
                try:
                    response = await client.post(
                        self.base_url, json=payload, headers=headers
                    )
                    
                    if response.status_code == 429:
                        error_text = response.text.lower()
                        # Fail-fast if daily quota (TPD) is blown
                        if "tokens per day" in error_text or "tpd" in error_text:
                            raise RuntimeError("Daily token quota exhausted. Cannot retry.")
                        
                        retry_after = float(response.headers.get("retry-after", 1.5))
                        if attempt < max_retries:
                            await asyncio.sleep(min(retry_after, 3.0))
                            continue
                            
                    response.raise_for_status()
                    data = response.json()
                    return data["choices"][0]["message"]["content"]

                except httpx.TimeoutException:
                    if attempt < max_retries:
                        await asyncio.sleep(1.0)
                        continue
                    raise RuntimeError("Groq inference timed out.")

        raise RuntimeError("Groq inference failed after bounded retries.")

4. Why Speed Enables Deterministic Verification

The primary advantage of high-speed inference is latency headroom for safety checks.

Financial answers must be verified before delivery. When inference takes four seconds, there is no time left to validate output against source data within the webhook window.

Because Groq inference returns in ~280ms, our pipeline has ample headroom within a 1-second webhook window to execute a complete multi-step safety harness:

What About Voice Notes?

For incoming voice notes, audio processing adds two steps before text RAG begins:

• Audio Download from Twilio: ~150–250ms (network-dependent media fetch).

• Groq Whisper Transcription: ~350–450ms (whisper-large-v3-turbo).

Adding the 305ms text pipeline results in an end-to-end voice turnaround of ~800–1,000ms, remaining safely under the 1.5-second messaging threshold.

Why Groq, trade-offs, and production hurdles

• [Symptom/Challenge]: Concurrent traffic bursts triggered HTTP 429 rate limit errors against the 8,000 TPM ceiling.

• [The Fix]: Added bounded jittered retries and daily quota fail-fast logic in app/ai/providers/groq.py.

• [Symptom/Challenge]: Transient WAN latency spikes risked exceeding Twilio's strict webhook timeout thresholds.

• [The Fix]: Wrapped calls in 10s HTTPX timeouts with human-escalation fallbacks via FallbackHandler.provider_failure().

6. Implementation summary

• Sub-second response budget: Operating at ~1,000 tokens/sec on Groq LPUs completes 200-token text generation in ~280ms, leaving over 600ms of headroom for retrieval and validation.

• Clear cost separation: High speed does not mean high bills. At \(0.075 / 1M input tokens and \)0.04 / hour of audio, typical member inquiries cost fractions of a cent ($0.00008).

• Unified text and voice endpoints: Running both gpt-oss-20b and whisper-large-v3-turbo on Groq consolidates credentials and keeps latency predictable across chat bubbles and voice clips.

• Defensive rate limiting: Enforce bounded backoff on 429 responses, parse Retry-After headers, and distinguish transient TPM spikes from permanent daily quota exhaustion (TPD).