Skip to main content

System Overview

High-level architecture of the Sprout platform.

Architecture Diagram​

Service Interactions​

Data Flow​

  1. Event Data Ingestion

    • Cloudflare Workers scrape event listings
    • Data flows to sprout_etl for processing
    • Processed data stored in PostgreSQL

  2. Inventory Management

    • Skybox API provides inventory data
    • Token extractor gets authentication tokens
    • Backend matches events with inventory

  3. Price Monitoring

    • ETL service monitors prices
    • Updates stored in database
    • Dashboard displays pricing data

  4. Background Processing

    • BullMQ queues handle async jobs
    • Workers process jobs from Redis
    • Results stored and monitored

Core Services​

sprout_backend​

Purpose: Main API server and background job processor

Responsibilities:

  • REST API for frontend
  • Event matching and pricing
  • Inventory management
  • User authentication
  • Background job processing

Technology: NestJS, Prisma, BullMQ, MeiliSearch

sprout_etl​

Purpose: Data extraction, transformation, and loading

Responsibilities:

  • Ticketmaster event synchronization
  • Skybox inventory processing
  • Event matching
  • Price monitoring
  • Data pipeline orchestration

Technology: Fastify, Prisma, BullMQ, MeiliSearch

pricing_dashboard​

Purpose: Web interface for pricing management

Responsibilities:

  • Display pricing data
  • Event management UI
  • Inventory visualization
  • Analytics and reporting

Technology: React, TypeScript, Vite, tRPC

cloudflare-workers​

Purpose: Serverless scraping infrastructure

Responsibilities:

  • Web scraping of event listings
  • Rate limiting and retry logic
  • Queue-based processing
  • State management

Technology: Cloudflare Workers, Durable Objects, KV, Queues

Data Storage​

PostgreSQL​

Primary database for:

  • Event data
  • Inventory records
  • User data
  • Purchase history
  • Monitoring data

Redis​

Used for:

  • Job queues (BullMQ)
  • Caching
  • Rate limiting
  • Session storage

MeiliSearch​

Full-text search for:

  • Event matching
  • Inventory search
  • Fast queries

Communication Patterns​

Synchronous​

  • HTTP REST APIs
  • GraphQL (if used)
  • Direct database queries

Asynchronous​

  • BullMQ job queues
  • Event-driven processing
  • Webhooks

Security​

  • Authentication via JWT
  • Google OAuth integration
  • Environment variables in Doppler
  • API rate limiting
  • CORS configuration

Monitoring & Observability​

Metrics​

  • Prometheus for metrics collection
  • Custom business metrics
  • Performance monitoring

Logging​

  • Structured logging with Pino
  • Centralized logging with Loki
  • Error tracking with Sentry

Tracing​

  • OpenTelemetry integration
  • Distributed tracing
  • Performance analysis

Scalability​

Horizontal Scaling​

  • Stateless API servers
  • Worker instances can scale
  • Load balancing ready

Vertical Scaling​

  • Database optimization
  • Caching strategies
  • Queue processing optimization

Next Steps​