Skip to main content

sprout_etl

Fastify-based ETL service for data extraction, transformation, and loading.

Overview​

The sprout_etl service handles:

  • Ticketmaster event synchronization
  • Skybox inventory management
  • Event matching between platforms
  • Price monitoring and comparison
  • Automated purchase processing

Technology Stack​

  • Framework: Fastify 4.x
  • Language: TypeScript 5.x
  • Database: PostgreSQL with Prisma ORM
  • Queue: BullMQ with Redis
  • Search: MeiliSearch
  • Monitoring: Prometheus, Sentry, OpenTelemetry
  • Logging: Pino with Loki integration

Getting Started​

Prerequisites​

  • Node.js 18+
  • Bun (preferred) or npm
  • Docker (for PostgreSQL and Redis)
  • Doppler CLI (for environment variables)

Setup​

  1. Install dependencies:

    bun install

  2. Start Docker services:

    docker-compose up -d

  3. Run database migrations:

    doppler run -- npx prisma migrate dev

  4. Start development server:

    doppler run -- bun run start:dev

  5. Start background worker (separate terminal):

    doppler run -- bun run worker:dev

Key Commands​

See Detailed Commands Reference for complete command reference.

Development​

  • bun run start:dev - Start ETL server
  • bun run worker:dev - Start background worker
  • bun run build - Build TypeScript

CLI Commands​

  • bun run cli:dev -- command-name [options] - Run CLI commands
  • See Commands Reference for full CLI reference

Code Quality​

  • bun run type:check - Type checking
  • bun run lint - Lint with auto-fix
  • bun run format - Format code

Testing​

  • bun test - Run tests
  • bun run test:cov - Coverage report
  • bun run test:e2e - E2E tests

Architecture​

The service has three main entry points:

  1. Main HTTP Server (src/main.ts) - Port 3001
  2. Background Worker (src/worker.ts) - Background processing
  3. CLI Interface (src/cli/index.ts) - Command-line tools

Core Services​

  • pipeline.service.ts - ETL pipeline orchestration
  • event-processing.service.ts - Event data processing
  • event-pricing.service.ts - Price monitoring
  • consumer.service.ts - External API consumption
  • cron.service.ts - Scheduled tasks

Database​

Uses Prisma ORM with PostgreSQL. Schema includes:

  • Event runs (EventRun, EventRunItem)
  • Inventory (SkyboxInventory, SkyboxSoldInventory)
  • Monitoring (MonitoredListing, MonitoredListingPriceHistory)

Queue System​

Uses BullMQ with Redis:

  • ETL_QUEUE (concurrency: 5) - Data processing
  • EVENT_MATCHER_QUEUE (concurrency: 1) - Event matching

API Endpoints​

  • GET /health - Health check
  • GET /metrics - Prometheus metrics
  • GET /api-docs - Swagger documentation
  • REST API endpoints for data processing

Environment Variables​

All environment variables are in Doppler. See Environment Variables for complete list.

Key variables:

  • DATABASE_URL - PostgreSQL connection
  • REDIS_URL - Redis connection
  • MEILISEARCH_HOST - MeiliSearch URL
  • TICKETMASTER_API_KEY - Ticketmaster API key
  • PLAYWRIGHT_SERVER_URL - Playwright server URL

Monitoring​

  • Metrics: Prometheus at /metrics, Pushgateway for workers
  • Logging: Structured logging with Pino, Loki integration
  • Tracing: OpenTelemetry integration
  • Error Tracking: Sentry

Documentation​