Skip to main content

sprout_etl Architecture

Fastify-based ETL service for data extraction, transformation, and loading.

Overview​

The sprout_etl service handles:

  • Ticketmaster event synchronization
  • Skybox inventory management
  • Event matching between platforms
  • Price monitoring and comparison
  • Automated purchase processing

Architecture​

Entry Points​

1. Main HTTP Server (src/main.ts)​

  • Fastify server on port 3001
  • REST API endpoints
  • Health checks
  • Metrics endpoint at /metrics
  • Swagger documentation

2. Background Worker (src/worker.ts)​

  • BullMQ worker for job processing
  • Queue-based processing
  • Metrics collection via Pushgateway

3. CLI Interface (src/cli/index.ts)​

  • Command-line tools
  • Event comparison and matching
  • Inventory synchronization
  • Purchase creation
  • Testing utilities

Core Services​

Pipeline Service (src/services/pipeline.service.ts)​

  • Orchestrates ETL pipeline
  • Coordinates data flow
  • Error handling and retries

Event Processing Service (src/services/event-processing.service.ts)​

  • Processes event data
  • Transforms and normalizes
  • Stores in database

Event Pricing Service (src/services/event-pricing.service.ts)​

  • Fetches pricing data
  • Updates price history
  • Monitors price changes

Consumer Service (src/services/consumer.service.ts)​

  • Consumes external APIs
  • Rate limiting
  • Error handling

Cron Service (src/services/cron.service.ts)​

  • Scheduled tasks
  • Periodic data synchronization
  • Maintenance jobs

Modules​

Data Modules (src/modules/)​

  • Event matching logic
  • Data transformation
  • Integration modules

Services (src/services/)​

  • Core business logic
  • External API clients
  • Queue management
  • Monitoring services

CLI Commands (src/cli/)​

  • Operational commands
  • Data management
  • Testing tools

Queue System​

Queues​

  1. ETL_QUEUE (concurrency: 5)

    • Data processing jobs
    • Event synchronization
    • Inventory updates

  2. EVENT_MATCHER_QUEUE (concurrency: 1)

    • Event matching
    • Price comparisons

Job Types​

  • Event synchronization
  • Inventory updates
  • Price monitoring
  • Purchase processing
  • Data cleanup

Database Schema​

Uses Prisma ORM with PostgreSQL:

  • Events: EventRun, EventRunItem
  • Inventory: SkyboxInventory, SkyboxSoldInventory
  • Monitoring: MonitoredListing, MonitoredListingPriceHistory
  • Pipeline: EventRun tracking

API Endpoints​

Health & Metrics​

  • GET /health - Health check
  • GET /metrics - Prometheus metrics
  • GET /api-docs - Swagger documentation

Data Processing​

  • Event synchronization endpoints
  • Inventory management
  • Price monitoring

Monitoring​

Metrics​

  • Request processing duration
  • Job queue performance
  • Inventory matching metrics
  • Error tracking
  • Cost analysis

Logging​

  • Structured logging with Pino
  • Loki integration
  • Error tracking

Tracing​

  • OpenTelemetry integration
  • Distributed tracing
  • Performance monitoring

Configuration​

Environment variables validated with Joi schema in src/config.ts:

  • Database connection
  • Redis connection
  • API keys
  • Service URLs
  • Monitoring configuration

Development​

Running Locally​

# Start ETL server
doppler run -- bun run start:dev

# Start worker
doppler run -- bun run worker:dev

# Run CLI command
doppler run -- bun run cli:dev -- command-name [options]

Testing​

# Unit tests
bun test

# E2E tests
bun run test:e2e

# Coverage
bun run test:cov

CLI Commands​

The ETL service provides extensive CLI commands:

  • Event comparison and matching
  • Inventory synchronization
  • Purchase creation and monitoring
  • Data investigation tools
  • Testing utilities

See CLI_COMMANDS.md for full reference.

References​