Skip to main content

ETL Monitoring Stack Deployment Guide

This guide walks you through deploying the ETL monitoring stack on a DigitalOcean droplet with Cloudflare tunnel for secure domain access.

🎯 Overview​

The monitoring stack includes:

  • Grafana: Dashboard visualization
  • Prometheus: Metrics collection and storage
  • Alertmanager: Alert routing and notification
  • Cloudflare Tunnel: Secure domain access without opening ports

🚀 Step 1: Create DigitalOcean Droplet​

Droplet Specifications​

  • Size: Basic plan, $6/month (1GB RAM, 1 vCPU, 25GB SSD)
  • Region: Choose closest to your users
  • Image: Ubuntu 22.04 LTS
  • Authentication: SSH key (recommended) or password

Create Droplet​

  1. Go to DigitalOcean Console
  2. Click "Create" → "Droplets"
  3. Choose Ubuntu 22.04 LTS
  4. Select Basic plan ($6/month)
  5. Choose datacenter region
  6. Add SSH key or set password
  7. Click "Create Droplet"

Get Droplet IP​

Once created, note the public IP address (e.g., 164.92.123.456)

🔧 Step 2: Deploy Monitoring Stack​

Prerequisites​

  • SSH access to your droplet
  • Your droplet's public IP address

Deploy the Stack​

# Set your droplet IP
export DROPLET_IP=your_droplet_ip_here

# Deploy monitoring stack
./scripts/deploy-monitoring-droplet.sh

This script will:

  • Install Docker and Docker Compose
  • Set up Cloudflare tunnel
  • Deploy the monitoring stack
  • Configure all services
  • Provide you with access URLs

Expected Output​

After deployment, you'll see URLs like:

Cloudflare Tunnel URLs:
  Grafana: https://grafana.abc123.trycloudflare.com
  Prometheus: https://prometheus.abc123.trycloudflare.com
  Alertmanager: https://alertmanager.abc123.trycloudflare.com

🔗 Step 3: Configure ETL Service​

Update ETL Configuration​

# Update ETL service to point to monitoring droplet
./scripts/update-etl-config.sh

Start ETL Service with Monitoring​

# Start ETL service with monitoring enabled
./scripts/start-etl-with-monitoring.sh

🧪 Step 4: Test the Setup​

Run Tests​

# Test the complete monitoring setup
./scripts/test-monitoring-setup.sh

Manual Verification​

  1. Access Grafana: Visit your Cloudflare URL

    • Default credentials: admin/admin
    • Change password on first login

  2. Check Prometheus: Visit Prometheus URL

    • Go to Status → Targets
    • Verify ETL service target is "UP"

  3. View Dashboard: In Grafana

    • Navigate to Dashboards
    • Find "ETL System Health - Assigned Purchase Monitoring"

🔒 Security Configuration​

Change Default Passwords​

# SSH to your droplet
ssh root@your_droplet_ip

# Change Grafana admin password
# (Do this through Grafana UI)

# Secure the droplet
ufw enable
ufw allow ssh
ufw allow 80
ufw allow 443

Cloudflare Tunnel Security​

  • No open ports: Cloudflare tunnel doesn't require opening ports
  • End-to-end encryption: All traffic is encrypted
  • Access control: You can add authentication later

📊 Accessing Your Monitoring​

Primary URLs​

  • Grafana Dashboard: https://grafana.your-tunnel-id.trycloudflare.com
  • Prometheus: https://prometheus.your-tunnel-id.trycloudflare.com
  • Alertmanager: https://alertmanager.your-tunnel-id.trycloudflare.com

Alternative Access Methods​

If you have a domain:

  1. Point your domain to Cloudflare
  2. Create CNAME records: 
    monitoring.yourdomain.com → grafana.your-tunnel-id.trycloudflare.com
    prometheus.yourdomain.com → prometheus.your-tunnel-id.trycloudflare.com
  3. Update Cloudflare tunnel config with custom hostnames

Option 2: Ngrok (Alternative)​

# Install ngrok on droplet
curl -s https://ngrok-agent.s3.amazonaws.com/ngrok.install.sh | bash

# Create tunnels
ngrok http 3000 --subdomain=your-grafana
ngrok http 9090 --subdomain=your-prometheus
# Open ports on droplet (less secure)
ufw allow 3000  # Grafana
ufw allow 9090  # Prometheus
ufw allow 9093  # Alertmanager

🔧 Management Commands​

SSH to Droplet​

ssh root@your_droplet_ip

View Service Logs​

# View all logs
docker-compose -f /opt/monitoring-stack/docker-compose.monitoring.yml logs -f

# View specific service logs
docker-compose -f /opt/monitoring-stack/docker-compose.monitoring.yml logs -f grafana
docker-compose -f /opt/monitoring-stack/docker-compose.monitoring.yml logs -f prometheus

Restart Services​

# Restart all services
docker-compose -f /opt/monitoring-stack/docker-compose.monitoring.yml restart

# Restart specific service
docker-compose -f /opt/monitoring-stack/docker-compose.monitoring.yml restart grafana

Update Configuration​

# Edit Prometheus config
nano /opt/monitoring-stack/prometheus.yml

# Reload Prometheus
docker-compose -f /opt/monitoring-stack/docker-compose.monitoring.yml restart prometheus

📈 Monitoring Your Monitoring​

System Resources​

# Check system resources
htop
df -h
free -h

# Monitor Docker containers
docker stats

Service Health​

# Check service health
curl http://localhost:3000/api/health  # Grafana
curl http://localhost:9090/-/healthy   # Prometheus
curl http://localhost:9093/-/healthy   # Alertmanager

🔔 Alerting Configuration​

Note: All alerting is now managed in Grafana. Prometheus no longer loads alert rules from etl-alerts.yml.

Configure Slack Notifications​

  1. Create Slack webhook
  2. Update Alertmanager config: 
    nano /opt/monitoring-stack/grafana/provisioning/alerting/alertmanager.yml
  3. Replace YOUR_SLACK_WEBHOOK_URL with your actual webhook
  4. Restart Alertmanager: 
    docker-compose -f /opt/monitoring-stack/docker-compose.monitoring.yml restart alertmanager

Test Alerts​

# Generate test metrics to trigger alerts
node scripts/test-metrics.js

💰 Cost Breakdown​

Monthly Costs​

  • DigitalOcean Droplet: $6/month
  • Cloudflare Tunnel: Free
  • Domain (optional): $1-15/year
  • Total: ~$6/month

Cost Optimization​

  • Droplet size: Start with $6/month, scale up if needed
  • Storage: 25GB included, sufficient for months of metrics
  • Bandwidth: 1TB included, more than enough for monitoring

🚨 Troubleshooting​

Common Issues​

Services Not Starting​

# Check Docker status
systemctl status docker

# Check container logs
docker-compose -f /opt/monitoring-stack/docker-compose.monitoring.yml logs

Cloudflare Tunnel Not Working​

# Check tunnel status
cloudflared tunnel list

# Check tunnel logs
journalctl -u cloudflared -f

# Restart tunnel
systemctl restart cloudflared

ETL Service Not Reachable​

# Check if ETL service is running
curl http://your_etl_ip:3001/metrics

# Check firewall
ufw status

# Test connectivity
telnet your_etl_ip 3001

Prometheus Targets Down​

  1. Check ETL service is running
  2. Verify network connectivity
  3. Check Prometheus configuration
  4. View target status in Prometheus UI

Getting Help​

  • Logs: Check service logs for error messages
  • Health checks: Use the provided health check endpoints
  • Documentation: Refer to this guide and the README files

🔄 Backup and Recovery​

Backup Configuration​

# Backup all configs
tar -czf monitoring-backup-$(date +%Y%m%d).tar.gz /opt/monitoring-stack/

Restore Configuration​

# Extract backup
tar -xzf monitoring-backup-YYYYMMDD.tar.gz -C /

# Restart services
docker-compose -f /opt/monitoring-stack/docker-compose.monitoring.yml restart

🎉 Success!​

Your monitoring stack is now deployed and ready to monitor your ETL service. You have:

✅ Secure access via Cloudflare tunnel
✅ Comprehensive dashboards for ETL health
✅ Alerting system for proactive monitoring
✅ Cost-effective solution at ~$6/month
✅ Full control over your monitoring infrastructure

The dashboard will show you real-time insights into your assigned purchase processing pipeline, helping you maintain high availability and performance.