How to install in Docker

Docker is the easiest way to get started with self-hosted Sparrow. This guide assumes you are running the command from the machine you intend to host from.

System Requirements

Linux VM (Ubuntu 20.04+ recommended)
Docker Engine 20.10+
Docker Compose 2.0+
Minimum 4GB RAM
At least 20GB free disk space
Internet connectivity
Custom domain names with DNS configured

Architecture Overview

The Sparrow stack consists of the following components:

MongoDB: Database server
Kafka: Message broker
Sparrow API: Main API service
Sparrow Auth: Authentication service
Sparrow Proxy: Proxy service
Sparrow Web: Web application frontend
Nginx: Reverse proxy for SSL termination and routing

Pre-requisites

1. Install Docker and Docker Compose

# Update the package index
sudo apt-get update

# Install required packages
sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common

# Add Docker's official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

# Set up the Docker repository
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

# Install Docker
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io

# Install Docker Compose
sudo curl -L "https://github.com/docker/compose/releases/download/v2.17.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

# Add your user to the docker group to run Docker without sudo
sudo usermod -aG docker ${USER}

Log out and log back in for the group changes to take effect.

2. Configure DNS

Ensure you have configured DNS records for your custom domains pointing to your server's IP address:

api.yourdomain.com → Your VM's public IP
auth.yourdomain.com → Your VM's public IP
app.yourdomain.com → Your VM's public IP

Deployment Steps

:::

First clone the SparrowAPI repository:

git clone https://github.com/sparrowapp-dev/sparrow-api.git
cd sparrowapp-dev

Create env

cp .env.example .env

cd ~/sparrow-app

# Create necessary directories for Nginx
mkdir -p ./nginx/conf.d
mkdir -p ./nginx/ssl
mkdir -p ./nginx/html
mkdir -p ./nginx/logs

3. Create Environment Variables File

Update the .env.docker-setup with the environment variables:

Environment variables

4. Create Nginx Configuration

Create a file named nginx/conf.d/sparrow.conf:

# Redirect HTTP to HTTPS
server {
    listen 80;
    server_name api.yourdomain.com auth.yourdomain.com app.yourdomain.com;
    
    location /.well-known/acme-challenge/ {
        root /usr/share/nginx/html;
    }
    
    location / {
        return 301 https://$host$request_uri;
    }
}

# Sparrow API configuration
server {
    listen 443 ssl;
    server_name api.yourdomain.com;
    
    ssl_certificate /etc/nginx/ssl/api.yourdomain.com.crt;
    ssl_certificate_key /etc/nginx/ssl/api.yourdomain.com.key;
    
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    
    location / {
        proxy_pass http://sparrow-api:9000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 90;
    }
    
    # WebSocket support for socket.io
    location /socket.io/ {
        proxy_pass http://sparrow-api:9001;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
    
    # WebSocket support for AI assistant
    location /ai-assistant {
        proxy_pass http://sparrow-api:9000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

# Sparrow Auth configuration
server {
    listen 443 ssl;
    server_name auth.yourdomain.com;
    
    ssl_certificate /etc/nginx/ssl/auth.yourdomain.com.crt;
    ssl_certificate_key /etc/nginx/ssl/auth.yourdomain.com.key;
    
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    
    location / {
        proxy_pass http://sparrow-auth:80;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 90;
    }
}

# Sparrow Web configuration
server {
    listen 443 ssl;
    server_name app.yourdomain.com;
    
    ssl_certificate /etc/nginx/ssl/app.yourdomain.com.crt;
    ssl_certificate_key /etc/nginx/ssl/app.yourdomain.com.key;
    
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    
    location / {
        proxy_pass http://sparrow-web:80;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 90;
    }
}

Note: Replace all instances of yourdomain.com with your actual domain name.

5. Create Dockerfile for Auth Service

Create a file named Dockerfile for the Sparrow Auth service:

FROM node:18-alpine AS builder

WORKDIR /app

# Copy package files and install dependencies
COPY package.json package-lock.json ./
RUN npm ci

# Copy source files
COPY . .

# Build the application
RUN npm run build

# Production stage
FROM nginx:alpine

# Copy built files
COPY --from=builder /app/dist /usr/share/nginx/html

# Copy nginx configuration
COPY nginx.conf /etc/nginx/conf.d/default.conf

EXPOSE 80

CMD ["nginx", "-g", "daemon off;"]

6. Configure SSL Certificates

Option 1: Self-signed certificates (for development only)

Create a script named generate-ssl.sh:

#!/bin/bash

# Create directories if they don't exist
mkdir -p ./nginx/ssl

# Generate self-signed certificates for each domain
generate_self_signed_cert() {
  domain=$1
  echo "Generating self-signed certificate for $domain"
  openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
    -keyout ./nginx/ssl/$domain.key \
    -out ./nginx/ssl/$domain.crt \
    -subj "/CN=$domain/O=Sparrow/C=US"
}

# Generate certificates for each domain
generate_self_signed_cert "api.yourdomain.com"
generate_self_signed_cert "auth.yourdomain.com"
generate_self_signed_cert "app.yourdomain.com"

echo "Self-signed certificates generated successfully!"

Make the script executable and run it:

chmod +x generate-ssl.sh
./generate-ssl.sh

Note: Replace all instances of yourdomain.com with your actual domain name.

Option 2: Let's Encrypt certificates (recommended for production)

Install Certbot:

sudo apt-get update
sudo apt-get install -y certbot

Stop any services running on ports 80 and 443:

sudo systemctl stop nginx # If you have nginx installed

Obtain certificates:

sudo certbot certonly --standalone -d api.yourdomain.com --agree-tos --email your@email.com
sudo certbot certonly --standalone -d auth.yourdomain.com --agree-tos --email your@email.com
sudo certbot certonly --standalone -d app.yourdomain.com --agree-tos --email your@email.com

Copy certificates to Nginx directory:

sudo cp /etc/letsencrypt/live/api.yourdomain.com/fullchain.pem ./nginx/ssl/api.yourdomain.com.crt
sudo cp /etc/letsencrypt/live/api.yourdomain.com/privkey.pem ./nginx/ssl/api.yourdomain.com.key
sudo cp /etc/letsencrypt/live/auth.yourdomain.com/fullchain.pem ./nginx/ssl/auth.yourdomain.com.crt
sudo cp /etc/letsencrypt/live/auth.yourdomain.com/privkey.pem ./nginx/ssl/auth.yourdomain.com.key
sudo cp /etc/letsencrypt/live/app.yourdomain.com/fullchain.pem ./nginx/ssl/app.yourdomain.com.crt
sudo cp /etc/letsencrypt/live/app.yourdomain.com/privkey.pem ./nginx/ssl/app.yourdomain.com.key

Set appropriate permissions:

sudo chmod 644 ./nginx/ssl/*.crt
sudo chmod 600 ./nginx/ssl/*.key

7. Deploy the Sparrow Stack

Build and start the Docker containers:

# Build images and start containers
docker-compose up -d

# Check if all containers are running
docker-compose ps

8. Configure Automatic SSL Renewal (Let's Encrypt only)

Create a script named renew-ssl.sh:

#!/bin/bash

# Stop nginx container
docker-compose stop nginx-proxy

# Renew certificates
sudo certbot renew

# Copy renewed certificates
sudo cp /etc/letsencrypt/live/api.yourdomain.com/fullchain.pem ./nginx/ssl/api.yourdomain.com.crt
sudo cp /etc/letsencrypt/live/api.yourdomain.com/privkey.pem ./nginx/ssl/api.yourdomain.com.key
sudo cp /etc/letsencrypt/live/auth.yourdomain.com/fullchain.pem ./nginx/ssl/auth.yourdomain.com.crt
sudo cp /etc/letsencrypt/live/auth.yourdomain.com/privkey.pem ./nginx/ssl/auth.yourdomain.com.key
sudo cp /etc/letsencrypt/live/app.yourdomain.com/fullchain.pem ./nginx/ssl/app.yourdomain.com.crt
sudo cp /etc/letsencrypt/live/app.yourdomain.com/privkey.pem ./nginx/ssl/app.yourdomain.com.key

# Start nginx container
docker-compose start nginx-proxy

echo "SSL certificates renewed successfully!"

Make the script executable:

chmod +x renew-ssl.sh

Add a cron job to run the renewal script monthly:

crontab -e

Add the following line:

0 0 1 * * /home/username/sparrow-deployment/renew-ssl.sh >> /home/username/sparrow-deployment/ssl-renewal.log 2>&1

Replace username with your actual username.

Verification and Troubleshooting

Verify Deployment

Once the setup is complete, verify that your services are accessible:

Web app: https://app.yourdomain.com
Auth service: https://auth.yourdomain.com
API service: https://api.yourdomain.com

Troubleshooting

Check Container Logs

# Check logs for a specific service
docker-compose logs sparrow-api
docker-compose logs sparrow-auth
docker-compose logs sparrow-web
docker-compose logs nginx-proxy

# Follow logs in real-time
docker-compose logs -f nginx-proxy

Check Nginx Configuration

# Enter the Nginx container
docker-compose exec nginx-proxy sh

# Test Nginx configuration
nginx -t

# Check Nginx logs
cat /var/log/nginx/error.log

SSL Issues

# Check SSL certificate validity
openssl x509 -in ./nginx/ssl/api.yourdomain.com.crt -text -noout

Network Issues

# Check if ports are open
sudo netstat -tulpn | grep -E '80|443'

# Test connectivity between containers
docker-compose exec sparrow-web ping sparrow-api

Maintenance

Updating Services

To update a service with a new image or configuration:

# Pull the latest images
docker-compose pull

# Rebuild and restart services
docker-compose up -d --build

Backup and Restore

Backup MongoDB Data

# Create a backup directory
mkdir -p ~/sparrow-backups

# Backup MongoDB data
docker-compose exec mongo mongodump --out /data/db/backup
docker cp $(docker-compose ps -q mongo):/data/db/backup ~/sparrow-backups/mongo-$(date +%Y%m%d)

Restore MongoDB Data

# Copy backup to container
docker cp ~/sparrow-backups/mongo-20230101 $(docker-compose ps -q mongo):/data/db/restore

# Restore data
docker-compose exec mongo mongorestore /data/db/restore

Monitoring

Consider adding a monitoring solution like Prometheus and Grafana:

# Add to docker-compose.yml
services:
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    networks:
      - localnet
    restart: unless-stopped

  grafana:
    image: grafana/grafana
    volumes:
      - grafana_data:/var/lib/grafana
    ports:
      - "3000:3000"
    networks:
      - localnet
    restart: unless-stopped

volumes:
  prometheus_data:
  grafana_data:

Security Considerations

Firewall Configuration: Configure your server's firewall to allow only necessary ports:
```
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable
```
Strong passwords: Ensure all services use strong passwords.
Regular updates: Keep all containers updated:
```
docker-compose pull
docker-compose up -d
```
SSL certificate renewal: Regularly renew SSL certificates.
Access logs monitoring: Regularly check Nginx access logs for suspicious activity.

Support and Resources

Docker Documentation: https://docs.docker.com/
Nginx Documentation: https://nginx.org/en/docs/
Let's Encrypt Documentation: https://letsencrypt.org/docs/
MongoDB Documentation: https://docs.mongodb.com/
Kafka Documentation: https://kafka.apache.org/documentation/

Contributing

Contributions to improve this guide are welcome. Please submit pull requests or open issues to discuss proposed changes.

System Requirements​

Architecture Overview​

Pre-requisites​

1. Install Docker and Docker Compose​

2. Configure DNS​

Deployment Steps​

3. Create Environment Variables File​

4. Create Nginx Configuration​

5. Create Dockerfile for Auth Service​

6. Configure SSL Certificates​

Option 1: Self-signed certificates (for development only)​

Option 2: Let's Encrypt certificates (recommended for production)​

7. Deploy the Sparrow Stack​

8. Configure Automatic SSL Renewal (Let's Encrypt only)​

Verification and Troubleshooting​

Verify Deployment​

Troubleshooting​

Check Container Logs​

Check Nginx Configuration​

SSL Issues​

Network Issues​

Maintenance​

Updating Services​

Backup and Restore​

Backup MongoDB Data​

Restore MongoDB Data​

Monitoring​

Security Considerations​

Support and Resources​

Contributing​