Installation
- Docker
- Ubuntu
Docker is the easiest way to get started with self-hosted Sparrow. This guide assumes you are running the command from the machine you intend to host from.
Before you begin
First clone the SparrowAPI repository:
git clone https://github.com/sparrowapp-dev/sparrow-api.git
cd sparrowapp-dev
Create env
cp .env.example .env
ENV Configuration
# APP
PORT=9000
APP_ENV= DEV
APP_URL= http://localhost
# AUTH
JWT_SECRET_KEY= Provide the secret key for JWT (JSON Web Tokens) encryption
JWT_EXPIRATION_TIME= Set the expiration time for JWT tokens in seconds.
# DATABASE
DB_URL= Input the URL of your database.
# Google OAuth
GOOGLE_CLIENT_ID= Specify the Google Client ID.
GOOGLE_CLIENT_SECRET= Provide the Google Client Secret.
GOOGLE_APP_URL= Set the URL of the Google application.
LOGIN_REDIRECT_URL=Define the URL to redirect after successful login.
GOOGLE_ACCESS_TYPE=Specify the access type for Google OAuth.
# EXPIRY TIME FOR EMAIL VALIDATION CODE/LINK
EMAIL_VALIDATION_CODE_EXPIRY_TIME=Set the expiry time for email validation codes/links.
# KAFKA
KAFKA_BROKER=Input the Kafka broker details.
#EMAIL
SMTP_SENDER_EMAIL= Provide the sender's email address for SMTP.
SMTP_SENDER_PASSWORD= Input the sender's password for SMTP authentication.
SMTP_MAIL_HOST= Set the SMTP mail host.
SMTP_MAIL_PORT= Specify the SMTP mail port.
SMTP_MAIL_SECURE= Define the SMTP mail security method
SMTP_USER_NAME= Provide the SMTP username.
# REFRESH TOKEN
REFRESH_TOKEN_SECRET_KEY=refresh-token-secret
REFRESH_TOKEN_EXPIRATION_TIME=604800
REFRESH_TOKEN_MAX_LIMIT=50
# APP UPDATER
APP_UPDATE_AVAILABLE=true
APP_VERSION=
WINDOWS_APP_SIGNATURE=
WINDOWS_APP_URL=
MAC_APPLE_SILICON_APP_SIGNATURE=
MAC_APPLE_SILICON_APP_URL=
MAC_INTEL_APP_SIGNATURE=
MAC_INTEL_APP_URL=
# SUPPORT
SPARROW_EMAIL=contactus@sparrowapp.dev
Now, you can use docker-compose
to start Sparrow API.
# Pull the latest images
docker compose pull
# Start the services (in detached mode)
docker compose up -d
After all the services have started you can see them running in the background
docker compose ps
Accessing Sparrow API swagger
http://localhost/api/docs
Production Environments Docker Setup
Custom Domian using NGNIX reverse proxy and SSL with Let's Encrypt.
Need to create below file structure in existing repository.
sparrowapp-dev/
│
├── docker-compose.yml
├── nginx/
│ ├── Dockerfile
│ ├── nginx.conf
│ └── certbot/
│ └── Dockerfile
└── volumes/
└── nginx/
Production docker-compose.yml with Ngnix and SSL
version: "3.8"
services:
# Sparrow API
sparrow:
build:
dockerfile: Dockerfile
ports:
- "9000:9000"
env_file:
- .env
networks:
- localnet
# MongoDB
mongo:
image: mongo:7.0
environment:
MONGO_INITDB_ROOT_USERNAME: YOUR_USERNAME
MONGO_INITDB_ROOT_PASSWORD: YOUR_PASSWORD
ports:
- "27017:27017"
volumes:
- mongo_data:/data/db
networks:
- localnet
# Kafka
kafka:
image: bitnami/kafka:3.4.1
hostname: kafka
ports:
- "9092:9092"
volumes:
- "kafka_data:/bitnami"
environment:
- KAFKA_ENABLE_KRAFT=yes
- KAFKA_CFG_PROCESS_ROLES=broker,controller
- KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
- KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092,EXTERNAL://kafka:9094
- KAFKA_BROKER_ID=1
- KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@localhost:9093
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_CFG_NODE_ID=1
- KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
- KAFKA_CFG_NUM_PARTITIONS=2
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
- KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1
- KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1
- KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0
- KAFKA_MESSAGE_MAX_BYTES=1000000
deploy:
resources:
limits:
cpus: "1"
memory: 1G
# NGINX Reverse Proxy
nginx:
build:
context: ./nginx
dockerfile: Dockerfile
ports:
- "80:80"
- "443:443"
volumes:
- ./volumes/nginx:/etc/nginx
networks:
- localnet
volumes:
mongo_data:
kafka_data:
driver: local
networks:
localnet:
attachable: true
NGINX Configuration NGINX Dockerfile (nginx/Dockerfile)
FROM nginx:latest
RUN rm /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/conf.d
NGINX Configuration File (nginx/nginx.conf)
server {
listen 80;
server_name your_domain.com;
location / {
proxy_pass http://sparrow:9000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
location ~ /.well-known/acme-challenge {
allow all;
root /var/www/certbot;
}
}
server {
listen 443 ssl;
server_name your_domain.com;
ssl_certificate /etc/letsencrypt/live/your_domain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/your_domain.com/privkey.pem;
location / {
proxy_pass http://sparrow:9000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
Certbot Dockerfile (nginx/certbot/Dockerfile)
FROM certbot/certbot
Run docker-compose up -d
to start all services.
Ensure you have a valid domain name and update your_domain.com with your actual domain in NGINX configuration files.
How to install in ubuntu
Hosting our Sparrow API on Ubuntu, This guide provides step-by-step instructions for self-hosting our Sparrow API on an Ubuntu server.
Before you begin
Prerequisites
- Ubuntu server version (22.04.2)
- Sparrow API codebase
- Node.js
- MongoDB
- Kafka
Install MongoDB Community Edition
From a terminal, install gnupg
and curl
if they are not already available:
sudo apt-get install gnupg curl
To import the MongoDB public GPG key, run the f`ollowing command:
curl -fsSL https://www.mongodb.org/static/pgp/server-7.0.asc | \ sudo gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg \ --dearmor
Create a list file for MongoDB
Create the list file /etc/apt/sources.list.d/mongodb-org-7.0.list
for version Ubuntu version.
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/7.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-7.0.list
Reload local package database
Issue the following command to reload the local package database:
sudo apt-get update
Install the MongoDB packages
Install the latest stable version of MongoDB.
sudo apt-get install -y mongodb-org
Start MongoDB
Start the mongod
process by issuing the following command:
sudo systemctl start mongod
If you receive an error similar to the following when starting mongod
:
Failed to start mongod.service: Unit mongod.service not found.
Run the following command first:
sudo systemctl daemon-reload
Then run the start command above again.
Verify that MongoDB has started successfully.
sudo systemctl status mongod
Optionally ensure that MongoDB will start following a system reboot by issuing the following command:
sudo systemctl enable mongod
Stop MongoDB
As needed, stop the mongod process by issuing the following command:
sudo systemctl stop mongod
Restart MongoDB
Restart the mongod
process by issuing the following command:
sudo systemctl restart mongod
Follow the state of the process for errors or important messages by watching the output in the /var/log/mongodb/mongod.log
file.
Install Kafka
Since Apache Kafka is written in Java, Installation of Java is a prerequisite. So, log in to server and refresh the local package index.
sudo apt update
Next, install OpenJDK which is a free and open-source implementation of the Java Standard Edition Platform. Here, we are installing OpenJDK 11 which is an LTS release.
sudo apt install openjdk-11-jdk -y
Once installed, verify the version of Java as shown.
java -version
Install Apache Kafka
To download it, use the wget
command-line utility.
wget https://downloads.apache.org/kafka/3.5.1/kafka_2.13-3.5.1.tgz
Next, extract the tarball file using the tar
command-line tool.
tar xvf kafka_2.13-3.5.1.tgz
Once extracted, a folder called kafka_2.12-3.5.0 is created. Next, move this folder to the /usr/local directory and rename it kafka.
sudo mv kafka_2.13-3.5.1 /usr/local/kafka
Create Kafka and ZooKeeper Systemd Unit files
In this step, we will create systemd unit files for Kafka and ZooKeeper services. This will allow to easily manage the services using the systemctl command.
Let’s start by creating the Zookeeper systemd file using the nano editor as shown.
$ sudo nano /etc/systemd/system/zookeeper.service
Paste the following lines of code which define Zookeeper’s systemd service.
[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]
Type=simple
ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target
Save the changes and exit.
Next, create Kafka’s systemd file
$ sudo nano /etc/systemd/system/kafka.service
Paste the following lines of code which define Kafka’s systemd service.
[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh
[Install]
WantedBy=multi-user.target
Be sure to save the changes and exit
sudo systemctl start zookeeper
sudo systemctl start kafka
Confirm if the services are running. For Zookeeper, run:
sudo systemctl status zookeeper
For Apache Kafka service , execute:
$ sudo systemctl status kafka
Install Node.js
Ensure that Node.js is installed on Ubuntu server. You can do this by running the following commands:
sudo apt update
sudo apt install nodejs
Install npm & pnpm
npm (Node Package Manager) should be installed along with Node.js. Verify its installation by running:
npm -v
pnpm install
Set Up Environment Variables
Application requires environment variables for MongoDB and Kafka connection URLs. Create a .env
file in your project directory and add the necessary variables:
MONGODB_URI=mongodb://username:password@mongodb_host:27017/database_name
KAFKA_URL=kafka://kafka_host:9092
Make sure to replace username
, password
, mongodb_host
, database_name
, and kafka_host
with your actual MongoDB and Kafka connection details.
Start Your Sparrow API
pnpm start:dev
Accessing Sparrow API swagger
http://localhost/api/docs