Skip to main content

Self-Hosting Deployment Guide for Sparrow on Ubuntu

Hosting our Sparrow API on Ubuntu, This guide provides step-by-step instructions for self-hosting our Sparrow API on an Ubuntu server.

info

Prerequisites

  • Ubuntu server version (22.04.2)
  • Java version v17
  • Kafka (Kafka 4.0.0)
  • MongoDB (v7.0.18)
  • Node.js (v20.04 LTS)
  • Sparrow API codebase

Install Apache Kafka on Ubuntu

Step 1: Creating a user for Kafka

The first step is to create a dedicated user to ensure that Kafka's operations do not interfere with the system's other functionalities.

Add a new user called kafka:

sudo adduser kafka

Next, you need to add the kafka user to the sudo group to have the necessary privileges for Kafka installation.

sudo adduser kafka sudo

Then, log in to the kafka account:

su -l kafka

The kafka user now is ready to be used.

Step 2: Installing Java Development Kit (JDK)

Open the terminal and update the package index:

sudo apt update

Install the OpenJDK 17 package 

sudo apt install openjdk-17-jdk -y
java -version

Now that you’ve installed the JDK, you can start downloading Kafka.

Step 3: Downloading Kafka

Start by creating a folder named downloads to store the archive:

mkdir ~/downloads
cd ~/downloads
wget "https://downloads.apache.org/kafka/4.0.0/kafka_2.13-4.0.0.tgz"

Then, move to ~ and extract the archive you downloaded:

cd ~
tar -xvzf ~/downloads/kafka_2.13-4.0.0.tgz

Let’s rename the directory kafka_2.13-4.0.0 to kafka.

mv kafka_2.13-4.0.0/ kafka/

Now that you’ve downloaded Kafka, you can start configuring your Kafka server.

Step 4: Configuring the Kafka server

First, start by setting the log.dirs property to change the directory where the Kafka logs are.

To do so, you need to edit the server.properties file:

nano ~/kafka/config/server.properties

Look for log.dirs and set the value to /home/kafka/kafka-logs.

info

Note: This is where Kafka stores its data logs, not application logs.

Step 5: Starting the Kafka server

To start the Kafka server, you need to first start Zookeeper and then start Kafka.

But, to be more efficient, you need to create systemd unit files and use systemctl instead.

Unit File for Zookeeper 
sudo nano /etc/systemd/system/zookeeper.service
[Unit]
Description=Apache Zookeeper Service
Requires=network.target                 
After=network.target                 

[Service]
Type=simple
User=kafka
ExecStart=/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties        
ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target
Unit File for Kafka 
sudo nano /etc/systemd/system/kafka.service
[Unit]
Description=Apache Kafka Service that requires zookeeper service
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
ExecStart= /home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties                            
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Then, you can start the Kafka server:

sudo systemctl start kafka

Check the status:

sudo systemctl status kafka

info
If Kafka is Not Working, Switch to KRaft Mode

Initialize Kafka in KRaft Mode (Kafka 4.x)

Kafka 4.x no longer requires Zookeeper to store cluster metadata. Instead, it uses KRaft mode (Kafka Raft Metadata mode), where Kafka stores metadata like topics, partitions, and brokers in its own internal log. To run Kafka in KRaft mode, you must initialize and format this internal log before starting the Kafka server.

Create the Logs Directory

Kafka needs a dedicated directory to store its internal logs and metadata. This step creates that directory.

mkdir -p /tmp/kraft-combined-logs

Generate a Cluster UUID

Kafka requires a unique Cluster UUID to identify the Kafka cluster. This UUID is used across reboots, upgrades, and multi-node setups.

/home/kafka/kafka/bin/kafka-storage.sh random-uuid

Save the UUID generated by this command; you will need it for the next step.

Format the Kafka Metadata Storage

In KRaft mode, Kafka stores all metadata (like topics, partitions) internally. You need to format this metadata log using the generated Cluster UUID.

/home/kafka/kafka/bin/kafka-storage.sh format \   
--config /home/kafka/kafka/config/server.properties \   
--cluster-id <cluster-id> \   
--standalone

Replace cluster-id with the UUID generated in the previous step. The --standalone flag indicates that this is a single-node setup.

Start Kafka

Now that the metadata storage is initialized, you can start Kafka.

sudo systemctl start kafka
sudo systemctl status kafka

Kafka will now run in KRaft mode and use its internal log to store metadata.

Step 6: Testing the Kafka server

You can check if the Kafka server is up with netcat. By default, Kafka server runs on 9092

nc -vz localhost 9092

Install MongoDB Community Edition

Step 1: Import the public key.

From a terminal, install gnupg and curl if they are not already available.

sudo apt-get install gnupg curl

To import the MongoDB public GPG key, run the following command:

curl -fsSL https://www.mongodb.org/static/pgp/server-7.0.asc | \
sudo gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg \
--dearmor

Step 2: Create the list file

Create the list file /etc/apt/sources.list.d/mongodb-org-7.0.list for version Ubuntu version.

echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-7.0.list

Step 3: Reload the package database

Issue the following command to reload the local package database:

sudo apt-get update

Step 4: Install MongoDB Community Server

You can install either the latest stable version of MongoDB.

sudo apt-get install -y mongodb-org
#Start MongoDB
sudo systemctl start mongod

#Verify that MongoDB has started successfully
sudo systemctl status mongod

#You can optionally ensure that MongoDB will start following a system reboot by issuing the following command:
sudo systemctl enable mongod

Install Node.js

Ensure that Node.js is installed on Ubuntu server. You can do this by running the following commands:

sudo apt update
sudo apt upgrade
sudo apt install -y curl
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt install -y nodejs
node --version

Install npm & pnpm

npm (Node Package Manager) should be installed along with Node.js. Verify its installation by running:

sudo apt install npm
npm -v
sudo apt install pnpm
curl -fsSL https://get.pnpm.io/install.sh | sh -

Then add to your shell config:

export PATH="$HOME/.local/share/pnpm:$PATH"
source ~/.bashrc  # or ~/.zshrc
pnpm --version

Clone Sparrow api repository 

git clone https://github.com/sparrowapp-dev/sparrow-api.git

Move into the project directory

cd sparrow-api

Install PNPM globally 

sudo npm i -g pnpm

Rename .env.example to .env 

cp .env.example .env

Install dependencies 

pnpm i

To Secure Nginx with Let's Encrypt on Ubuntu 20.04

info

Prerequisites

  • Ubuntu server
  • A registered domain name
  • Nginx installed

Step 1: Installing Certbot

The first step to using Let’s Encrypt to obtain an SSL certificate is to install the Certbot software on your server.

Install Certbot and it’s Nginx plugin with apt:

sudo apt install certbot python3-certbot-nginx

Certbot is now ready to use, but in order for it to automatically configure SSL for Nginx, we need to verify some of Nginx’s configuration.

Step 2: Confirming Nginx's Configuration

Certbot needs to be able to find the correct server block in your Nginx configuration for it to be able to automatically configure SSL. Specifically, it does this by looking for a server_name directive that matches the domain you request a certificate for.

Install and Configure Nginx on Ubuntu

Nginx is available in Ubuntu’s default package repositories, which makes installation straightforward using the apt package manager. Before installing, it's a good practice to update your local package index to ensure you have access to the latest software listings.

Update the package index 

sudo apt update

Install Nginx 

sudo apt install nginx

Adjusting the firewall

Before testing Nginx, the firewall software needs to be adjusted to allow access to the service. Nginx registers itself as a service with ufw upon installation, making it straightforward to allow Nginx access.

List the application configurations that ufw knows how to work with by typing.

sudo ufw app list

Allow Nginx Full (for HTTP and HTTPS)

sudo ufw allow 'Nginx HTTP'

You can verify the change by typing

sudo ufw status

Checking Your Webserver

At the end of the installation process, Ubuntu starts Nginx. The web server should already be up and running.

We can check with the systemd init system to make sure the service is running by typing:

systemctl status nginx

To find your server’s public IP address, run:

curl -4 icanhazip.com

When you have your server’s IP address, enter it into your browser’s address bar:

http://your_server_ip

you should have a server block for your domain at /etc/nginx/sites-available/==example.com== with the server_name directive already set appropriately.

To check, open the configuration file for your domain using nano or your favorite text editor:

sudo nano /etc/nginx/sites-available/default
server {
    server_name <host name>;

    location / {
        proxy_pass http://127.0.0.1:9000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    listen 80;
}

To enable the server block configuration for your domain, create a symbolic link from the sites-available directory to the sites-enabled directory:

sudo ln -s /etc/nginx/sites-available/default /etc/nginx/sites-enabled/

Test Nginx Configuration

Verify your configuration syntax to ensure there are no errors:

sudo nginx -t

If you see an error, reopen the config file and fix any typos. Once the syntax is correct, reload Nginx to apply changes:

sudo systemctl reload nginx

Certbot can now find the correct server block and update it automatically.

Step 3: Allowing HTTPS Through the Firewall

If UFW is enabled, allow HTTPS traffic:

sudo ufw status

To additionally let in HTTPS traffic, allow the Nginx Full profile and delete the redundant Nginx HTTP profile allowance:

sudo ufw allow 'Nginx Full'
sudo ufw status

Next, let’s run Certbot and fetch our certificates.

Step 4: Obtaining an SSL Certificate with Certbot

Use Certbot’s Nginx plugin to get and install an SSL certificate:

sudo certbot --nginx -d <domain_name>

This runs certbot with the --nginx plugin, using -d to specify the domain names we’d like the certificate to be valid for.

Step 5: Verifying Certbot Auto-Renewal

Let’s Encrypt certificates expire every 90 days. Certbot automatically sets up a systemd timer to renew certificates within 30 days of expiration.

To check the status of the timer:

sudo systemctl status certbot.timer

To test the renewal process, you can do a dry run with certbot:

sudo certbot renew --dry-run

Now you can start your backend service

Run App in development mode 

pnpm start:dev

Access swagger on public IP or domain:

Go to https://<public_ip/domain_name>:{PORT}/api/docs