Manual Deployment

Note

We recommend the Docker install for a consistent experience.

Requirements

• Python 3.8 or later
• MySQL 8.0.11 or later / MariaDB 10.4 or later (the minimum for Django 4.2; the official Docker image uses MariaDB 10.8)
• Redis 5.0 or later
• Git 2.17.0 or later (needed to create standard deploy requests)
• rsync, sshfs (file distribution and pipeline data transfer), ping (monitoring ping checks)
• A modern browser

Steps

The steps assume CentOS 7 and the directory /data/spug; on Ubuntu / Debian only the package installation command differs.

1. Clone the repository

git clone https://github.com/openspug/spug /data/spug
cd /data/spug
git checkout v4.0.0

2. Download the pre-built front-end

Extract the bundle into spug_web (it creates the build directory):

curl -o /tmp/web_v4.0.0.tar.gz https://cdn.spug.cc/spug/web_v4.0.0.tar.gz
tar xf /tmp/web_v4.0.0.tar.gz -C /data/spug/spug_web/

Tip

You can also build it yourself with npm run build as described in Development setup.

3. Create the runtime environment

# CentOS dependencies
yum install -y mariadb-devel python3-devel gcc pkgconfig openldap-devel cyrus-sasl-devel openssl-devel redis nginx supervisor rsync fuse-sshfs iputils git

# Ubuntu / Debian dependencies
# apt install -y libmariadb-dev python3-dev python3-venv gcc pkg-config libldap2-dev libsasl2-dev libssl-dev redis-server nginx supervisor rsync sshfs iputils-ping git

# virtual environment
cd /data/spug/spug_api
python3 -m venv venv
source venv/bin/activate

# Python packages
pip install -U pip setuptools -i https://pypi.tuna.tsinghua.edu.cn/simple/
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple/
pip install mysqlclient -i https://pypi.tuna.tsinghua.edu.cn/simple/   # gunicorn and daphne are already in requirements.txt

4. Configure the back-end

The back-end uses SQLite by default; switch it to MySQL with the settings below. Using SQL Server?

Note

Create overrides.py in spug_api/spug/; it overrides the defaults when the services start. Do not edit settings.py, so that future updates apply cleanly.

spug_api/spug/overrides.py

DEBUG = False
# replace with a random string, used to sign sessions
SECRET_KEY = 'replace-me-with-a-random-string'

DATABASES = {
    'default': {
        'ATOMIC_REQUESTS': True,
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'spug',             # database name, create it with the utf8mb4 charset first
        'USER': 'spug_user',        # database user
        'PASSWORD': 'spug_passwd',  # database password
        'HOST': '127.0.0.1',        # database host
        'OPTIONS': {
            'charset': 'utf8mb4',
            'sql_mode': 'STRICT_TRANS_TABLES',
            #'unix_socket': '/opt/mysql/mysql.sock' # socket path for a local, non-default MySQL installation
        }
    }
}

5. Initialize the database

cd /data/spug/spug_api
python manage.py updatedb

6. Create the administrator account

python manage.py user add -u admin -p spug.cc -s -n Administrator

# -u user name
# -p password
# -s super administrator
# -n display name

7. Create the service definitions

/etc/supervisord.d/spug.ini

[program:spug-api]
command = bash /data/spug/spug_api/tools/start-api.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/api.log
redirect_stderr = true

[program:spug-ws]
command = bash /data/spug/spug_api/tools/start-ws.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/ws.log
redirect_stderr = true

[program:spug-worker]
command = bash /data/spug/spug_api/tools/start-worker.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/worker.log
redirect_stderr = true

[program:spug-monitor]
command = bash /data/spug/spug_api/tools/start-monitor.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/monitor.log
redirect_stderr = true

[program:spug-scheduler]
command = bash /data/spug/spug_api/tools/start-scheduler.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/scheduler.log
redirect_stderr = true

This file is identical to spug_api/tools/supervisor-spug.ini in the repository, so you can simply copy it: cp /data/spug/spug_api/tools/supervisor-spug.ini /etc/supervisord.d/spug.ini. The services: spug-api is the HTTP API (gunicorn on 127.0.0.1:9001), spug-ws the WebSocket service (daphne on 127.0.0.1:9002), spug-worker runs batch executions, deployments and pipelines, and spug-monitor / spug-scheduler are the monitoring and scheduling processes. On Ubuntu / Debian the supervisor directory is /etc/supervisor/conf.d/ and files end with .conf.

8. Create the Nginx configuration

/etc/nginx/conf.d/spug.conf

server {
        listen 80;
        server_name _;     # set your domain
        root /data/spug/spug_web/build/;
        client_max_body_size 20m;   # limits uploads in the file manager, adjust as needed

        gzip  on;
        gzip_min_length  1k;
        gzip_buffers     4 16k;
        gzip_http_version 1.1;
        gzip_comp_level 7;
        gzip_types       text/plain text/css text/javascript application/javascript application/json;
        gzip_vary on;

        location ^~ /api/ {
                rewrite ^/api(.*) $1 break;
                proxy_pass http://127.0.0.1:9001;
                proxy_read_timeout 180s;
                proxy_redirect off;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        location ^~ /api/ws/ {
                rewrite ^/api(.*) $1 break;
                proxy_pass http://127.0.0.1:9002;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "Upgrade";
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        location / {
                try_files $uri /index.html;
        }
}

Note

If you do not set a server_name in spug.conf, comment out or delete the default server block in /etc/nginx/nginx.conf, otherwise the Nginx welcome page is shown.

9. Start the services

# enable on boot
systemctl enable nginx
systemctl enable redis
systemctl enable supervisord

# start
systemctl restart nginx
systemctl restart redis
systemctl restart supervisord

10. Open Spug

Open the site in the browser and log in with the account created in step 6.

11. Security recommendations

• Make sure Redis listens on 127.0.0.1 only. For a password-protected Redis see Using a Redis server with a password.
• When adding another proxy layer, forward the X-Forwarded-For header correctly; Spug uses it for security (tokens expire when the client IP changes), see Best practices.
• Enable login MFA under Settings / Security Settings and restrict access with a firewall.