Testnet Node migration to Ubuntu 20.04

1. Introduction

Since Canonical stopped supporting Ubuntu 18.04 for free, nodes should be migrated to a newer version. 2.1.1 release is supporting of Ubuntu 20.04 (Focal Fossa).

22.04 (Jammy Jellyfish) is not yet fully supported. Please, use 20.04 exclusively.
If you have SGX wallet on the same machine as a node you still can to the steps below, but be extra careful, since this setup is not verified by Core Team.

2. Backup 18.04 SGX server

Do not register, exit nodes or shut down old SGX server before Core Team confirmation.

Switch to run_sgx folder

Change directory to sgxwallet/run_sgx, which is usually located in /root directory.

cd ~/sgxwallet/run_sgx

Stop SGX container

docker-compose down

Save sgx_data folder

You can use scp, rsync or other tools. For example from your local machine run

scp -r user@SGX_IP:~/sgxwallet/run_sgx/sgx_data /path/to/create/local/backup

Restart SGX container

docker-compose up -d

Make sure backup key is saved

Make sure backup key is stored somewhere. Otherwise, recovery is impossible and the node funds are lost.

You display check backup key like this

cat ~/sgxwallet/run_sgx/sgx_data/sgxwallet_backup_key.txt

3. Provision 20.04 machine

Setup machine with Ubuntu 20.04 installed following the requirements.

For OVHCloud you can use the steps below:

  • Open your server control panel.

  • Click "Install" in OS/Distribution section (under three dots button).

  • Choose Ubuntu 20.04.

To install required dependencies you can run

sudo apt-get install -y docker.io docker-compose libelf-dev cpuid build-essential

4. Setup 20.04 SGX server

Clone sgxwallet repo

git clone https://github.com/skalenetwork/sgxwallet.git; cd sgxwallet; git checkout 1.9.0-stable.0

Copy backup folder to the new server

You can use scp to do it.

scp -r /path/to/local/backup user@SGX_NEW_IP:~/sgxwallet/run_sgx/sgx_data

Copy backup key to the new server

Verify that your saved SGX backup key is equal on both 18.04 and 20.04 servers. 
cat ~/sgxwallet/run_sgx/sgx_data/sgxwallet_backup_key.txt

Check connection to intel enclave whitelist

Verify that your network and firewall configuration allows connections to enclave whitelist updates.

curl -I http://whitelist.trustedservices.intel.com/SGX/LCWL/Linux/sgx_white_list_cert.bin

Verify that SGX is supported and enabled

cpuid | grep SGX:

Output should be: SGX: Software Guard Extensions supported = true.

Enable software-controlled SGX

sudo ~/sgxwallet/sgx_enable

Install SGX driver

bash ~/sgxwallet/scripts/sgx_linux_x64_driver_*.bin

Reboot machine

sudo reboot

Check if new sgx device is present

Verify that /dev/isgx appears in ls output.

ls /dev/isgx

Increase max process limit

Make sure DefaultLimitNOFILE is set to 65535.

cat /etc/systemd/system.conf | grep 'DefaultLimitNOFILE'

If it’s not, then edit the file and reboot to apply the changes.

Find device names

SGX container will need serveral devices to be linked. You can find them using

ls /dev/ | grep -E '^sg.$|^mei.$|^bt.$'

One of such devices should be /dev/isgx.

Modify docker-compose.yml file

Make sure ~/sgxwallet/run_sgx/docker-compose.yml has modifications listed below.

  • image is set to skalenetwork/sgxwallet_release:1.9.0-stable.0.

  • set devices from the previous step in device and healthcheck → test sections.

  • remove -d flag and add -b flag in the command section to restore from backup.

Run SGX container

Run the following command to spin up container on the new server.

cd ~/sgxwallet/run_sgx
docker-compose up -d
docker-compose logs --tail 100

If SGX Server started printed in logs - restore is completed.

5. Migrate nodes to the new server

For every node that is connected to 18.04 server run the steps below to link them to the new one.

Modify .env

Update the following parameters to the new values

SGX_SERVER_URL=[NEW SERVER URL]

Execute update

skale node update .env -y

6. Shut down 18.04 SGX server

After sgx was restored successfully you can safely shut down the old one.

SGX migration success status should be verified with Core Team. Please, contact the team using default communication channel.

7. Upgrade 18.04 node server to 2.1.1

If you have multiple nodes, please, do this and the following step node by node.

Upgrade node-cli

Download node-cli binary

curl -L https://github.com/skalenetwork/node-cli/releases/download/2.3.0/skale-2.3.0-Linux-x86_64 > /usr/local/bin/skale

Verify checksum

sha512sum /usr/local/bin/skale

Expected checksum is stored here https://github.com/skalenetwork/node-cli/releases/download/2.3.0/skale-2.3.0-Linux-x86_64.sha512

Make node-cli executable

chmod +x /usr/local/bin/skale

Modify .env

Update the following parameters to the new values.

CONTAINER_CONFIGS_STREAM=2.1.16

Execute update

skale node update .env -y

8. Backup the 18.04 server

Run skale node backup

skale node backup .

Output example:

Backup archive successfully created ~/skale-node-backup-2023-04-20-09-24-53.tar.gz

Check that backup archive is present

ls | grep skale-node-backup

Save backup tarball and .env config on another machine

You can use scp, rsync or some other tools. For example from your local machine run

scp user@NODE_IP:~/skale-node-backup* /path/to/create/local/backup
scp user@NODE_IP:~/.env /path/to/create/local/.env

9. Turn off the 18.04 node

To switch off the old 18.04, please, run

skale node turn-off --yes

10. Prepare 20.04 machine

You can reinstall OS on the current server or provision a new one.

The IP of the node should not be changed during the migration procedure.

SKALE node uses two different disks:

  • mounted as /.

  • separate one for chains' data.

The doc assumes that in case of reinstallation the first (root) disk was wiped out. If it’s not, please, contact Core Team.

Provision the machine

Setup machine with Ubuntu 20.04 installed following the requirements.

Or reinstall a new one. For OVHCloud you can use the steps below:

  • Open your server control panel.

  • Click "Install" in OS/Distribution section (under three dots button).

  • Choose Ubuntu 20.04.

To configure required dependencies you can run the following script.

#!/usr/bin/env bash
set -ea

sudo apt-get update
sudo apt install apt-transport-https ca-certificates curl gnupg software-properties-common iptables-persistent -y
sudo mkdir -m 0755 -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
wget -qO - apt-key add https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
echo \
"deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin -y
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
sudo ulimit -Sc 1000000000
sudo ulimit -Hc 1000000000
sudo timedatectl set-ntp on
sudo service systemd-timesyncd enable > /dev/null 2>&1

It will install iptables-persistent, docker, docker-compose, adjust resource limits and enable timesyncd daemon.

Newer version of docker-compose is required - 1.29.2 (on Ubuntu 18.04 it was 1.27.4)

Copy the backup archive to the new machine

Like during the backup step you can use scp.

scp /path/to/create/local/backup user@NEW_NODE_IP:~/
scp /path/to/create/local/.env user@NEW_NODE_IP:~/

Download node-cli

Download node-cli binary

curl -L https://github.com/skalenetwork/node-cli/releases/download/2.3.0/skale-2.3.0-Linux-x86_64 > /usr/local/bin/skale

Verify checksum

sha512sum /usr/local/bin/skale

Expected checksum is stored here https://github.com/skalenetwork/node-cli/releases/download/2.3.0/skale-2.3.0-Linux-x86_64.sha512.

Make node-cli executable

chmod +x /usr/local/bin/skale

11. Restore backup on the 20.04 node

There is a different restore procedure depending on how the chain’s disk was migrated.

If the disk was saved from the previous machine lvs | grep shared-space should output something like

shared-space         schains -wi-ao---- <8.35g

In case shared-space volume is present execute.

skale node restore skale-node-backup.tar.gz .env --no-snapshot

Otherwise, please, run regular restore.

skale node restore skale-node-backup.tar.gz .env

12. Contact Core Team

To finalize migration, please, connect with the team using default communication channel.

Edit on Github