Installation and Conﬁguration¶

This document describes installation, conﬁguration and basic monitoring and management tasks for on-premises installation of the TrackNet TrackCentral (TC) LoRaWAN network server (LNS). The installation described in this document targets an installation of the LNS on an Ubuntu 18.04 LTS server installation. Both LNS server and database are deployed in a co-located setup. Multi-VM and high-availability conﬁgurations are outside of the scope of this document. On request, a TC LNS can also be shipped as a pre-installed and pre-conﬁgured VM which only requires the documented post-installation steps.

Installation Depedencies¶

The TC LNS depends on a number of available system environment services and pre-installed system packages. The TC LNS install scripts will install any required software packages, the system environment has to be setup before the installation can take place.

System Environment Dependencies¶

The TC LNS installation process requires the following environment to be setup before installation:

NTP services and UTC time zone: It is expected that time is well synchronized on the server and the timezone is set to Etc/UTC. Use timedatectl on Ubuntu 18.04 to verify and modify the timezone settings.
The fully qualiﬁed domain name of the server host must point to the target host/VM of the installation. This installation document assumes that DNS is used and correctly setup to provide domain name and address resolution.
HTTP API server and Web UI are secured by HTTPS where by default the installation requests a certiﬁcate from ‘Let’s Encrypt’ (https://letsencrypt.org). Alternatively, the server processes may rely on self-signed certiﬁcates which are required for secure web socket communication between back end applications and server.
Setup a slack workspace, channel (https://slack.com/) and an incoming webwook for notiﬁcations of system and router events. This allows for an eﬃcient group communication of operatots on alarms. Dependent on the on-premises environment, notiﬁcations can be redirected to other services.
This installation documentation assumes installation for the user ‘ubuntu’ with a home directory /home/ubuntu. During installation, the user must have sudo rights. This is not necessarily required during operation. Language must be set on the command line to US English, add ‘export LC_ALL=C’ to ~/.proﬁle. It is expected that a SSH connection or console is available on the target host where the installation scripts are executed.

System Packages¶

The TC LNS installation process downloads and installs a number of Ubuntu 18.04 system packages and their dependencies:

build-essential, iptables-persistent, libdpkg-perl
git, python3.6, python3.6-dev, libﬃ-dev, virtualenv
gcc, g++, make, libssl-dev, libwebsockets-dev
postgresql–10, postgresql-server-dev–10
curl, letsencrypt
htop, tmux, zip, awscli

Python Packages and Environment¶

The TC LNS installation process downloads and installs a number of python packages. The packages are stored in a virtual python environment (~/pyenv) and are not installed in a system directory. The virtual python environment requires python 3.6. The TC LNS installation process uses a pip mirror site for download which is conﬁgured in ~/.pip/pip.conf with the following contents:

[global]
index-url = https://pypi.python.org/simple

Installation Process¶

The installation requires copying and unpacking the installation ﬁle and the execution of several installation scripts. The following instructions reference the following variables:

TCSERVERHOST: fully qualiﬁed domain name of server such as tc01.domain.com

The individual steps of the installation are:

Check whether you have received amore recent installation instructions with the installation ﬁle.
Copy over the installation ﬁle to the ubuntu home directory. The version and ﬁle name of the installation ﬁle may vary.

$ scp tc_TrackCentral_1.2.2-29-g4ff40fe1.tgz  TCSERVERHOST:~

Login to the server and unpack the installation ﬁle in the home directory of the ubuntu user.

$ ssh TCSERVERHOST
$ tar xzf tc_TrackCentral_1.2.2-29-g4ff40fe1.tgz

Run the script ‘setup-tcbase’ from the unpacked directory. This installs the required binary packages and python environment. After that, a directory /home/ubuntu/pyenv with the python environment and python packages exist. This script can be restarted when required.

$ ./recipes/ubuntu-18.04/setup-tcbase.sh

Run the script ‘setup-tcdev’ from the unpacked directory. This prepares the database and code repository. If ‘fqdn’ is set, the “Let’s Encrypt” certiﬁcates are installed. This script can be restarted when required.

$ fqdn=TCSERVERHOST ./recipes/ubuntu-18.04/setup-tcdev.sh release=./trackcentral

Install the server conﬁguration into the directory ~/tc. Run the script without parameters for help and options.

$ ./recipes/ubuntu-18.04/setup-simple-tc-server/install ~/tc hostname=TCSERVERHOST tls=extern

The script creates and ﬁlls the initial database, creates and populates the conﬁguration directory ~/tc and creates and links the systemd script tc.tc

The script prints the server conﬁguration after successful installation. This can also be found in ~/tc/README.txt. If installation only partly succeeds, ~/tc/uninstall can be used to remove the installation. Installation can then be restarted using the install script.

Server Execution¶

For checking a successful installation, start the server:

 $ sudo systemctl start tc.tc

Log into the user interface (https://TCSERVERHOST:7008/index.html) with login ‘admin’ and password ‘admin’. The UI interface lists a router and device which already generate traﬃc. Router and device are simulated and part of the Health Check service.

Stop the server with:

$ sudo systemctl stop tc.tc

Post-Installation Steps¶

The following post-installation steps are advised:

Enable tc.tc to restart TrackCentral services automatically after a restart/reboot:

$  sudo systemctl enable tc.tc

Change the admin password in the UI.
Remove executable permission for ~/tc/uninstall. ~/tc/uninstall removes the server installation and deletes the database contents. After a ~/tc/uninstall, the server can be re-installed using the install script.
Add ‘. ~/tc/tcactivate’ to ‘~/.proﬁle’ so that the tc environment is sourced on startup and the family of swark command line tools are accessible on the command line by default.
Optional: If required, setup additional region-speciﬁc channels using the ‘swark.region’ command. Execute ‘swark.regions’ for command line help.
Optional: Setup the backup process. For details, see the ‘Backup and Restore’ section.
Optional: Setup the Slack notiﬁcation for the Health Check process.

TC LNS Installation Overview¶

The following gives an overview of the installed server conﬁguration and processes.

TC LNS Installation Files¶

After the installation of the TC LNS, the following ﬁles and directories reside in the folder of the ubuntu user (/home/ubuntu):

Files/Directory	Content
~/.aws	AWS credentials if required.
~/.pgpass	Postgresql credentials with access speciﬁcation for the postgresql instances.
~/.pip/pip.conf	Pip mirror conﬁguration if required.
~/.pyenv	The virtual python environment with links to python binaries and the downloaded python module. The links to python executables must be executable!
~/tc	Server conﬁguration directory.
~/tc/tc.tc.servicee	Systemd service unit installed for start and stop by systemctl command.
~/tc/circus.ini	Conﬁguration for the process manager Circusd monitoring the TC LNS service processes.
~/tc/setup.yaml	Contains the Postgresql DB Uri used by all TC LNS service processes.
~/tc/logging.yaml	Logger conﬁgurations for TC LNS service processes.
~/tc/tcactivate	Script to be sourced by bash to setup TC LNS environment in bash.
~/tc/s3bucket.txt	Optional. Name of s3 bucket for AWS S3 backups.
~/tc/temp	Directory with application-speciﬁc log ﬁles, the application loggers performs log ﬁle rotation to limit log ﬁle sizes.

Additionally, the TC LNS installation process installs and links a service unit conﬁguration in /etc/systemd/system.

TC LNS Postgresql Conﬁguration¶

The TC LNS requires access to a single database on local/remote Postgresql server instances (by default, the table ‘tc’). The table must be created at installation time. The TC LNS processes do not require permission to create databases during operation but thet require all Postgresql permissions for the created and selected database.

TC LNS Server Processes¶

The tc.tc.service service unit managed by systemd starts the python process manager Circusd with the user permissions of the ubuntu user which then manages all TC LNS processes. The TrackNet LNS service processes do not require administrator privileges. The tc.tc.service service unit can be used and inspected with the following commands:

Starts the service, i.e. the Circusd process manager which then starts all LNS service processes. The Circus process manager logs start messages in the syslog.

$ sudo systemctl start tc.tc.service

Stops the service, i.e. all LNS service processes and Circusd exit. The Circus process manager logs stop/exit messages in the syslog.

$ sudo systemctl stops tc.tc.service

Prints status messages of the service, in a running system, the list of LNS processes.

$ sudo systemctl status tc.tc.service

The TC LNS uses Circusd to manage the LNS service processes. The full documentation for Circusd can be found at https://circus.readthedocs.io/en/latest. The following details the basic commands to list the TC LNS processes managed by Circusd:

# Source environment in the ubuntu home directory
$ . ./tc/tcactivate
# List processes controlled by circusd
$ circusctl status
apis: active
appx: active
door: active
infos: active
isalive: active
joins: active
muxs: active
nwks: active
wmifs: active

This list of Circusd processes translate to the following system process list (output taken from ps command):

python -m swark.apis_cmd ...
python -m swark.appx_cmd ...
python -m swark.door_cmd ...
python -m swark.infos_cmd ...
python -m swark.isalive_cmd ...
python -m swark.joins_cmd start ...
python -m swark.muxs_cmd start ...
python -m swark.nwks_cmd start ...
python -m swark.wmifs_cmd start ...

The roles of the process are as follows and are described for clariﬁcation purposes:

apis: Provides a HTTP interface for backend applications to provision/query devices/routers.
appx: Provides a web socket interface for backend applications to pull/push device data.
infos: Provides a directory service and http/web socket interface to route routers and backend connections to the appropriate processes.
muxs: Handles the communication with routers/gateways and forwards device traﬃc to the nwks.
nwks: Handles the traﬃc from muxs and manages devices. Traﬃc from routers is forwarded to appx, traﬃc from appx is routed to doors.
doors: Manages downlink traﬃc from backend applications (from appx) to devices via muxs.
joins: Handles join requests from devices.
isalives: Health-Check process.
wmifs: HTTP and user interface server

TC LNS Monitoring Overview¶

System Health Checks by IsAlive Process¶

The isalive process provides an end-to-end check for the system health status of the TC LNS server processes. It registers a router and end device in the system and simulates the typical communication behaviour, e.g. router connection, device join operation, device uplink traﬃc, device downlink traﬃc, back end application connection and back end application communicatiion.

Additionally, it measures the latencies for device data traﬃc to pass from end to end through the system. The isalive process logs into ‘~/tc/temp/isalives.log’ and may report alarm conditions to a slack channel.

If a slack channel has been setup and a webhook token is available, it can be added to curcus.ini in ~/tc/circus.ini. The following represents a sample entry in circus.ini with a conﬁgured slack token. Invoke ‘swark.isalive’ for all available options.

...
args = $(circus.env.PYTHONARGS) -m swark.isalive_cmd --id 0 -t H27G7877A/CDFE66DKS/XHHWWaa...
...

Note that the isalive process is executed on the server and cannot detect any connection problems for physical routers or back end applications originating from network problems such as blocking ﬁrewall rules, limited bandwidth, high latency.

Router Alarm Notiﬁcations¶

The ‘swark.ralarm’ command can be used to deﬁne alarm and notiﬁcation proﬁles for routers. Routers can then be assigned to the deﬁned proﬁles for alarm notiﬁcations (by either using the swark.ralarm command or the user interface). The following commands setup and add a router to a proﬁle. Run ‘swark.ralarm’ to see a list of all available options.

# Send an alarm if the router is disconnect for more than one of the listed time spans.
swark.ralarm set --id 1 --name MyProfile --discon 1h,2h --slack H27G7877A/CDFE66DKS/XHHWWaaa678HHjjdjdUTTSdd
# Add a router to this profile
swark.ralarm add 1 router-200...

CPU Monitoring¶

The individual LNS processes should not inhibit extremely high CPU usage if the machine resources were properly selected for the targeted network size. If such behaviour is encountered, stopping and restarting all LNS processes using systemctl may help. High CPU usage typically leads to a failing Health Check process and an alarm notiﬁcation. Dependent on the on premises environment, additonal CPU monitoring tools can be installed and conﬁgured on request.

Disk Monitoring¶

The LNS processes log to rotating log ﬁles in ~/tc/temp and otherwise only rely on a static conﬁguration. Only the database grows over time with the incoming traﬃc of devices and routers. The TC LNS only keeps the data for a retain period which is conﬁgurable for each owner account. With the targeted network size and number of owners, a conservative estimation for the database size can be given and thus disk size is not a concern in normal operation. For drive failures etc, dependent on the on premises environment, additonal disk monitoring tools can be installed and conﬁgured on request.

Memory Monitoring¶

With growing number of devices and gateways, the memory consumption of the LNS processes is expected to increase as the LNS processes partly cache LoRaWAN network state. The cache sizes are conﬁgured for the resources oﬀered by the host machine. Any unusual behaviour such as high memory usage or constant swapping must be examined by TC LNS speciﬁc inspection. These failures are typically detected by the Health Check process and thus lead to an alarm notication. Dependent on the on premises environment, additonal memory monitoring tools can be installed and conﬁgured on request.

Postgresql Monitoring¶

All LNS processes require the connection to the Postgreql server instance. The processes should survive short interruptions of the Postgresql service but a restart might be required for longer or severe outages. These system failures are also typically detected by the Health Check process. Dependent on the on premises environment, additonal Postgresql monitoring tools can be installed and conﬁgured on request.

Backup and Restore¶

Backup¶

The system script ‘tc-backup’ can be used to backup the server conﬁguration directory, code repository and database contents, either into a directory in the ﬁle system (which might be a NSF or SMB share) or to an Amazon S3 store. The tc-backup script is typically managed by circusd and listed in the circus.ini conﬁguration (see System Conﬁguration and Installation).

The following command immediately creates a backup of the TC LNS installation in a temporary folder:

$ tc-backup local=/tmp/TCSERVERHOST

‘tc-backup’ without arguments lists and documents the available options. The created directory contains ﬁles tc_dump.sqlz, tc_install.tgz, trackcentral.tgz which are required for a restore operation.

If the backup is to be stored in an Amazon S3 bucket, the following steps have to be performed:

The home directory must contain an ‘.aws’ folder with the credentials and region setup for AWS authentication.
The ﬁle ‘~/tc’s3bucket.txt’ must exist and contain the name of the S3 bucket the backups are stored.

Finally, an entry in the circus.ini conﬁguration ﬁle looks like this:

[watcher:backup]
virtualenv = /home/ubuntu/pyenv
cmd = tc-backup
args = stime=03:00:00 interval=12 slack=H27G7877A/CDFE66DKS/XHHWWaaa678HHjjdjdUTTSdd
copy_env = true
copy_path = true
graceful_timeout = 5
singleton = true
stdout_stream.class = FileStream
stdout_stream.filename = /home/ubuntu/tc/temp/backup.log
stdout_stream.max_bytes = 100000
stdout_stream.backup_count = 5
stderr_stream.class = FileStream
stderr_stream.filename = /home/ubuntu/tc/temp/backup.log
stderr_stream.max_bytes = 100000
stderr_stream.backup_count = 5

Restore¶

A system can be restored using the ‘tc-restore’ script. A sample restore session works as follows:

Re-create an empty system using the ‘~/trackcentral/recipes/ubuntu–18.04/setup-tcbase’ and ‘~/trackcentral/recipes/ubuntu–18.04/setup-tcdev’ scriipts.
Copy the contents of a ‘tc-backup’ backup directory into the home directory. A backup directory contains ﬁles: tc_dump.sqlz, tc_install.tgz, trackcentral.tgz
Restore the TC LNS server, in the simplest case:

tc-restore backup=~/my-backup-directory

NOTES:

The base machine images for restauration can be pre-populated and saved in an image registry
The ‘tc-restore’ script has an option ‘fqdn=…’ which allows to change the fully qualiﬁed domain name if the restored server is migrated to a new host name.
The ‘tc-restore’ script may optionally restore to a newer version of the code repository.

System Update and Upgrade¶

TC LNS allows to update the code to ﬁx minor implementation bugs or update both code and data to migrate to newer versions with additional functionality. Both updates require the same process, in case of data migration, an upgrade involves updating database rows and might incur a longer downtime. A system update/upgrade involves the following steps.

Shutdown the TC LNS processes:

$ sudo systemctl stop tc.tc

Move the old code base in the trackcentral folder to a temporary or archive directory.

$ mv ~/trackcentral.old trackcentral.prev

Unpack or move the up-to-date code package to ~/trackcentral.
Verify that the update does not involve a data upgrade migration. This command prints the version of the code repository and the data version, i.e. the code version the data was created/manipulated with.

$ swark.system version

If the code version is newer, the database must be upgraded. This updates the rows in the database to the newer code version. Dependent on necessary migration and database size, this step requires longer execution time.

$ swark.system upgrade

Optionally, validate the database and the system.

$ swark.system validate

Restart the TC LNS processes.

$ sudo systemctl start tc.tc

NOTES:

A small code update might not require the restart of all components. Follow the instructions with the concrete update which will state whether just a single process must be restarted.
It is good practice to start a manual backup before the update or upgrade. Use tc-backup for this.
It is also possible to combine a restore from a backup with a code upgrade. In this case, one creates a system with a newer code base, restores a backup and then executes ‘swark.system upgrade’.