HTS Bioinf - Running the Anno service

Scope

The procedure describes the process for running Anno as an API service in production:

start and stop the service
monitor the infrastructure
handle errors

Responsibility

Responsible person: Production bioinformatician.

Definitions

ELLA: software for interpretations of genetic variants.

Supervisor: software module to monitor processes.

Supervisor UI: the web page to access Supervisor functionality.

TSD: Tjenester for Sensitive Data (infrastructure at USIT, University of Oslo)

What are Anno and Anno-targets

Background and specifications of Anno and Anno-targets are provided here.

We currently run Anno as both a service on TSD and an integrated sub-workflow in the pipeline and the backup pipeline.

Location of the service

The service is running on TSD at the University of Oslo.

The URL of the (production, staging, test and validation) API instances is http://p22-anno-01:9000. Each instance is connected to the corresponding ELLA instance.

The base location of Anno on TSD is /ess/p22/data/durable/production/anno.

Supervisor (process manager)

Anno is setup to be managed by Supervisor. Supervisor's configuration is located in ops/supervisor/supervisor.cfg.

To start the Supervisor process orchestrating all instances of Anno:

{serviceuser login} p22-anno-01

run

cd /ess/p22/data/durable/production/anno
ops/run-supervisor.sh

This will start a Supervisor process as service user.

tail the log at ops/logs/prod.log and check that the production service has started correctly and that requests are handled.

To check running Anno processes:

# check running Supervisor processes
pgrep supervisord | wc -l
# 5

# show info for all supervisord processes
ps uww $(pgrep supervisord)
# USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
# 1200       11084  0.0  0.0  28108 22880 ?        S    10:23   0:04 /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisord.conf
# tsnowlan   11106  0.0  0.0  68244 23148 ?        S    10:23   0:07 /dist/ella-python/bin/python3.7 /dist/ella-python/bin/supervisord -c /ella/ops/dev/supervisor.cfg
# 1200       11657  0.0  0.0  33504 23104 ?        S    10:23   0:05 /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisord.conf
# 1200       11808  0.0  0.0  33392 23268 ?        S    10:23   0:05 /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisord.conf
# tsnowlan   14791  0.0  0.0  68244 23032 ?        S    10:23   0:06 /dist/ella-python/bin/python3.7 /dist/ella-python/bin/supervisord -c /ella/ops/dev/supervisor.cfg

To view Supervisor's UI and check the status of the various processes run:

ops/supervisorctl.sh

On Supervisor's UI p22-anno-01:9000/ click on the respective instance:

figure-anno-1

It will show you the log with the current Singularity image version pinned at the top and tail of the same log (ops/logs/prod.log for instance prod):

figure-anno-2

To stop Anno entirely:

stop all the processes from Supervisor's UI
kill all processes on p22-anno-01:9000 run by the service user:
```
killall -u p22-serviceuser01
```

Logs

Log files are located in ops/logs/.

Handling TSD Errors

When infrastructure errors are detected, send an email to tsd-drift@usit.uio.no. Use the email link on the TSD contact page and fill in the required fields. Put diag-bioinf in CC so updates are distributed to relevant people.

Handling other Errors

When users report errors, try to figure the source of error (user, application, system) and take appropriate action. Add diag-bioinf in CC to any relevant emails.

If a task fails, try to inspect the logs first, and then the work directory of this task (located in anno-<instance>/work/<taskid>). If you don't have the task ID, check the timestamp of the directory in work. Some useful files here are:

STATUS (check latest step)
target.source (check environment variables. Useful to determine what was requested, and what was sent to the ella target)
<annotation step>/output.log (output from each annotation step, possibly containing error messages)