Skip to content

HTS Bioinf - Running annoservice

Scope

The procedure describes the process for running anno as an API service in production:

  • start and stop the service
  • monitor the infrastructure
  • handle errors

Responsibility

Responsible person: Production bioinformatician.

Definitions

ELLA: software for interpretations of genetic variants.

Supervisor: software module to monitor processes.

Supervisor UI: the web page to access Supervisor functionality.

TSD: Tjenester for Sensitive Data (infrastructure at USIT, University of Oslo)

What is anno and anno-targets

Background and specifications of anno and anno-targets are provided here.

We currently run anno with anno-targets as both a service on TSD and an integrated sub-workflow in the pipeline and the backup pipeline.

Location of the service

The service is running on TSD at the University of Oslo.

The URL of the (production, staging, test and validation) API instances is http://p22-anno-01:9000. Each instance is connected to the corresponding ELLA instance.

The base location of anno on TSD is /ess/p22/data/durable/production/anno.

Supervisor (process manager)

Anno is setup to be managed by Supervisor. Supervisor's configuration is located in ops/supervisor/supervisor.cfg.

To start the Supervisor process running anno:

  • {serviceuser login} p22-anno-01

  • run

    cd /ess/p22/data/durable/production/anno
ops/run-supervisor.sh

This will start the Supervisor process as service user.

tail the log at ops/logs/prod.log and check that the service has started correctly and that requests are handled.

To check running anno processes:

# check for running supervisor processes
pgrep supervisord | wc -l
# 5

# show info for all supervisord processes
ps uww $(pgrep supervisord)
# USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
# 1200       11084  0.0  0.0  28108 22880 ?        S    10:23   0:04 /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisord.conf
# tsnowlan   11106  0.0  0.0  68244 23148 ?        S    10:23   0:07 /dist/ella-python/bin/python3.7 /dist/ella-python/bin/supervisord -c /ella/ops/dev/supervisor.cfg
# 1200       11657  0.0  0.0  33504 23104 ?        S    10:23   0:05 /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisord.conf
# 1200       11808  0.0  0.0  33392 23268 ?        S    10:23   0:05 /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisord.conf
# tsnowlan   14791  0.0  0.0  68244 23032 ?        S    10:23   0:06 /dist/ella-python/bin/python3.7 /dist/ella-python/bin/supervisord -c /ella/ops/dev/supervisor.cfg

To view the supervisor's UI and check the status of the various processes run:

ops/supervisorctl.sh

Go to the Supervisor page p22-anno-01:9000/ and click on the respective instance:

figure-anno-1

It will show you the log with the current singularity image version pinned at the top and tail of the same log (ops/logs/prod.log for instance prod):

figure-anno-2

To stop supervisor running anno:

  • stop all the processes via supervisor UI

  • kill all processes on p22-anno-01:9000 run by the service user:

    killall -u p22-serviceuser01

Logs

Log files are located in ops/logs/.

Handling TSD Errors

When infrastructure errors are detected, send an email to tsd-drift@usit.uio.no. Use the email link on the TSD contact page and fill in the required fields. Put diag-bioinf in CC so updates are distributed to relevant people.

Handling other Errors

When users report errors, try to figure the source of error (user, application, system) and take appropriate action. Add diag-bioinf in CC to any relevant emails.

If a task fails, try to inspect the logs first, and then the work directory of this task (located in anno-<instance>/work/<taskid>). If you don't have the task ID, check the timestamp of the directory in work. Some useful files here are:

  • STATUS (check latest step)
  • target.source (check environment variables. Useful to determine what was requested, and what was sent to the ella target)
  • <annotation step>/output.log (output from each annotation step, possibly containing error messages)

References