Skip to content

HTS Bioinf - Running annoservice

Scope

The procedure describes the process for running anno as an API service in production:

  • starting and stopping the service
  • monitor the infrastructure
  • handle errors

Responsibility

Responsible person: Production bioinformatician.

Definitions

ELLA: software for interpretataions of genetic variants.

Supervisor: software module to monitor processes.

Supervisor UI: the webpage to access Supervisor functionality

TSD: Tjenester for Sensitive Data (infrastructur at USIT, University of Oslo)


What is anno and anno-targets?

The specifications and background on anno and anno-targets is provided here.

We currently run anno with anno-targets as both a service on TSD, and an integrated sub-workflow in the pipeline and the backup pipeline.

Location of the service

The service is running on TSD at the University of Oslo.

The url of the production API is http://p22-anno-01:9000 (staging, test and validation instances). Each instance is connected to the corresponding ELLA instance.

The base of anno on TSD can be found at /tsd/p22/data/durable/production/anno.

Supervisor (process manager)

anno is setup to be managed by Supervisor. The configuration is located in ops/supervisor.conf.

To start the Supervisor process running anno:

  • {serviceuser login} p22-anno-01
  • run

    cd /tsd/p22/data/durable/production/anno/ops/
    ./run-supervisor.sh
    

This will start the Supervisor process by service user.

Tail the log at ops/prod.log and check that it has started correctly and that requests are handled.

To check running anno processes:

```bash
# check for running supervisor processes
pgrep supervisord | wc -l
# 5
```

```bash
# show info for all supervisord processes
ps uww $(pgrep supervisord)
# USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
# 1200       11084  0.0  0.0  28108 22880 ?        S    10:23   0:04 /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisord.conf
# tsnowlan   11106  0.0  0.0  68244 23148 ?        S    10:23   0:07 /dist/ella-python/bin/python3.7 /dist/ella-python/bin/supervisord -c /ella/ops/dev/supervisor.cfg
# 1200       11657  0.0  0.0  33504 23104 ?        S    10:23   0:05 /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisord.conf
# 1200       11808  0.0  0.0  33392 23268 ?        S    10:23   0:05 /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisord.conf
# tsnowlan   14791  0.0  0.0  68244 23032 ?        S    10:23   0:06 /dist/ella-python/bin/python3.7 /dist/ella-python/bin/supervisord -c /ella/ops/dev/supervisor.cfg
```

To view supervisor UI and check process status:

./supervisor-ctl.sh

Go to the Supervisor page p22-anno-01:9000/ and click on the respective instance:

figure-anno-1

It will show you the log with the current singularity image version pinned at the top and tail of the same log ( ops/prod.log for instance prod):

figure-anno-2

To stop supervisor running anno:

  • stop all the processes via supervisor UI
  • kill all processes on p22-anno-01:9000 run by serviceuser
killall -u p22-serviceuser01

Logs

Log files are located in ops/logs/.

Handling TSD Errors

When infrastructure errors are detected, send email to tsd-drift@usit.uio.no. Use the email link on http://www.uio.no/english/services/it/research/sensitive-data/contact/index.html and fill in the required fields. Put diag-bioinf in CC so updates are distributed to relevant people.

Handling other Errors

When users report errors try to figure the source of error (user, application, system) and take appropriate action. Add diag-bioinf in CC to any relevant emails.

If a task fails, try to inspect the logs first, and then the work directory of this task (located in anno-<instance>/work/<taskid> ). If you don't have the task ID, check timestamp of the directory in work. Some useful files here are:

  • STATUS (check latest step)
  • target.source (check environment variables. Useful to determine what was requested, and what was sent to the ella - target)
  • <annotation step>/output.log (output from each annotation step, possibly containing error message)

References