Skip to content

HTS Bioinf - Running annoservice


The procedure describes the process for running anno as an API service in production:

  • starting and stopping the service
  • monitor the infrastructure
  • handle errors


Responsible person: Production bioinformatician.


ELLA: software for interpretataions of genetic variants.

Supervisor: software module to monitor processes.

Supervisor UI: the webpage to access Supervisor functionality

TSD: Tjenester for Sensitive Data (infrastructur at USIT, University of Oslo)

What is anno and anno-targets?

The specifications and background on anno and anno-targets is provided here.

We currently run anno with anno-targets as both a service on TSD, and an integrated sub-workflow in the pipeline and the backup pipeline.

Location of the service

The service is running on TSD at the University of Oslo.

The url of the production API is http://p22-anno-01:9000 (staging, test and validation instances). Each instance is connected to the corresponding ELLA instance.

The base of anno on TSD can be found at /tsd/p22/data/durable/production/anno.

Supervisor (process manager)

anno is setup to be managed by Supervisor. The configuration is located in ops/supervisor.conf.

To start the Supervisor process running anno:

  • {serviceuser login} p22-anno-01
  • run

    cd /tsd/p22/data/durable/production/anno/ops/

This will start the Supervisor process by service user.

Tail the log at ops/prod.log and check that it has started correctly and that requests are handled.

To check running anno processes:

# check for running supervisor processes
pgrep supervisord | wc -l
# 5

# show info for all supervisord processes
ps uww $(pgrep supervisord)
# 1200       11084  0.0  0.0  28108 22880 ?        S    10:23   0:04 /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisord.conf
# tsnowlan   11106  0.0  0.0  68244 23148 ?        S    10:23   0:07 /dist/ella-python/bin/python3.7 /dist/ella-python/bin/supervisord -c /ella/ops/dev/supervisor.cfg
# 1200       11657  0.0  0.0  33504 23104 ?        S    10:23   0:05 /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisord.conf
# 1200       11808  0.0  0.0  33392 23268 ?        S    10:23   0:05 /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisord.conf
# tsnowlan   14791  0.0  0.0  68244 23032 ?        S    10:23   0:06 /dist/ella-python/bin/python3.7 /dist/ella-python/bin/supervisord -c /ella/ops/dev/supervisor.cfg

To view supervisor UI and check process status:


Go to the Supervisor page p22-anno-01:9000/ and click on the respective instance:


It will show you the log with the current singularity image version pinned at the top and tail of the same log ( ops/prod.log for instance prod):


To stop supervisor running anno:

  • stop all the processes via supervisor UI
  • kill all processes on p22-anno-01:9000 run by serviceuser
killall -u p22-serviceuser01


Log files are located in ops/logs/.

Handling TSD Errors

When infrastructure errors are detected, send email to Use the email link on and fill in the required fields. Put diag-bioinf in CC so updates are distributed to relevant people.

Handling other Errors

When users report errors try to figure the source of error (user, application, system) and take appropriate action. Add diag-bioinf in CC to any relevant emails.

If a task fails, try to inspect the logs first, and then the work directory of this task (located in anno-<instance>/work/<taskid> ). If you don't have the task ID, check timestamp of the directory in work. Some useful files here are:

  • STATUS (check latest step)
  • target.source (check environment variables. Useful to determine what was requested, and what was sent to the ella - target)
  • <annotation step>/output.log (output from each annotation step, possibly containing error message)