HTS Bioinf - Running annoservice
Scope
The procedure describes the process for running anno as an API service in production:
- starting and stopping the service
- monitor the infrastructure
- handle errors
Responsibility
Responsible person: Production bioinformatician.
Definitions
ELLA: software for interpretataions of genetic variants.
Supervisor: software module to monitor processes.
Supervisor UI: the webpage to access Supervisor functionality
TSD: Tjenester for Sensitive Data (infrastructur at USIT, University of Oslo)
What is anno and anno-targets?
The specifications and background on anno and anno-targets is provided here.
We currently run anno with anno-targets as both a service on TSD, and an integrated sub-workflow in the pipeline and the backup pipeline.
Location of the service
The service is running on TSD at the University of Oslo.
The url of the production API is http://p22-anno-01:9000 (staging, test and validation instances). Each instance is connected to the corresponding ELLA instance.
The base of anno on TSD can be found at /tsd/p22/data/durable/production/anno.
Supervisor (process manager)
anno is setup to be managed by Supervisor. The configuration is located in ops/supervisor.conf.
To start the Supervisor process running anno:
- {serviceuser login}
p22-anno-01 -
run
This will start the Supervisor process by service user.
Tail the log at ops/prod.log and check that it has started correctly and that requests are handled.
To check running anno processes:
```bash
# check for running supervisor processes
pgrep supervisord | wc -l
# 5
```
```bash
# show info for all supervisord processes
ps uww $(pgrep supervisord)
# USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
# 1200 11084 0.0 0.0 28108 22880 ? S 10:23 0:04 /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisord.conf
# tsnowlan 11106 0.0 0.0 68244 23148 ? S 10:23 0:07 /dist/ella-python/bin/python3.7 /dist/ella-python/bin/supervisord -c /ella/ops/dev/supervisor.cfg
# 1200 11657 0.0 0.0 33504 23104 ? S 10:23 0:05 /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisord.conf
# 1200 11808 0.0 0.0 33392 23268 ? S 10:23 0:05 /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisord.conf
# tsnowlan 14791 0.0 0.0 68244 23032 ? S 10:23 0:06 /dist/ella-python/bin/python3.7 /dist/ella-python/bin/supervisord -c /ella/ops/dev/supervisor.cfg
```
To view supervisor UI and check process status:
Go to the Supervisor page p22-anno-01:9000/ and click on the respective instance:
It will show you the log with the current singularity image version pinned at the top and tail of the same log ( ops/prod.log for instance prod):
To stop supervisor running anno:
- stop all the processes via supervisor UI
- kill all processes on
p22-anno-01:9000run by serviceuser
Logs
Log files are located in ops/logs/.
Handling TSD Errors
When infrastructure errors are detected, send email to tsd-drift@usit.uio.no. Use the email link on
http://www.uio.no/english/services/it/research/sensitive-data/contact/index.html and fill in the required fields. Put diag-bioinf in CC so updates are distributed to relevant people.
Handling other Errors
When users report errors try to figure the source of error (user, application, system) and take appropriate action. Add diag-bioinf in CC to any relevant emails.
If a task fails, try to inspect the logs first, and then the work directory of this task (located in anno-<instance>/work/<taskid> ). If you don't have the task ID, check timestamp of the directory in work. Some useful files here are:
STATUS(check latest step)target.source(check environment variables. Useful to determine what was requested, and what was sent to the ella - target)<annotation step>/output.log(output from each annotation step, possibly containing error message)