Skip to content

HTS Bioinf - Release and deployment of variantcalling system

Scope

This document describes how to update the softwares under the variantcalling system on TSD and NSC. This assumes that a software package has been created and is available for deployment. The procedure for developing, testing and documenting pipeline software is described in HTS Bioinf - Process for updates to production pipelines.

Responsibility

See the role descriptions in the procedure HTS Bioinf - Bioinformatics Group roles Release: Release coordinator variant calling Deployment: On duty production duty responsible

Note that "Endringskontroll for bioinformatiske endringer" should be filled in and approved by a manager prior to deploying any major or minor versions (does not apply to hotfixes).

Platforms

All production paths (starting with {production}/) are given relative to the base path for the system in question:

System Directory Location
TSD production /ess/p22/data/durable
NSC production /boston/diag/

As the system can run on different production platforms, the setup can vary slightly.

The relevant servers are gdx-login, gdx-executor and gdx-webui on NSC and p22-hpc-01, p22-hpc-02 and p22-hpc-03 on TSD.

1. Copy and archive the deployable package to the production platforms

Transfer the package, its checksum and any Singularity images to TSD using the data transfer API. The files will end up in /ess/p22/data/durable/file-import.

Make copies of the transferred files in /ess/p22/data/durable/file-export in order to make them accessible from NSC later.

For TSD

  • Log into p22-hpc-01 (or p22-hpc-03 if using the backup server).
  • Move the package and its checksum to the {production}/sw/archive directory.
  • Move the Singularity images to their respective directories in /ess/p22/data/durable/production/sw/variantcalling.

For NSC

  • Log into sleipnir.
  • Use tacl to download the files from TSD and place them in /boston/diag/transfer.
  • Back on gdx-login, move the package and its checksum to {production}/sw/archive.
  • Always on gdx-login, move the Singularity images to /boston/diag/singularity.

2. Stop production processes

Refer to section "Finish production" for instructions on how to stop executor and webui on TSD and NSC.

3. Extract the package in production and staging

{production}/sw/deploy.sh <path to package>  ENVIRONMENT_NAME SYSTEM_NAME
Available ENVIRONMENT_NAME: production, staging Available SYSTEM_NAME*: vcpipe, vcpipe-utilities, site-packages

*: site-packages are python modules required in the vcpipe

Verify that the version of the deployed package matches the one intended:

{production}/sw/show-versions.sh

Note

The deployment must always be done on both NSC and TSD, production and staging

4. Start production processes

Refer to section "Start production" for instructions on how to start executor and webui on TSD and NSC as service user.