Skip to content

HTS Bioinf - Deployment of vcpipe for production

Scope

This document describes how to update the software modules part of the variant calling pipeline on TSD and NSC. This assumes that a software package has been created and is available for deployment. The procedure for developing, testing and documenting pipeline software is described in HTS Bioinf - Process for updates to production pipelines.

Responsibility

See the role descriptions in the procedure HTS Bioinf - Bioinformatics Group roles Release: Release coordinator variant calling Deployment: On duty production duty responsible

Note that "Endringskontroll for bioinformatiske endringer" should be filled in and approved by a manager prior to deploying any major or minor versions (does not apply to hotfixes).

Platforms

All production paths (starting with {production}/) are given relative to the base path for the system in question:

System Directory Location
TSD production /ess/p22/data/durable
NSC production /boston/diag/

As the system can run on different production platforms, the setup can vary slightly.

The relevant servers are beta, diag-executor and diag-webui on NSC and p22-submit, p22-submit2 and p22-submit-dev on TSD.

1. Copy and archive the deployable package to the production platforms

Transfer the package, its checksum and any Singularity images to TSD using the data transfer API. The files will end up in /ess/p22/data/durable/file-import.

Make copies of the transferred files in /ess/p22/data/durable/file-export in order to make them accessible from NSC later.

For TSD

  • Log into p22-submit (or p22-submit-dev if using the backup server).
  • Move the package and its checksum to the {production}/sw/archive directory.
  • Move the Singularity images to their respective directories in /ess/p22/cluster/singularity.

For NSC

  • Log into sleipnir.
  • Use tacl to download the files from TSD and place them in /boston/diag/transfer.
  • Back on beta, move the package and its checksum to {production}/sw/archive.
  • Always on beta, move the Singularity images to /boston/diag/singularity.

2. Stop production processes

Refer to section "Finish production" for instructions on how to stop executor and webui on TSD and NSC.

3. Extract the package

{production}/sw/deploy.sh <path to package> production vcpipe

or, for Python modules:

{production}/sw/deploy.sh <path to package> production site-packages

Verify that the version of the deployed package matches the one intended:

{production}/sw/show-versions.sh

Note! The deployment must always be done on both NSC and TSD.

4. Start production processes

Refer to section "Start production" for instructions on how to start executor and webui on TSD and NSC as service user.