Skip to content

HTS Bioinf - Deployment of vcpipe for production

Scope

This document describes how to update the software modules part of the variant calling pipeline on TSD and NSC. This assumes that a software package has been created and is available for deployment. The development, testing and documentation are described in HTS Bioinf - Process for updates to production pipelines.

Responsibility

The bioinformatician responsible for pipeline release.

Note that "Endringskontroll for bioinformatiske endringer" should be filled in and approved by a manager prior to deploying any major or minor versions (does not apply to hotfixes).

Platforms

All production paths (starting with production/) are given relative to the base path for the system in question.

System Directory Location
TSD production /ess/p22/data/durable/production
NSC production /boston/diag/

As the system can run on different production platforms, the setup can vary slightly.

On NSC the relevant servers are beta, diag-executor and diag-webui and on TSD the servers are p22-submit, p22-submit2 or p22-submit-dev.

1. Copy and archive the deployable package to the production platforms

Transfer the artifact to TSD using the data transfer API. The files will end up in /ess/p22/data/durable/file-import.

Make a copy of the transferred files into in /ess/p22/data/durable/file-export (in order to get files to NSC).

For TSD

  • Log in to p22-submit (or p22-submit-dev if using the backup server):

    ssh p22-submit

  • Copy the package to the {production}/sw/archive directory.

  • For singularity, copy the image to /ess/p22/cluster/singularity.

For NSC

  • Log in to sleipnir.
  • Use tacl to transfer files from file-export on TSD to /boston/diag/transfer.
  • On beta, copy the files to {production}/sw/archive. - Singularity images go into /boston/diag/singularity.

2. Stop production processes

Refer to Execution and monitoring of pipeline section "Finish production" for instructions on how to stop executor and webui on TSD and NSC.

3. Extract the package

{production}/sw/deploy.sh <path to package> vcpipe

or, for Python modules:

{production}/sw/deploy.sh <path to package> site-packages

Verify that the version of the deployed package matches the one intended

{production}/sw/show-version.sh

Note! The deploy is always run on both NSC/beta and TSD.

4. Start production processes

Refer to Execution and monitoring of pipeline section "Start production" for instructions on how to start executor and webui on TSD and NSC under {serviceuser}.