Skip to content

HTS Bioinf - Release and deployment of tsd-import

Table of contents

Scope

This procedure explains how to properly make new releases of the tsd-import package, and how to deploy the software in the production environment.

Responsibility

See the role descriptions in the procedure HTS Bioinf - Bioinformatics Group roles Release: Release coordinator automation Deployment: On duty production duty responsible

Platforms

The software is located on different platforms, see the table below:

Host Directory Location Servers
TSD production /ess/p22/data/durable/production p22-cluster-sync   p22-submit-dev
NSC transfer /boston/diag/transfer sleipnir and beta

1. Tag the repositories for the new release

Merge all changes into master and tag the new release accordingly (from HEAD of master). The version should follow semantic versioning according to the format vX.X.X. The tag must end with -rel:

git tag -a vX.X.X-rel -m "Tag release vX.X.X"

Immediately push the tag to origin:

git push origin vX.X.X-rel

2. Export the repository

Open the pipelines view of Gitlab's CI and find a pipeline execution that was run on the relevant release tag.

Start the job release-artifact.

Once the job is finished, download the tar archive from the web page.

Transfer to TSD and NSC.

3. Deploy

3.1. Deploy on NSC

As lims-exporter-api is run on beta and nsc-exporter on sleipnir, to simplify the deployment process, we deploy on NSC only when either of those has changed.

Note! The services running on beta and sleipnir must be stopped and started on their respective server. The removal of the old package and the extraction of the package to be deployed are done on beta.

Copy and archive the deployable package

Copy the exported tarball into the {transfer}/sw/archive directory on beta. This is in order to ensure that there always is an archived version of the software before deployment.

Stop nsc-exporter and lims-exporter-api

Refer to section "Finish production" for instructions on how to stop nsc-exporter and lims-exporter-api.

Extract the package

Log into the server: ssh beta

Remove the old version:

rm -rf {transfer}/sw/tsd-import

Extract the package and verify that the version is as expected:

cd {transfer}/sw/
tar -xvf archive/{tarball}
cat tsd-import/version

Start nsc-exporter and lims-exporter-api

Refer to section "Start production" for instructions on how to start nsc-exporter and lims-exporter-api as service user.

3.2. Deploy on TSD

Since the filelock-exporter is only running on TSD, to simplify the deployment process, we go through this step only when filelock-exporter has changed.

Copy and archive the deployable packages

Copy the exported tarball into the {production}/sw/archive directories on TSD.

Stop filelock-exporter-api

Refer to section "Finish production" for instructions on how to stop filelock-exporter-api.

Extract the package

Remove the current module:

rm -rf {production}/sw/tsd-import

Extract the package and confirm that the version is correct:

On cluster:

cd {production}/sw
./slurm-wrapper-untar-tsd-import.sh {tarball}
# Wait until the job has finished on the cluster.
cat tsd-import/version

On durable:

cd {production}/sw
tar xvf {tarball}
cat tsd-import/version

Start filelock-exporter-api

Refer to section "Start production" for instructions on how to start filelock-exporter-api as service user.

Background

The tsd-import package consists of three different programs, two of which run on the NSC side while the other runs on the TSD side. Its sole purpose is to package samples from NSC and upload them safely into the production area on TSD.

lims-exporter-api

The lims-exporter-api script monitors the lims exporter queue in Clarity for samples and analyses ready to be exported. It creates sample and analysis packages in the format expected in our automation system.

nsc-exporter

This script assists the user with uploading new samples/analyses and backup pipeline results to the S3 API endpoint on TSD in an automatic fashion.

filelock-exporter-api

This filelock-exporter-api script watches the S3 API endpoint on TSD for incoming samples and backup pipeline results that are not already copied to durable. When new samples or new backup pipeline results are detected, they are copied to the production area or durable. There are checks for data integration. If the checks pass, the samples/analyses/ella-incoming are marked with a READY file for pickup by the automation system. If nsc-exporter was run on the NSC side, filelock-exporter-api must be run on TSD side.