HTS Bioinf - Release and deployment of tsd-import
Table of contents
- HTS Bioinf - Release and deployment of tsd-import
- Table of contents
- Scope
- Responsibility
- Platforms
- 1. Tag the repositories for the new release
- 2. Export the repository
- 3. Deploy
- Background
Scope
This procedure explains how to properly make new releases of the tsd-import package, and how to deploy the software in the production environment.
Responsibility
See the role descriptions in the procedure HTS Bioinf - Bioinformatics Group roles Release: Release coordinator automation Deployment: On duty production duty responsible
Platforms
The software is located on different platforms, see the table below:
Host | Directory | Location | Servers |
---|---|---|---|
TSD | production | /ess/p22/data/durable/production |
p22-hpc-02 p22-hpc-03 |
NSC | transfer | /boston/diag/transfer |
sleipnir and gdx-login |
1. Tag the repositories for the new release
Merge all changes into master
and tag the new release accordingly (from HEAD
of master
). The version should follow semantic versioning according to the format vX.X.X. The tag must end with -rel
:
Immediately push the tag to origin:
2. Export the repository
Open the pipelines view of Gitlab's CI and find a pipeline execution that was run on the relevant release tag.
Start the job release-artifact
.
Once the job is finished, download the tar
archive from the web page.
Transfer to TSD and NSC.
3. Deploy
3.1. Deploy on NSC
As lims-exporter-api
is run on gdx-login and nsc-exporter
on sleipnir, to simplify the deployment process, we deploy on NSC only when either of those has changed.
Note! The services running on gdx-login and sleipnir must be stopped and started on their respective server. The removal of the old package and the extraction of the package to be deployed are done on gdx-login.
Copy and archive the deployable package
Copy the exported tarball into the {transfer}/sw/archive
directory on gdx-login. This is in order to ensure that there always is an archived version of the software before deployment.
Stop nsc-exporter
and lims-exporter-api
Refer to section "Finish production" for instructions on how to stop nsc-exporter
and lims-exporter-api
.
Extract the package
Log into the server: ssh gdx-login
Remove the old version:
Extract the package and verify that the version is as expected:
Start nsc-exporter
and lims-exporter-api
Refer to section "Start production" for instructions on how to start nsc-exporter
and lims-exporter-api
as service user.
3.2. Deploy on TSD
Since the filelock-exporter
is only running on TSD, to simplify the deployment process, we go through this step only when filelock-exporter
has changed.
Copy and archive the deployable packages
Copy the exported tarball into the {production}/sw/archive
directories on TSD.
Stop filelock-exporter-api
Refer to section "Finish production" for instructions on how to stop filelock-exporter-api
.
Extract the package
Remove the current module:
Extract the package and confirm that the version is correct:
Start filelock-exporter-api
Refer to section "Start production" for instructions on how to start filelock-exporter-api
as service user.
Background
The tsd-import
package consists of three different programs, two of which run on the NSC side while the other runs on the TSD side.
Its sole purpose is to package samples from NSC and upload them safely into the production area on TSD.
lims-exporter-api
The lims-exporter-api
script monitors the lims exporter queue in Clarity for samples and analyses ready to be exported. It creates sample and analysis packages in the format expected in our automation system.
nsc-exporter
This script assists the user with uploading new samples/analyses and backup pipeline results to the S3 API endpoint on TSD in an automatic fashion.
filelock-exporter-api
This filelock-exporter-api
script watches the S3 API endpoint on TSD for incoming samples and backup pipeline results that are not already copied to durable
. When new samples or new backup pipeline results are detected, they are copied to the production area or durable
. There are checks for data integration. If the checks pass, the samples/analyses/ella-incoming
are marked with a READY
file for pickup by the automation system. If nsc-exporter
was run on the NSC side, filelock-exporter-api
must be run on TSD side.