HTS Bioinf - Export and deploy gene panels

Scope

Herein is described the procedure for exporting gene panel files from the gene panel database, eventually comparing them to extant production instances.

Responsibility

A bioinformatician from GDx is responsible for performing these tasks.

Definitions

The gene panel builder is a service for designing and building gene panels running on https://gpb.allel.es.
The gene panels files builds are themselves under Git version control. The corresponding repository is called gene panel store.
The build date announced by the bioinformaticians marks the time of the latest accepted changes and will be assumed to be stored in the variable DATE=$(date '+%Y-%m-%d-T%H%M%S').

Review changes

SSH as user gpbuilder into gpb.allel.es (ask a system administrator for the SSH key) and issue the following commands:

Make sure that the gene panels repository is up to date with the remote:

# fetch information about any recent genepanel-store updates
git -C genepanel-store remote update origin --prune

# get the latest genepanel-store release tag
GPSTORE_TAG=$( git -C genepanel-store tag | tail -n 1 )

# make sure that the gene panels repository is up to date with the remote
git -C genepanel-store checkout ${GPSTORE_TAG}

Compare the active panels to the ones currently in production and write a diff report:

docker compose --project-directory config exec -it gpbuilder \
    gpb diff /current-panels

Copy the diff report to the designated area on the hospital network and request a review from ELL.

NOTE: the gene panel builder service is designed to streamline the entire procedure but multiple iterations of this process might occasionally still be necessary. In such cases, the latest build DATE will be considered valid.

Release

Once the diff report is approved and the "Endringskontroll" is finalized, the gene panels repository must be updated and the updated files transferred to the production infrastructures (NSC and TSD).

The release procedure at a glance

Export the gene panels;
Update the 'genepanel-store' repository with the exported gene panel files;
Tag the repository state and push the tag to remote;
Transfer the archive to NSC and TSD;
Deploy the panels on NSC and TSD;
Import the panels in ELLA;
Notify everyone that the gene panels have been updated.

The release procedure in detail

SSH as user gpbuilder into gpb.allel.es and issue the following commands:

Make sure that the gene panels repository is up to date with the remote
```
git -C genepanel-store checkout '<latest-release-tag>'
```

Export the gene panel files:

docker compose --project-directory config exec -it gpbuilder \
    gpb export panels -o /exported-panels/${DATE}

If you are running a periodic release, export whole exome panels for each unit in (EGG, EHG, EKG).

NOTE: the version should receive a major bump for every update (e.g. 1.0.0 -> 2.0.0), so fetch the latest release of the gene panel store for the latest version.

for unit in EGG EHG EKG ; do
    docker compose --project-directory config exec -it gpbuilder \
        gpb export genes --coding-only -o /exported-panels/${DATE} Eksom${unit} <version>
    docker compose --project-directory config exec -it gpbuilder \
        gpb export genes -o /exported-panels/${DATE} EksomNMNR${unit} <version>
done

Copy the new gene panel files into the repository:

rsync -avz exported-panels/${DATE}/ genepanel-store/panels/

Change into the gene panel store repository, tag a new release and push it to remote:

cd genepanel-store
# make sure that all existing panels with an incoming version update are removed _before_
# issuing the following instructions.
git add -A
git commit -m "Build ${DATE}"
git tag -a -m "Tag Build ${DATE}" ${DATE}-rel
git push; git push origin ${DATE}-rel
# Once the pipeline for the tag is complete (<5 minutes; https://gitlab.com/alleles/genepanel-store/-/pipelines), run this:
wget -P ../production-panels/releases https://genepanel-store.fra1.digitaloceanspaces.com/${DATE}-rel/genepanel-store-${DATE}-rel.tar.gz

Upload the archive to TSD and to NSC's transfer area.
Stop LIMS and filelock exporters (see how here) in a production gap window.

Extract the gene panels files and sync them with the local ones on TSD and NSC:

mkdir /tmp/${DATE}-rel
tar xvzf {transfer}/genepanel-store-${DATE}-rel.tar.gz --directory /tmp/${DATE}-rel
rsync -avz /tmp/${DATE}-rel/ {production,staging}/reference/public/genepanel-store/

NOTE: the production locations table shows where transfer, staging and production place-holders actually point to.

Import the (new) gene panels in ELLA (see ELLA-production/Adding new gene panels).
Restart LIMS and filelock exporters (see how here).
Fill in the update date in "Endringskontroll" and notify all units about the gene panels update using the dedicated Microsoft Teams channel.