Skip to content

NIPT repository

This repository contains in-house scripts for the NIPT pipeline. Scripts are divided into following modules: * adhoc_batching
* lot_instrument_tracing * duplicate_handling * fcs_reports * swl_integration * genologics * genologics_sql

Each module(except for the helper modules) has an executor and is run continuously through cron jobs(./exe/cron.job)

adhoc_batching

Officially, two options for VeriSeq NIPT batching are provided: 1. Using Clarity
Manually selected the samples to be run in Clarity and start the workflow and complete step 1. Then the sample sheet for VeriSeq wil be generated automatically. 2. Do ad hoc batching (without Clarity)
Directly load a rack of "random" samples to robot, then either use the default sample sheet(with default metadata e.g. singleton or twin, basic analyis or genomewide analyis etc.) or edit the sample sheet manually in VeriSeq Workflow Manager.

We however want to be able to load a rack of "random" samples as ad hoc batching does, but still want Clarity to generate the sample sheet automatically(the metadata is already in Clarity). The adhoc_batching module makes this possible by automatically pick up the "random" samples loaded on robot in Clarity, start the workflow and complete step 1 to let Clarity generate the sample sheet automatically.

lot_instrument_tracing

We want to automatically register the reagents and kits used for lab prep. This module parse the reagent report and register the info in Clarity.

duplicate_handling

VeriSeq NIPT Integraton for Clarity does not allow the same sample registered in Clarity multiple times. But lab does not want to screen the duplicated samples before importing them into Clarity. For samples that failed the initial analysis, lab wants to include them in a new project(for better tracing). So we need to handle duplications inside Clarity to make sure the VeriSeq NIPT Integraton does not crash.

fcs_report

We want to also register the sequencing reagent kits as well as instrments, e.g. NextSeq, PCR machine etc. used for each nipt run. To make this happen, one additional step (step 2) has been added to the official nipt workflow supplied by Illumina. This addition step must be completed manually first before samples can reach the step 3 queue of the nipt workflow. VeriSeq NIPT Integraton service monitors new batch initiation reports. If a batch initiation report is found, VeriSeq NIPT Integraton service will check if the samples are already in the queue of step 3, if samples do not reach step 3 queue within 5 tries, the workflow will be aborted. This causes problems for our workflow because lab can not complete step 2 in time. This module solves this problem by feeding the batch initiation report to VeriSeq NIPT Integraton only after lab has completed step 2.

swl_integration

This module parses the original nipt report produced by VeriSeq, does interpretation following pre-defined rules and generates an interpreted nipt report. The interpreted nipt report will be the final product which will be copied over to OUS network. Once the interpreted nipt report is on OUS network, SwissLab nipt automation will parse it and data into SwissLab and trigger automatic answering out if no findings (otherwise leave it to lab doctors to process manually).

genologics

Helper module for using Clarity API.

genologics_sql

Helper module for accessing Clarity database.

Common Problems and Solutions

🔴 Sample sheet is not generated

How to find out:

  • Lab will inform bioinformatician when sample sheet is not generated automatically.

🟢 Steps to solve:

  1. Check the log of "adhoc_batching" module. The logs are in /opt/gls/clarity/customextensions/nipt-clarity/logs/adhoc_batching/ folder on lims server. Normally the error message in the log clearly describes the problem. The most common error message is: 98765432101 not in "Generate Batch & Sample Sheet" queue. If there are no error messages in the log, go to step 3 below.
  2. Confirm in Clariy the error sample is missing in the "Generate Batch & Sample Sheet" queue of "VeriSeq NIPT v1.0_EHD" protocol. Inform lab that the sample is not in the queue. Lab will then check the reason. Two common reasons:
    1. The sample is not in Clariy.
    2. Lab should import the missing sample to Clarity and assign it NIPT workflow. After this, the sample sheet will be generated.
    3. The sample is in Clarity, but the barcode scanned by the Hamilton robot doesn't match the sample name in Clarity. Since that once the bardcode is scanned, it is not possible to change the barcode. So the sample name in Clarity needs to be changed.
    4. Lab will give you the sample name to be changed in Clarity (e.g. 10001) and the barcode (e.g. 20002).
    5. Download the lims sheet in Clarity.
    6. In the downloaded lims sheet, find the sample name to be changed and change it to the barcode (e.g. change 10001 to 20002).
    7. Upload the modified lims sheet to Clarity.
    8. Wait around 5 minutes and confirm that the sample sheet is generated in /mnt/nipt-lab/SampleSheets/.
    9. Open the sample sheet, change the sample name to be changed to the barcode (e.g. change 10001 to 20002).
    10. Inform Lab that the sample sheet is ready to use.
    11. Wait until the sequencing and nipt analysis are finished (This takes around 30 hours).
    12. Once the nipt report(s) have been generated(/mnt/veriseq-output//xxxx_nipt_report_xxx.tab):
      • Find the interpreted nipt report in /mnt/nipt-lab/swl-nipt-report/xxxx_nipt_report_xxx.tsv file, change the sample name to be changed to the barcode (e.g. change 10001 to 20002). Normally, Lab will do this themselves.
      • Append the absolute path of the nipt report(s)(2 nipt report if the project has 96 samples) containing the modified sample to /opt/gls/clarity/customextensions/nipt-clarity/processed_fcs_reports.txt file.
      • Copy aforementioned nipt report(s) to /opt/gls/clarity/customextensions/nipt-clarity/nipt-fcs/Diag-NIPTXX-YYYY-MM-DD/ folder. Diag-NIPTXX-YYYY-MM-DD is the project of the corrected sample.
      • If there are sample_invalidation_report.tab files, do the same, i.e. append file absolute path to /opt/gls/clarity/customextensions/nipt-clarity/processed_fcs_reports.txt file; copy the file to /opt/gls/clarity/customextensions/nipt-clarity/nipt-fcs/Diag-NIPTXX-YYYY-MM-DD/ folder.
      • Find the ongoing "AUTOMATED - Data Analysis(VeriSeq NIPT v1.0_EHD)" step of the project, manually add the "qc_flag", "ff", "flowcell", "Comments"(only if this sample failed QC; use "qc_reason" value) with value in the aforementioned nipt report.
      • Click "NEXT STEP"
      • Click "FINISH STEP"
  3. If the aforementioned scenarios do not apply, check if the in-house cron jobs have stopped or the VeriSeq NIPT Integraton service has stopped(described below).

🔴 The in-house automation cron jobs stopped

How to find out:

  • There is a new batch initiation report in /mnt/veriseq-server-output/<project>/ProcessLogs folder
  • In the log of "adhoc_batching" module, the new batch initiation report is not captured (the log stopped updating). Then the cron jobs may have stopped.

🟢 Steps to solve:

  1. ssh to ous-lims.sequencing.uio.no
  2. Run sudo -u glsai -s
  3. Run crontab /opt/gls/clarity/customextensions/nipt-clarity/nipt/exe/cron.job to restart all in-house cron jobs.

🔴 The VeriSeq NIPT Integraton service stopped

How to find out:

  • A new batch initiation report or a nipt report has been copied over to /opt/gls/clarity/customextensions/nipt-clarity/fcs_report folder, but the 3rd step of the "VeriSeq NIPT v1.0_EHD" protocol is not started or completing.
  • Run sudo -u glsai -s
  • Run cd /opt/gls/clarity/extensions/veriseq-nipt/FileCaptureService
  • The VeriSeqNIPTLauncher.log file is not updating.

🟢 Steps to solve:

  1. Go back to your own user(exit sudo glsai user).
  2. Run sudo /etc/init.d/veriseq-nipt_service stop
  3. Run sudo /etc/init.d/veriseq-nipt_service start to restart the service.

🔴 Remove NTC samples from workflow

When a project is stopped, all samples can be removed from workflow as normal, except for the NTC samples. NTC samples must finish the normal workflow as follow: 1. Select all the NTC samples to be removed, add them to Icebucket and start the step. 2. Upload the dummy_batch_initiation_report.tab in this repo when an bach initiation report is required in Clarity. 3. Keep Continue until Finish step.