Automation system

The automation system (named executor) monitors the folders containing sample and analysis files registering new samples/analysis in the database as they appear. It launches new analyses when they're ready, storing their status, run time and logs in the database. There are different types of analyses, each with a set of requirements that must be met before the analysis is started.

You can run it by using a command similar to:

SETTINGS=/path/to/settings.json DB_URL="postgres://user:pass@localhost/dbname" exe/executor

The value of SETTINGS and DB_URL depends on where you want to run it, either - locally on your development machine - on a test server - in production (on TSD) - in staging (on TSD)

There are wrapper scripts for each environment to start it correctly.

In production the SETTINGS file is 'config/settings-tsd_production.json'.

The env variable 'SYSTEM' is used to pick out the Nextflow profile relevant for current environment. See the various pipeline start scripts exe/{base,anno,trio}pipe

New analyses and samples

The automation system will monitor the paths configured in the SETTINGS file for new samples and analyses. Each new sample/analysis will have a separate folder. More specifically, it will traverse the two paths and look for files ending in .sample or .analysis respectively. An empty READY file must be present in the same directory as the .sample/.analysis file in order for it to be imported.

For each type of analysis (e.g. basepipe) there is a triplet of scripts in 'exe/pipeline/' that are important: - pre (e.g. basepipe/basepipe-dependency) Checks that all dependencies are met - main (basepipe/basepipe) Invokes Nextflow - post (basepipe/basepipe-post). Could copy result files generated by the main script.

Each type of analysis has it's own Nextflow based recipe with extension '.nf'. The nextflow program will be invoked with the .nf file as argument. In addition other options/parameters mentioned in the .nf file is supplied to the nextflow program. One important option is '-profile' which tells Nextflow which properties to use. See https://www.nextflow.io/docs/latest/config.html