picard.illumina

Class IlluminaBasecallsToSam

java.lang.Object

picard.cmdline.CommandLineProgram

picard.illumina.IlluminaBasecallsToSam

public class IlluminaBasecallsToSam
extends CommandLineProgram

IlluminaBasecallsToSam transforms a lane of Illumina data file formats (bcl, locs, clocs, qseqs, etc.) into SAM or BAM file format.

In this application, barcode data is read from Illumina data file groups, each of which is associated with a tile. Each tile may contain data for any number of barcodes, and a single barcode's data may span multiple tiles. Once the barcode data is collected from files, each barcode's data is written to its own SAM/BAM. The barcode data must be written in order; this means that barcode data from each tile is sorted before it is written to file, and that if a barcode's data does span multiple tiles, data collected from each tile must be written in the order of the tiles themselves.

This class employs a number of private subclasses to achieve this goal. The TileReadAggregator controls the flow of operation. It is fed a number of Tiles which it uses to spawn TileReaders. TileReaders are responsible for reading Illumina data for their respective tiles from disk, and as they collect that data, it is fed back into the TileReadAggregator. When a TileReader completes a tile, it notifies the TileReadAggregator, which reviews what was read and conditionally queues its writing to disk, baring in mind the requirements of write-order described in the previous paragraph. As writes complete, the TileReadAggregator re-evalutes the state of reads/writes and may queue more writes. When all barcodes for all tiles have been written, the TileReadAggregator shuts down.

The TileReadAggregator controls task execution using a specialized ThreadPoolExecutor. It accepts special Runnables of type PriorityRunnable which allow a priority to be assigned to the runnable. When the ThreadPoolExecutor is assigning threads, it gives priority to those PriorityRunnables with higher priority values. In this application, TileReaders are assigned lowest priority, and write tasks are assigned high priority. It is designed in this fashion to minimize the amount of time data must remain in memory (write the data as soon as possible, then discard it from memory) while maximizing CPU usage.

Author:

jburke@broadinstitute.org, mccowan@broadinstitute.org

Field Summary

Fields

Modifier and Type	Field and Description
java.util.List<IlluminaUtil.IlluminaAdapterPair>	ADAPTERS_TO_CHECK
boolean	APPLY_EAMSS_FILTER
java.io.File	BARCODE_PARAMS
java.io.File	BARCODES_DIR
java.io.File	BASECALLS_DIR
java.lang.Integer	FIRST_TILE
java.lang.Boolean	FORCE_GC
boolean	IGNORE_UNEXPECTED_BARCODES
boolean	INCLUDE_NON_PF_READS
java.lang.Integer	LANE
java.lang.String	LIBRARY_NAME
java.io.File	LIBRARY_PARAMS
int	MAX_READS_IN_RAM_PER_TILE
int	MINIMUM_QUALITY
java.lang.String	MOLECULAR_INDEX_BASE_QUALITY_TAG
java.lang.String	MOLECULAR_INDEX_TAG
java.lang.Integer	NUM_PROCESSORS
java.io.File	OUTPUT
java.lang.String	PLATFORM
java.lang.String	READ_GROUP_ID
java.lang.String	READ_STRUCTURE
java.lang.String	RUN_BARCODE
java.util.Date	RUN_START_DATE
java.lang.String	SAMPLE_ALIAS
java.lang.String	SEQUENCING_CENTER
java.util.List<java.lang.String>	TAG_PER_MOLECULAR_INDEX
java.lang.Integer	TILE_LIMIT
static java.lang.String	USAGE

Fields inherited from class picard.cmdline.CommandLineProgram

COMPRESSION_LEVEL, CREATE_INDEX, CREATE_MD5_FILE, GA4GH_CLIENT_SECRETS, MAX_RECORDS_IN_RAM, QUIET, REFERENCE_SEQUENCE, TMP_DIR, VALIDATION_STRINGENCY, VERBOSITY

Constructor Summary

Constructors

Constructor and Description
IlluminaBasecallsToSam()

Method Summary

Modifier and Type	Method and Description
protected java.lang.String[]	customCommandLineValidation() Put any custom command-line validation in an override of this method.
protected int	doWork() Do the work after command line has been parsed.
static void	main(java.lang.String[] args)

Methods inherited from class picard.cmdline.CommandLineProgram

getCommandLine, getCommandLineParser, getDefaultHeaders, getMetricsFile, getNestedOptions, getNestedOptionsForHelp, getStandardUsagePreamble, getVersion, instanceMain, instanceMainWithExit, parseArgs, setDefaultHeaders

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

USAGE

public static final java.lang.String USAGE

See Also:

Constant Field Values

BASECALLS_DIR

@Option(doc="The basecalls directory. ",
        shortName="B")
public java.io.File BASECALLS_DIR

BARCODES_DIR

@Option(doc="The barcodes directory with _barcode.txt files (generated by ExtractIlluminaBarcodes). If not set, use BASECALLS_DIR. ",
        shortName="BCD",
        optional=true)
public java.io.File BARCODES_DIR

LANE

@Option(doc="Lane number. ",
        shortName="L")
public java.lang.Integer LANE

OUTPUT

@Option(doc="Deprecated (use LIBRARY_PARAMS).  The output SAM or BAM file. Format is determined by extension.",
        shortName="O",
        mutex={"BARCODE_PARAMS","LIBRARY_PARAMS"})
public java.io.File OUTPUT

RUN_BARCODE

@Option(doc="The barcode of the run.  Prefixed to read names.")
public java.lang.String RUN_BARCODE

SAMPLE_ALIAS

@Option(doc="Deprecated (use LIBRARY_PARAMS).  The name of the sequenced sample",
        shortName="ALIAS",
        mutex={"BARCODE_PARAMS","LIBRARY_PARAMS"})
public java.lang.String SAMPLE_ALIAS

READ_GROUP_ID

@Option(doc="ID used to link RG header record with RG tag in SAM record.  If these are unique in SAM files that get merged, merge performance is better.  If not specified, READ_GROUP_ID will be set to <first 5 chars of RUN_BARCODE>.<LANE> .",
        shortName="RG",
        optional=true)
public java.lang.String READ_GROUP_ID

LIBRARY_NAME

@Option(doc="Deprecated (use LIBRARY_PARAMS).  The name of the sequenced library",
        shortName="LIB",
        optional=true,
        mutex={"BARCODE_PARAMS","LIBRARY_PARAMS"})
public java.lang.String LIBRARY_NAME

SEQUENCING_CENTER

@Option(doc="The name of the sequencing center that produced the reads.  Used to set the RG.CN tag.",
        optional=true)
public java.lang.String SEQUENCING_CENTER

RUN_START_DATE

@Option(doc="The start date of the run.",
        optional=true)
public java.util.Date RUN_START_DATE

PLATFORM

@Option(doc="The name of the sequencing technology that produced the read.",
        optional=true)
public java.lang.String PLATFORM

READ_STRUCTURE

@Option(doc="A description of the logical structure of clusters in an Illumina Run, i.e. a description of the structure IlluminaBasecallsToSam assumes the  data to be in. It should consist of integer/character pairs describing the number of cycles and the type of those cycles (B for Sample Barcode, M for molecular barcode, T for Template, and S for skip).  E.g. If the input data consists of 80 base clusters and we provide a read structure of \"28T8M8B8S28T\" then the sequence may be split up into four reads:\n* read one with 28 cycles (bases) of template\n* read two with 8 cycles (bases) of molecular barcode (ex. unique molecular barcode)\n* read three with 8 cycles (bases) of sample barcode\n* 8 cycles (bases) skipped.\n* read four with 28 cycles (bases) of template\nThe skipped cycles would NOT be included in an output SAM/BAM file or in read groups therein.",
        shortName="RS")
public java.lang.String READ_STRUCTURE

BARCODE_PARAMS

@Option(doc="Deprecated (use LIBRARY_PARAMS).  Tab-separated file for creating all output BAMs for barcoded run with single IlluminaBasecallsToSam invocation.  Columns are BARCODE, OUTPUT, SAMPLE_ALIAS, and LIBRARY_NAME.  Row with BARCODE=N is used to specify a file for no barcode match",
        mutex={"OUTPUT","SAMPLE_ALIAS","LIBRARY_NAME","LIBRARY_PARAMS"})
public java.io.File BARCODE_PARAMS

LIBRARY_PARAMS

@Option(doc="Tab-separated file for creating all output BAMs for a lane with single IlluminaBasecallsToSam invocation.  The columns are OUTPUT, SAMPLE_ALIAS, and LIBRARY_NAME, BARCODE_1, BARCODE_2 ... BARCODE_X where X = number of barcodes per cluster (optional).  Row with BARCODE_1 set to \'N\' is used to specify a file for no barcode match.  You may also provide any 2 letter RG header attributes (excluding PU, CN, PL, and DT)  as columns in this file and the values for those columns will be inserted into the RG tag for the BAM file created for a given row.",
        mutex={"OUTPUT","SAMPLE_ALIAS","LIBRARY_NAME","BARCODE_PARAMS"})
public java.io.File LIBRARY_PARAMS

ADAPTERS_TO_CHECK

@Option(doc="Which adapters to look for in the read.")
public java.util.List<IlluminaUtil.IlluminaAdapterPair> ADAPTERS_TO_CHECK

NUM_PROCESSORS

@Option(doc="The number of threads to run in parallel. If NUM_PROCESSORS = 0, number of cores is automatically set to the number of cores available on the machine. If NUM_PROCESSORS < 0, then the number of cores used will be the number available on the machine less NUM_PROCESSORS.")
public java.lang.Integer NUM_PROCESSORS

FIRST_TILE

@Option(doc="If set, this is the first tile to be processed (used for debugging).  Note that tiles are not processed in numerical order.",
        optional=true)
public java.lang.Integer FIRST_TILE

TILE_LIMIT

@Option(doc="If set, process no more than this many tiles (used for debugging).",
        optional=true)
public java.lang.Integer TILE_LIMIT

FORCE_GC

@Option(doc="If true, call System.gc() periodically.  This is useful in cases in which the -Xmx value passed is larger than the available memory.")
public java.lang.Boolean FORCE_GC

APPLY_EAMSS_FILTER

@Option(doc="Apply EAMSS filtering to identify inappropriately quality scored bases towards the ends of reads and convert their quality scores to Q2.")
public boolean APPLY_EAMSS_FILTER

MAX_READS_IN_RAM_PER_TILE

@Option(doc="Configure SortingCollections to store this many records before spilling to disk. For an indexed run, each SortingCollection gets this value/number of indices.")
public int MAX_READS_IN_RAM_PER_TILE

MINIMUM_QUALITY

@Option(doc="The minimum quality (after transforming 0s to 1s) expected from reads.  If qualities are lower than this value, an error is thrown.The default of 2 is what the Illumina\'s spec describes as the minimum, but in practice the value has been observed lower.")
public int MINIMUM_QUALITY

INCLUDE_NON_PF_READS

@Option(doc="Whether to include non-PF reads",
        shortName="NONPF",
        optional=true)
public boolean INCLUDE_NON_PF_READS

IGNORE_UNEXPECTED_BARCODES

@Option(doc="Whether to ignore reads whose barcodes are not found in LIBRARY_PARAMS.  Useful when outputting BAMs for only a subset of the barcodes in a lane.",
        shortName="INGORE_UNEXPECTED")
public boolean IGNORE_UNEXPECTED_BARCODES

MOLECULAR_INDEX_TAG

@Option(doc="The tag to use to store any molecular indexes.  If more than one molecular index is found, they will be concatenated and stored here.",
        optional=true)
public java.lang.String MOLECULAR_INDEX_TAG

MOLECULAR_INDEX_BASE_QUALITY_TAG

@Option(doc="The tag to use to store any molecular index base qualities.  If more than one molecular index is found, their qualities will be concatenated and stored here (.i.e. the number of \"M\" operators in the READ_STRUCTURE)",
        optional=true)
public java.lang.String MOLECULAR_INDEX_BASE_QUALITY_TAG

TAG_PER_MOLECULAR_INDEX

@Option(doc="The list of tags to store each molecular index.  The number of tags should match the number of molecular indexes.",
        optional=true)
public java.util.List<java.lang.String> TAG_PER_MOLECULAR_INDEX

Constructor Detail

IlluminaBasecallsToSam

public IlluminaBasecallsToSam()

Method Detail

doWork

protected int doWork()

Description copied from class: CommandLineProgram

Do the work after command line has been parsed. RuntimeException may be thrown by this method, and are reported appropriately.

Specified by:

doWork in class CommandLineProgram

Returns:

program exit status.

main

public static void main(java.lang.String[] args)

customCommandLineValidation

protected java.lang.String[] customCommandLineValidation()

Put any custom command-line validation in an override of this method. clp is initialized at this point and can be used to print usage and access args. Any options set by command-line parser can be validated.

Overrides:

customCommandLineValidation in class CommandLineProgram

Returns:

null if command line is valid. If command line is invalid, returns an array of error message to be written to the appropriate place.