Python Scripts

The pipeline includes helper Python scripts that handle data preparation and quality control. You generally don’t need to modify these scripts, but understanding what they do helps with debugging when things go wrong.

Overview¶

These scripts live in the bin/ directory and are automatically available to all pipeline processes:

• generate_load_data_csv.py: Universal CSV generator for CellProfiler

• qc_barcode_align.py: Barcode alignment quality control

• qc_barcode_preprocess.py: Barcode preprocessing quality control

• qc_painting_align.py: Multicycle painting alignment quality control

• check_duplicate_images.py: Pixel-duplicate safety-net check (optional, --check_duplicate_images)

• montage.py: QC montage generation

Used only by the STITCH_ALIGN_CROP entrypoint (see Pipeline Overview):

• stitch.py: Stitches one well (and cycle) of per-site images with ashlar

• align_intraarm.py: Cross-cycle alignment within one arm

• align_combined.py: Cross-arm (painting to barcoding) scaling and alignment

• crop.py: Crops full-well stitched images into tiles

These scripts are automatically available in the process PATH and are called during pipeline execution.

generate_load_data_csv.py¶

Purpose¶

Generates load_data.csv files required by CellProfiler processes. This is the primary data staging script used throughout the pipeline, and it is the same script for all 5 CellProfiler modules (illumcalc, illumapply, segcheck, preprocess, combinedanalysis) - there is no per-module branching, filename parsing, or file discovery in the script itself.

Usage¶

generate_load_data_csv.py \
    --metadata-json metadata.json \
    --images-dir ./images \
    --output load_data.csv \
    [--illum-dir ./images --include-illum-files] \
    [--range-skip N] \
    [--has-cycles] \
    [--cycle N] \
    [--cycle-metadata-name Cycle] \
    [--outdir <pipeline outdir>]

Parameters¶

The canonical metadata JSON¶

Every caller builds the same top-level object, with one image_metadata entry per (physical file, channel) pair:

{
  "plate": "Plate1", "batch": "Batch1", "cycles": [1, 2, 3],
  "image_metadata": [
    {
      "well": "A1", "site": 0, "arm": "painting",
      "cycle": 1, "channel": "DNA", "frame_index": 2, "column_prefix": "Orig",
      "filename": "...", "original_path": "...", "original_filename": "..."
    }
  ]
}

One physical multi-frame file (e.g. a 3-frame OME-TIFF) contributes multiple image_metadata entries sharing one filename, differing in channel/frame_index. A single-channel file contributes one entry with frame_index omitted.

Column naming¶

A single rule replaces the old per-pipeline-type branches:

name = f"{cycle_prefix}{column_prefix}{channel}"
cycle_prefix = f"Cycle{cycle:02d}_" if (cycle is not None and more than one distinct cycle across the plate) else ""

FileName_<name> and FinalFileName_<name> are always emitted; Frame_<name> is emitted only when the entry carries a frame_index. When --include-illum-files is set, a matching FileName_Illum<name> / FinalFileName_Illum<name> pair is added by looking up illumination .npy files collected from --illum-dir (these aren’t part of image_metadata - they’re still matched by scanning the directory and parsing {plate}_Illum{channel}.npy / {plate}_Cycle{N}_Illum{channel}.npy filenames).

CSV Output Structures¶

Standard (Cell Painting, illumcalc/illumapply):

With Cycles (Barcoding, preprocess):

Combined Analysis:

Implementation Details¶

The script:

1. Loads and validates the metadata JSON, cross-checking declared cycles against the cycles actually present in image_metadata

2. Joins and checks existence of images_dir + entry['filename'] directly for every entry - no directory glob, no filename regex, no substring search

3. Groups entries by (plate, well, site), optionally subsampling sites per well via --range-skip

4. Generates CSV rows using the single column-naming rule above

Error Handling¶

The script fails loudly (non-zero exit, descriptive ValueError) on:

• A metadata JSON entry referencing a file that doesn’t exist under --images-dir

• Missing required per-entry fields (well, site, arm, channel, column_prefix, filename)

• An arm or column_prefix value outside the allowed set

• A mismatch between the declared top-level cycles and the cycles actually present in entries

qc_barcode_align.py¶

Purpose¶

Analyzes barcode alignment quality across cycles. The file is a jupytext “percent” script that the QC_BARCODEALIGN process converts to a notebook (jupytext), executes with papermill, and exports to HTML (nbconvert). qc_painting_align.py does the same for multicycle painting (QC_PAINTINGALIGN).

Functionality¶

1. Load alignment statistics: Reads each well’s BarcodingApplication_Image.csv

2. Compare cycles: Summarizes pixel shifts and correlation scores between cycles

3. Generate visualizations: Shift and correlation plots, including a spatial plot using the acquisition geometry (acquisition_geometry_rows / acquisition_geometry_columns)

4. Apply thresholds: Summarizes sites whose shift exceeds barcoding_shift_threshold and draws barcoding_corr_threshold as a reference line on the correlation plots

Parameters¶

Injected by papermill (see the parameters cell in the script): input_dir, output_dir, use_cache, numcycles, shift_threshold, corr_threshold, and optionally rows / columns. These are set by the process, not by command-line flags.

Output Files¶

• {id}_qc_barcode_align.ipynb: Executed notebook

• {id}_qc_barcode_align.html: HTML report

• *.png: Optional extra figures (the process accepts them if present)

Published to results/workspace/qc_reports/{batch}/ (see Outputs).

QC Criteria¶

The thresholds are for human review of the report; the notebook does not fail or stop the pipeline when they are exceeded.

check_duplicate_images.py¶

Fails (exit 1) if any two .npy/.tif/.tiff files under a directory have pixel-identical content, or any file cannot be read. Blank frames (all 0 or all max value, allowing up to 0.01% stray pixels) are ignored. Run by the QC_CHECKDUPLICATEIMAGES process only when --check_duplicate_images true.

check_duplicate_images.py <images_dir> [--report report.txt]

stitch.py¶

Stitches one well and one cycle of illumination-corrected per-site images into one full-well image per channel with ashlar. Sites are copied into a raster-ordered temporary grid using the layouts in assets/stitchcrop/well_site_layouts.json (empty grid corners of round wells are filled with seeded low-amplitude noise). Prints a WARNING: for sites that fall outside the layout (dropped) and for layout sites with no image (noise-filled). Writes Plate_{plate}_Well_{well}_Site_1_Stitched_Cycle{NN}_[Corr]{channel}.tiff plus 10X-downsampled copies. Square wells currently support only --stitchorder "Grid: snake by rows".

align_intraarm.py and align_combined.py¶

align_intraarm.py registers every cycle of one arm to its first cycle using phase cross-correlation on the reference channel, applying one shift to all channels of the cycle (single-cycle input is copied through). align_combined.py scales both arms to a common pixel size and registers painting to barcoding cycle 1, shifting painting only. Both write stats CSVs and QC overlay PNGs. --shift-threshold / --corr-threshold are recorded in the stats but not enforced.

crop.py¶

Crops each full-well stitched image into tileperside x tileperside tiles of final_tile_size pixels (zero-padding on the bottom/right if the stitched image is smaller). Tiles are numbered column-major starting at the well’s first site index and written as Plate_{plate}_Well_{well}_Site_{n}_{suffix}.tiff.

Development Guide¶

Adding a New CellProfiler Stage¶

Because every module already builds the same canonical metadata JSON, adding a new CellProfiler stage does not require touching generate_load_data_csv.py at all - the script has no per-stage branching left to extend. Instead:

1. Build the metadata JSON in the new module’s Groovy code, using expandImageChannels/buildLoadDataMetadata from subworkflows/local/utils_nfcore_nf-pooled-cellpainting_pipeline/main.nf (or constructing the equivalent {plate, batch, cycles, image_metadata} shape by hand), choosing the right column_prefix (""/"Orig"/"Corr") for whichever cppipe stage will read the resulting CSV.

2. Call the script unchanged from the new module’s script: block:

script:
def metadata_json_content = groovy.json.JsonOutput.toJson(image_metas)
def metadata_base64 = metadata_json_content.bytes.encodeBase64().toString()
"""
echo '${metadata_base64}' | base64 -d > metadata.json

generate_load_data_csv.py \
    --metadata-json metadata.json \
    --images-dir ./images \
    --output load_data.csv \
    --cycle-metadata-name "${params.cycle_metadata_name}"
"""

3. Add a modules/local/cellprofiler/<newstage>/tests/main.nf.test fixture using the same canonical schema as the other 5 modules’ fixtures.

Testing Scripts¶

Test the script in isolation with a hand-built metadata JSON:

# Create test data
mkdir -p test_images
touch test_images/WellA1_Site0_DAPI.tiff
touch test_images/WellA1_Site0_GFP.tiff

cat > metadata.json <<'EOF'
{
  "plate": "Plate1",
  "image_metadata": [
    {"well": "A1", "site": 0, "arm": "painting", "channel": "DAPI", "column_prefix": "Orig", "filename": "WellA1_Site0_DAPI.tiff"},
    {"well": "A1", "site": 0, "arm": "painting", "channel": "GFP", "column_prefix": "Orig", "filename": "WellA1_Site0_GFP.tiff"}
  ]
}
EOF

# Run script
python bin/generate_load_data_csv.py \
    --metadata-json metadata.json \
    --images-dir test_images \
    --output test_load_data.csv

# Validate output
head test_load_data.csv

Debugging¶

Enable verbose logging:

import logging
logging.basicConfig(level=logging.DEBUG)

Best Practices¶

1. Validate inputs: Check file existence before processing

2. Handle edge cases: Empty directories, missing frames, etc.

3. Consistent naming: Follow established filename conventions

4. Error messages: Provide clear, actionable error messages

5. Logging: Log key operations for debugging

6. Testing: Write unit tests for parsing logic