Getting Started with StarryNight

Implementation Approaches

GUI Approach (coming soon): Point-and-click interface through Canvas for biologists who prefer visual workflows.

CLI Approach (shown in this guide): Uses direct command-line commands for learning and exploration. It's a simpler way to understand workflow operations step-by-step.

Python/Module Approach (used in production): Most users will execute these operations through Python code (as shown in starrynight/notebooks/pypct/exec_pcp_generic_pipe.py). This approach provides standardized components, containerized execution, and integration with the Canvas UI.

This guide focuses on the CLI approach as a foundation. See Practical Integration for the Python implementation and Architecture Overview for system design details.

Focus of This Guide

This guide focuses only on the Cell Painting (CP) track and specifically on the illumination correction calculation step. The Complete Workflow Example will add the Sequencing by Synthesis (SBS) (commonly referred to as barcoding) track and show the full analysis workflow.

Environment Variable Organization

The directory structure separates input data from output workspace:

• DATADIR: Points to the downloaded sample data (input)
• WKDIR: Points to your working directory where outputs will be generated
• INPUT_WKDIR: Points to the input workspace folder, within which barcode files are located
• CP_PLUGINS: Points to CellProfiler plugins needed for advanced processing

Available Experiment Configurations

StarryNight currently supports these experiment types:

• "Pooled CellPainting [Generic]": Standard Cell Painting workflow with SBS barcoding
• "Pooled CellPainting [Stitchcrop]": Cell Painting workflow with image stitching and cropping

Both configurations use the same underlying experiment model and support the full range of Cell Painting and SBS processing steps. The choice determines which pipeline templates and processing modules are available for your workflow.

Path Parsing System

StarryNight automatically extracts metadata from file paths using a grammar-based parsing system. The default parser handles the file structure shown in this example, but you can write custom parsers for different organizations. If your data follows a different organization than the pattern used as an example here, you can customize the parser as described in the Parser Configuration guide.

Pipeline Generation Approaches

StarryNight offers two ways to generate CellProfiler pipelines:

• Pre-fabricated Pipelines: Uses established, tested pipeline templates (add --use_legacy flag)
• Dynamic Generation: Automatically generates pipelines based on configuration (omit the --use_legacy flag)

This guide uses the pre-fabricated approach for stability.

Directory Structure

Throughout this guide, we're creating a workspace with this directory structure:

${WKDIR}/
├── cellprofiler/                # CellProfiler-related files
│   ├── loaddata/                # Generated LoadData CSV files
│   └── cppipe/                  # Pipeline files
├── index/                       # Structured metadata
│   └── index.parquet            # Index file with extracted metadata
├── illum/                       # Illumination correction files
│   ├── cp/                      # Cell Painting illumination
│   └── sbs/                     # SBS illumination
└── experiment.json              # Experiment configuration

This structure separates inputs, intermediate results, and final outputs, maintaining clear data provenance throughout the workflow.

For Document Contributors

This section contains editorial guidelines for maintaining this document. These guidelines are intended for contributors and maintainers, not end users.

Purpose and Audience

• Introductory Focus - This document is a user's first hands-on experience with StarryNight
• CLI Emphasis - Prioritize the CLI approach as an accessible entry point
• Single Path with Options - Present one primary workflow while noting alternatives
• Assumed Knowledge - Users understand basic command line but not StarryNight architecture

Structure Principles

1. Clear section headings - Use H2 headings for main workflow steps without numbers
2. Notes for alternatives - Use MkDocs admonitions to present alternatives without disrupting flow
3. Quick start spirit - Keep explanations brief and focused on practical execution
4. Progressive detail - Start with setup, then basic workflow, then advanced options
5. Clear prerequisites - Ensure directory creation and dependencies are explicitly mentioned

Content Style Guidelines

1. Command formatting - Include descriptive comments in code blocks
2. Bold subheadings - Use bold text rather than deeper heading levels for substeps
3. Copy-pastable commands - Ensure commands work as written without modification
4. Environment variables - Use consistent variables (DATADIR, WKDIR)
5. Expected outputs - Show example outputs and file structures where appropriate

Terminology Consistency

• "CLI approach" vs "Python/Module approach" - Different ways to use StarryNight
• "Pre-fabricated pipelines" vs "Dynamic pipeline generation" - Two pipeline generation methods
• "Workflow" - The end-to-end image processing sequence
• "Pipeline" - The CellProfiler processing definition
• "LoadData files" - CSV files that tell CellProfiler which images to process