Core Concepts & Design Decisions

Experimental - Not Reviewed

Content may be incomplete or inaccurate.

This document explains the fundamental concepts that underpin StarryNight's architecture and the rationale behind key design decisions. Understanding these principles and trade-offs is essential for working effectively with the framework.

Pipeline Terminology

The term "pipeline" has two distinct meanings in StarryNight:

• CellProfiler Pipeline (.cppipe files): Image processing pipeline definition that contains CellProfiler modules and parameters, generated by algorithm functions, and executed by CellProfiler software.
• StarryNight Pipeline (Pipecraft graphs): Workflow orchestration pipeline that connects multiple modules into workflows, defines execution order and parallelism, and is executed by backends like Snakemake.

Use "CellProfiler pipeline" or ".cppipe" for image processing definitions, "workflow pipeline" or "Pipecraft pipeline" for StarryNight orchestration, and rely on context to make the distinction clear.

Core Architecture

Layered Architecture

StarryNight employs a strict layered architecture where each layer depends only on layers below it:

• Upward Dependencies Only: Higher layers import from lower layers, never the reverse
• Clean Interfaces: Each layer exposes well-defined interfaces
• Progressive Enhancement: Each layer adds capabilities without modifying lower layers
• Testability: Lower layers can be tested in isolation

Rationale: This design provides testability (lower layers can be tested without framework setup), reusability (algorithm functions work outside StarryNight), maintainability (changes in lower layers don't break higher layers), and clear contracts (each layer has well-defined interfaces).

Trade-offs: More layers mean more abstraction overhead, requires discipline to maintain layer boundaries, and may duplicate some logic across layers.

Algorithm Independence

The algorithm layer's complete independence is fundamental to StarryNight's design. Algorithm functions are implemented as standalone Python functions that accept standard Python types as inputs and produce outputs using only Python standard library and common third-party libraries.

Rationale: Algorithm layer has zero dependencies on other StarryNight components to ensure portability (algorithms can be used in any Python environment), testing (no framework setup required for unit tests), clarity (business logic separated from infrastructure), and evolution (framework can change without affecting algorithms).

Layer-Specific Patterns

Specification vs Implementation

StarryNight separates "what" from "how" at multiple levels:

• Module Level: Specification (Bilayers) defines inputs, outputs, and parameters; compute graph (Pipecraft) defines execution structure; and execution (backend) performs actual computation.
• Pipeline Level: Definition provides abstract workflow description, translation creates backend-specific execution plan, and runtime performs actual process execution.

Rationale: Modules define both specifications (Bilayers) and compute graphs (Pipecraft) to achieve backend independence (same module runs on different platforms), inspection (can examine workflow before execution), UI generation (specs enable automatic interface creation), and validation (type checking before runtime).

Trade-offs: Additional complexity in maintaining both specifications and implementations, requires understanding of multiple abstraction layers.

Module Layer as CLI Wrapper

Modules invoke CLI commands rather than calling algorithms directly. This provides consistency (single execution path for all invocations), isolation (container boundaries match CLI boundaries), testing (CLI provides natural integration test points), and debugging (can manually run same commands modules generate).

Rationale: Using CLI as the interface layer ensures all execution paths go through the same validated entry points, making behavior predictable and debugging straightforward.

Trade-offs: Additional overhead of process creation, string-based command construction.

Three-Function Pattern (CellProfiler Workflows)

CellProfiler-based workflows in StarryNight follow a consistent three-step pattern:

1. LoadData Generation: Identifies what images to process by reading indexes/inventories, filtering by criteria, and outputting CSV files for CellProfiler.
2. Pipeline Generation: Defines how to process images by inferring parameters from sample data, configuring CellProfiler modules, and outputting .cppipe pipeline definitions.
3. Execution: Runs the CellProfiler pipeline by managing parallel execution and organizing outputs.

Rationale: Organizing CellProfiler algorithm sets around LoadData → Pipeline → Execution pattern provides consistency (same pattern across CellProfiler analyses), modularity (each step has clear purpose), flexibility (can mix and match components), and understanding (predictable structure aids comprehension).

Clarification: Other algorithm types (indexing, inventory, quality control) follow different patterns based on their purpose.

Module & Pipeline Concepts

Module Composition

Modules compose into pipelines through typed interfaces, execution patterns, and resource management:

• Input/Output Contracts: Typed interfaces define connections, outputs of one module feed inputs of another, and type checking ensures compatibility.
• Execution Patterns: Sequential blocks for dependent operations, parallel blocks for independent operations, and nested blocks for complex workflows.
• Resource Management: Modules declare resource requirements, execution layer manages allocation, and automatic cleanup after completion.

Rationale: Build complex workflows by composing simple modules to achieve flexibility (mix and match components), testing (test modules independently), reuse (share modules across workflows), and understanding (smaller units easier to comprehend).

Pipecraft as Separate Package

Pipeline construction is implemented as a separate package within the monorepo. This provides focus (clean separation of concerns), reusability (could be used by other projects), testing (independent test suite), and evolution (can evolve independently).

Rationale: Separating workflow construction from core StarryNight functionality allows the pipeline engine to evolve independently and potentially be reused by other projects with similar needs.

Clarification: Pipecraft creates StarryNight workflow pipelines (compute graphs), not CellProfiler pipelines (.cppipe files).

Cross-Cutting Concerns

Path Abstraction

All StarryNight components use cloudpathlib's AnyPath for storage abstraction. The cloudpathlib AnyPath abstraction provides a unified interface for accessing both local filesystem paths and cloud storage locations (S3, Google Cloud Storage, Azure Blob Storage).

Rationale: Using AnyPath abstraction for all file operations provides storage agnostic code (same code for local/S3/GCS), future proofing (easy to add new storage backends), clean code (no if/else for different path types), and testing (can mock storage in tests).

Trade-offs: Adds dependency on cloudpathlib and may have slight performance overhead compared to native path operations.

Container-Based Execution

All processing runs in containers to ensure reproducibility (fixed software versions, isolated dependencies, consistent environments), portability (runs on any container platform, cloud-native execution, local development mirrors production), and scalability (parallel container instances, resource isolation, efficient scheduling).

Rationale: Container-based execution with explicit specifications ensures reproducibility (fixed versions and dependencies), portability (same containers run locally and in cloud), isolation (no dependency conflicts), and scalability (natural unit for parallel execution).

Trade-offs: Container overhead for small operations, requires container runtime, and more complex local development setup.

Parameter Inference

StarryNight minimizes configuration through intelligent inference:

• Index Analysis: Extract metadata from data
• Sample Inspection: Infer parameters from representative files
• Smart Defaults: Provide sensible defaults for common cases
• User Override: Allow explicit configuration when needed

Rationale: Infer parameters from data when possible rather than requiring explicit configuration to improve user experience (less configuration burden), correctness (parameters match actual data), adaptability (automatically adjusts to data changes), and defaults (smart defaults for common cases).

Trade-offs: More complex implementation logic, potential for incorrect inference in edge cases, and debugging complexity when inference fails.

Example: The system infers parameters from data when possible (e.g., counting unique channels from the index) rather than requiring users to specify values that can be determined automatically.

Cross-Cutting Configuration

The configuration layer operates orthogonally to other layers, influencing behavior across the entire stack. This enables consistent parameters across layers, experiment-specific adaptations, and runtime behavior modification.

Trade-offs: More complex parameter resolution logic and potential for configuration conflicts across layers.

Execution & Backend

Backend Abstraction

StarryNight supports multiple execution backends through abstraction:

• Current: Snakemake - Rule-based workflow engine with automatic dependency resolution and built-in parallelism.
• Possible: Others - AWS Batch, Google Cloud Run, Kubernetes, and local execution.

Rationale: Using Snakemake as the first execution backend provides maturity (battle-tested in bioinformatics), features (built-in parallelism, dependency resolution), community (active development and support), and flexibility (supports local and cluster execution). The architecture allows adding other backends (Nextflow, CWL, etc.) in the future.

Trade-offs: Initial implementation tied to Snakemake's execution model, requires abstraction layer to support multiple backends effectively.

Development Practices

Error Handling Philosophy

StarryNight follows consistent error handling principles:

• Fail Fast: Detect problems early
• Clear Messages: Provide actionable error information
• Graceful Degradation: Continue processing unaffected data
• Audit Trail: Log all operations for debugging

Rationale: Consistent error handling across all layers improves debugging experience, reduces time to resolution, and enables partial processing of large datasets when some components fail.

Trade-offs: Requires discipline to implement consistently and may add verbosity to error handling code.

Testing Strategy

StarryNight employs layer-specific testing approaches, though implementation is currently in progress:

Current Implementation:

• Algorithm: Unit tests for core algorithms (e.g., index generation)
• Integration: End-to-end workflow tests for key use cases
• Parsers: Component tests for data parsing functionality
• Utilities: Unit tests for helper functions

Planned Implementation:

• CLI: Integration tests with command execution
• Module: Specification validation tests
• Pipeline: Composition and graph tests
• Execution: Backend-specific execution tests
• Configuration: Parameter inference tests

Rationale: Layer-specific testing strategies should match the abstraction level and purpose of each component, enabling comprehensive coverage without unnecessary complexity.

Trade-offs: Requires maintaining multiple test suites and understanding different testing approaches for different layers.

Trade-off Summary

Decision	Benefits	Costs
Layered Architecture	Clear boundaries, testability	Abstraction overhead
Algorithm Independence	Portability, simplicity	No framework features
CLI Wrapping	Consistency, isolation	Process overhead
Container Execution	Reproducibility, scalability	Setup complexity
Parameter Inference	Better UX, correctness	Complex implementation

These decisions create a system that prioritizes:

1. Reproducibility over convenience
2. Flexibility over simplicity
3. Explicitness over magic
4. Composition over monoliths

Future architectural decisions should align with these principles while pragmatically addressing user needs.

Next Steps

For implementation details, see the individual layer documentation files (01-algorithm.md through 06-configuration.md).