Blocks and Connections

A pipeline is a graph made from blocks connected by inputs and outputs.

Blocks

FlaskTrack currently provides:

Data Source
Python
R
Output

Each block has a specific job.

Data Source → Python → R → Output

You do not need to use every block type in every pipeline.

For example:

Data Source → Python → Output

is a complete pipeline.

Inputs and outputs

Blocks exchange data using named ports.

A Python block might have:

Input:
input : Arrow table

Output:
output : Arrow table

A more advanced block can have several ports:

Inputs:
measurements : Arrow table
metadata     : JSON

Outputs:
normalized   : Arrow table
summary      : JSON

The names are important because Python and R scripts use those names.

For example:

measurements = ft.input_table("measurements")
metadata = ft.input_json("metadata")

Data types

Connections are typed.

Supported types are:

Arrow Table

Use for tabular data.

Examples:

sample rows;
measurements;
batches;
instrument result tables;
calculated datasets.

JSON

Use for structured values that are not primarily tables.

Examples:

{
  "mean": 12.4,
  "standard_deviation": 1.9,
  "passed": true
}

File

Use when a block needs to pass a file.

Examples:

CSV;
Parquet;
image;
generated document;
specialized scientific file.

Connecting blocks

Connect an output to a compatible input.

Valid:

Python:output [Arrow table]
          │
          ▼
R:input [Arrow table]

Invalid:

Python:output [Arrow table]
          │
          ▼
R:input [JSON]

One connection per input

An input receives data from one upstream output.

If you need data from multiple sources, give the block multiple named inputs.

Example:

Source A:data ───────────► Python:measurements

Source B:data ───────────► Python:sample_info

Then:

measurements = ft.input_table("measurements")
sample_info = ft.input_table("sample_info")

Required inputs

Inputs are normally required.

A pipeline cannot be published while a required input is disconnected.

Connections define what the script can access

A script does not automatically have access to all FlaskTrack data.

It only receives the inputs connected to its block.

If the block defines:

measurements

this works:

ft.input_table("measurements")

but this does not:

ft.input_table("all_samples")

unless all_samples is also defined and connected.

Pipelines cannot loop backward

Pipelines must flow forward.

Valid:

A → B → C

Invalid:

A → B → C
↑       │
└───────┘

This keeps execution order clear and reproducible.

Editing connected ports

If you need to substantially change the inputs or outputs of a connected block:

remove the affected connection;
edit the block;
save the new port definition;
reconnect the block.

This prevents an edit from silently changing which data another block receives.