Data Sources

Every pipeline starts with data.

A Data Source block brings data into the pipeline and makes it available to downstream Python, R, or Output blocks.

FlaskTrack supports four source types:

Datalake Dataset

Use Datalake Dataset when the data you need already exists as a dataset available through FlaskTrack Data Studio.

This is useful when you want to build a repeatable workflow from data that FlaskTrack already exposes.

Typical examples include:

Select the dataset in the source block and save it.

The block makes the dataset available to the next block in the pipeline.

Use Flight SQL when you want to define the source with SQL.

Example:

SELECT
    sample_id,
    measured_at,
    value
FROM measurement_results
WHERE measured_at >= CURRENT_DATE - INTERVAL '30 days'

Flight SQL sources are read-only.

Use them to select, filter, join, or aggregate data before it reaches Python or R.

For example:

SELECT
    batch_id,
    COUNT(*) AS sample_count,
    AVG(value) AS mean_value
FROM measurement_results
GROUP BY batch_id

When possible, do filtering and basic aggregation in the source query.

Instead of loading a very large table into Python and filtering it there:

SELECT *
FROM measurement_results

prefer:

SELECT
    sample_id,
    batch_id,
    value
FROM measurement_results
WHERE measured_at >= CURRENT_DATE - INTERVAL '30 days'

This reduces the amount of data that later blocks need to process.

Use Saved Report when you already have a report in Data Studio that defines the data you want.

This lets you reuse the same report definition inside a pipeline.

Example workflow:

Saved Report
     │
     ▼
Python
     │
     ▼
Output

This is useful when a report is already your organization's standard definition of a dataset.

Use FlaskTrack File when the source is a file already stored in FlaskTrack.

Examples:

The pipeline receives access only to the selected file.

Your Python or R block can then read the file using the FlaskTrack pipeline helpers.

Python:

import flasktrack as ft

path = ft.input_file("input")

R:

source("/opt/flasktrack/flasktrack.R")

path <- ft_input_file("input")

Data Source blocks expose an output port.

The most common type is:

Arrow table

Other pipeline data types are:

The output type must match the input type of the block you connect it to.

For example:

Arrow table → Arrow table

is valid.

Arrow table → JSON

is not.

Use this rule of thumb: