Tables and dataframes

Arrow table ports provide a consistent interchange format even when different blocks use different processing libraries. Choose one library inside a block and keep its methods and expressions consistent.

Grouping syntax by library

Grouping is a common source of generated-script errors because the method name is different in each library.

Library	Correct grouping syntax
PyArrow	table.group_by(keys).aggregate(specs)
Polars	frame.group_by(keys).agg(expressions)
pandas	frame.groupby(keys).agg(expressions)
R dplyr	dplyr::group_by(frame, ...) followed by dplyr::summarise()

groupBy is not valid for PyArrow, Polars, pandas, or dplyr.

PyArrow

ft.input_table returns a pyarrow.Table, making PyArrow the most direct choice when the required operation is supported.

import pyarrow.compute as pc

import flasktrack as ft

rows = ft.input_table("rows")
clean = rows.filter(pc.greater_equal(rows["value"], 0))
summary = clean.group_by("sample_id").aggregate(
    [("value", "mean"), ("value", "count")]
)
ft.output_table("summary", summary)

Polars

Convert once, process with Polars, and pass the final dataframe directly to output_table.

import polars as pl

import flasktrack as ft

rows = pl.from_arrow(ft.input_table("rows"))
summary = (
    rows
    .filter(pl.col("value").is_not_null())
    .group_by("sample_id")
    .agg(
        pl.col("value").mean().alias("value_mean"),
        pl.len().alias("observation_count"),
    )
)
ft.output_table("summary", summary)

pandas

Convert to pandas only when its API is needed. For large inputs, this may use more memory than processing the Arrow table directly.

import flasktrack as ft

rows = ft.input_table("rows").to_pandas()
summary = (
    rows.dropna(subset=["value"])
    .groupby("sample_id", as_index=False)
    .agg(value_mean=("value", "mean"), observation_count=("value", "size"))
)
ft.output_table("summary", summary)

R with dplyr

source("/opt/flasktrack/flasktrack.R")

rows <- ft_input_table("rows")
summary <- rows |>
  dplyr::filter(!is.na(.data$value)) |>
  dplyr::group_by(.data$sample_id) |>
  dplyr::summarise(
    value_mean = mean(.data$value),
    observation_count = dplyr::n(),
    .groups = "drop"
  )

ft_output_table("summary", summary)

Validate required columns

Fail early with a clear list of missing columns rather than allowing a later method call to fail ambiguously.

Python

required = {"sample_id", "value"}
missing = required.difference(rows.column_names)
if missing:
    raise ValueError(f"Missing required columns: {sorted(missing)}")

R

required <- c("sample_id", "value")
missing <- setdiff(required, names(rows))
if (length(missing) > 0L) {
  stop(paste("Missing required columns:", paste(missing, collapse = ", ")))
}

Also verify that numeric, timestamp, and categorical columns have the types your calculation expects.