Skip to content

Administrator Setup

This page is for FlaskTrack administrators deploying Data Pipelines.

Most users do not need these settings.

Architecture

Data Pipelines use two FlaskTrack processes:

FlaskTrack web application
        │
        │ queues runs
        ▼
PostgreSQL
        ▲
        │ claims runs
        │
Pipeline worker
        │
        ▼
Python / R runtime containers

The web application manages:

  • pipeline definitions;
  • editing;
  • publishing;
  • run requests;
  • run pages.

The worker executes published pipelines.

Required worker configuration

The pipeline worker requires access to the FlaskTrack database, file storage, and Data Lake.

Typical required environment variables include:

DATABASE_URL
S3_BUCKET
DATALAKE_FLIGHT_URL
JWT_SECRET

Your configured FlaskTrack file-storage backend may require additional S3, MinIO, or local-storage settings.

Worker settings

Optional worker settings include:

FLASKTRACK_PIPELINE_OCI_RUNTIME=podman

FLASKTRACK_PIPELINE_WORKSPACE=/tmp/ft-work
FLASKTRACK_PIPELINE_IPC_DIR=/tmp/ft-ipc
FLASKTRACK_PIPELINE_DB_MAX_CONNECTIONS=8
FLASKTRACK_PIPELINE_POLL_SECONDS=2
FLASKTRACK_PIPELINE_LEASE_SECONDS=300
FLASKTRACK_PIPELINE_FLIGHT_TOKEN_TTL_SECONDS=300
FLASKTRACK_PIPELINE_WORKER_ID=worker-01

Runtime

The current sandbox is designed for an OCI runtime such as Podman.

The worker host must have the approved Python and R pipeline images available.

Default image names are:

registry.digitalocean.com/flasktrack/flasktrack-pipeline-python:1
registry.digitalocean.com/flasktrack/flasktrack-pipeline-r:1

User package requests

Pipeline containers do not have normal network access.

If users need another Python or R package, add it to an approved runtime image and deploy that image through your normal FlaskTrack release process.

Workspace

Use local worker storage for temporary run files.

Example:

/var/lib/flasktrack/pipeline-work

The workspace is temporary.

Durable results belong in FlaskTrack file storage.

Starting the worker

Run the dedicated pipeline worker binary as a separate service from the main FlaskTrack web application.

The worker should be supervised by your normal service manager or container orchestration system.

Multiple workers

More than one worker can consume queued runs.

Use additional workers when pipeline demand exceeds the capacity of one worker host.

Monitoring

Administrators should monitor:

  • queued runs;
  • runs waiting unusually long;
  • failed runs;
  • timed-out blocks;
  • worker restarts;
  • container-runtime failures;
  • file-storage failures.

Security expectations

Pipeline scripts are intentionally isolated from normal FlaskTrack credentials.

Do not change the worker to inject database, object-storage, API, session, or Data Lake credentials directly into user scripts.

Scripts should access only the inputs and outputs declared by the pipeline graph.