Skip to content

Troubleshoot instrument data pipelines

Use this guide to identify where a data pipeline is failing.

A useful first question is: Did the problem occur before FlaskTrack received the artifact, during parsing, or after the run was created?

Instrument output
    ↓
Agent / connector
    ↓
Ingestion
    ↓
Parser
    ↓
Run
    ↓
Mapping / review

Run the agent doctor command

On the laboratory machine:

flasktrack-agent doctor --config ./agent.json

This is the fastest way to verify authentication and local connector accessibility.

Agent cannot authenticate

Symptoms:

  • doctor fails before loading connectors;
  • configuration retrieval is rejected;
  • the agent repeatedly reports authentication errors.

Check:

  1. the key begins with ftia_;
  2. the key belongs to the intended organization/agent;
  3. the key was not rotated in FlaskTrack;
  4. the agent was not revoked;
  5. the machine is using the expected config file;
  6. FLASKTRACK_AGENT_KEY is not overriding the stored key with an old value.

If needed, rotate the key in FlaskTrack and update the host.

Agent shows zero connectors

Check that each connector is:

  • enabled;
  • assigned to this Instrument Agent;
  • configured as Inbound or Bidirectional where ingestion is expected.

Remote connector configuration comes from FlaskTrack; changing agent.json will not assign a connector.

Watch path is not accessible

Check the exact path from the account that runs the agent daemon.

Linux/macOS:

ls -la /path/to/instrument/results

Windows PowerShell:

Get-ChildItem "\\LAB-NAS\InstrumentResults\Reader01"

Common causes:

  • typo in the connector watch path;
  • share not mounted;
  • service account lacks permission;
  • Windows mapped drive exists only for the interactive user;
  • folder moved after connector setup;
  • NAS unavailable.

Files exist but are not uploaded

Check:

  • file pattern matches the filename;
  • file is not still changing;
  • settle_seconds has elapsed;
  • file size does not exceed max_file_bytes;
  • the file was not already recorded as successfully seen;
  • the connector is enabled;
  • the agent is still running.

Example pattern:

*.csv

will not match an .xlsx result.

File remains in the source folder after success

This is normal when:

{
  "source_file_action": "leave"
}

If you want successful source files archived, use an approved archive configuration.

Files are repeatedly uploaded

The agent maintains local file fingerprints, and FlaskTrack also applies server-side idempotency.

Repeated new ingestions may indicate that:

  • the instrument rewrites the file after each scan;
  • the filename is reused for changing data;
  • the agent state directory was deleted or changed;
  • multiple agents are monitoring the same connector/source;
  • the connector assignment is incorrect.

Ingestion is Queued for too long

The artifact reached FlaskTrack, so the local agent path is probably working.

Investigate the FlaskTrack instrument processing worker and ingestion attempt history.

Ingestion is Failed

Open the ingestion and inspect its latest processing attempt.

Typical categories include:

  • unsupported source format;
  • parser configuration error;
  • malformed input;
  • parser/source mismatch;
  • persistence failure;
  • scientific field validation failure.

Do not repeatedly retry without understanding the error if the source or parser is unchanged.

Ingestion is Quarantined

A quarantined ingestion is intentionally held out of the normal processing queue.

Inspect the reason and source artifact before releasing it.

Ingestion says Mapping required

Parsing worked. Open the resulting Instrument Run and complete its position mappings.

Do not retry the ingestion merely because mappings are incomplete.

Check:

  • you selected Sample as the mapping target type;
  • the sample belongs to the same FlaskTrack organization;
  • the sample is active/not deleted;
  • your search text matches the sample name or supported identifier;
  • the run is still in Parsed or Mapping required state.

Search is locked once review begins.

Wrong type cannot be mapped

This is intentional. A Batch UUID cannot be submitted as a Sample target, and a record from another organization cannot be used as a mapping target.

Select the correct mapping target type and search again.

Mapping controls are locked

Mappings are editable only while the run is Parsed or Mapping required.

They are locked in:

  • Ready for review;
  • Accepted;
  • Rejected.

If a controlled run requires correction, follow your organization's correction/reprocessing procedure.

Submit for review is unavailable

Confirm all required positions are mapped. The run page shows mapped and remaining counts.

Opentrons Robot Server is unreachable

From the agent host, verify:

  • correct robot address;
  • port 31950 in the connector Base URL where appropriate to your robot setup;
  • laboratory firewall/VLAN access;
  • robot power/network state;
  • the connector is assigned to the correct agent.

Do not expose the robot directly to the public internet to solve this problem.

Offline spool grows

A growing spool/ directory means the agent is staging data faster than it can successfully deliver it.

Check:

  • FlaskTrack connectivity;
  • DNS and TLS connectivity;
  • server availability;
  • agent authentication;
  • server/API errors in agent.log.

The spool is intended to protect data during temporary outages, not to be an unbounded permanent queue.

Dead-letter directory contains files

Permanent API rejections are retained under dead-letter/.

Investigate the accompanying metadata before removing anything. Correct the underlying connector, source, or server-side issue, then reintroduce the artifact through an approved process if necessary.

Useful agent commands

flasktrack-agent version
flasktrack-agent doctor --config ./agent.json
flasktrack-agent once --config ./agent.json
flasktrack-agent run --config ./agent.json

Information to collect for support

When escalating an issue, collect:

  • FlaskTrack ingestion ID, if one exists;
  • instrument name;
  • connector name;
  • agent name;
  • agent version;
  • operating system;
  • source filename without confidential path information where possible;
  • ingestion status;
  • parser key and version;
  • latest attempt error;
  • relevant agent.log lines with credentials removed.

Never include the ftia_... agent key.