How do I run a Data Studio pipeline?

A Bearer JWT, organization membership and the data-studio add-on. Editing the pipeline needs DATA_STUDIO_CREATE; running it needs DATA_STUDIO_EXECUTE.

1. How a pipeline is defined

The pipeline is part of the project's workspace configuration, set via PUT .../projects/:projectId. It has three parts: layers (the datasets brought in), transformers (the ordered processing steps), and output (how the result is shaped). Each transformer has an id, a type, an order, and a config whose shape depends on the type. Transformer types include geographic intersection, deduplication, field mapping, merging, attribute and radius filtering, sort/limit, clustering, and data-quality checks.

"transformers": [
  { "id": "step-filter", "type": "attribute_filter", "order": 1,
    "config": { "conditions": [ { "field": "stars", "operator": "in", "value": ["4", "5"] } ], "logic": "and" } },
  { "id": "step-dedupe", "type": "deduplicate", "order": 2,
    "config": { "strategy": "proximity", "proximityMeters": 50, "keepStrategy": "most_complete" } }
]

Steps run in order — here, filter on an attribute, then deduplicate by proximity.

2. Preview (dry run)

POST .../projects/:projectId/preview runs the pipeline without materializing a stored result — useful for checking the output before committing compute.

3. Execute

POST .../projects/:projectId/execute runs the transformers in order over the layers and materializes an output dataset shaped by output. The response identifies the generated result, which you retrieve via the export step. Execution is compute-heavy, so it is gated behind the DATA_STUDIO_EXECUTE permission and is more tightly rate limited.

Preview before you execute

Preview is free of a stored result and lets you validate the shape of the output; run execute only once the dry run looks right, since execution is the compute-heavy, more tightly limited step.