Workflow Pipelines

Once one or more Datasets have been added to a Workflow, the next step is to create Pipelines to process, transform, and prepare the data for output. Pipelines are where your business logic, mapping, and data harmonization take place.

The Pipelines tab is divided into two sections: Output Pipelines and Supporting Pipelines.

You must create at least one Output Pipeline before your Workflow can be executed. When no Output Pipelines have been created yet, the tab displays a "No Output Pipelines yet" empty state with a warning prompting you to add one.

To get started, navigate to the Pipelines tab. If you are coming from the Data tab, click Go to Pipelines in the next step prompt at the bottom of the page, or select Pipelines from the top navigation within the Workflow.

Click the Add button in the upper right of the Output Pipelines section to create your first Output Pipeline.

After clicking Add, a Pipeline setup panel will open. Enter a name for your Pipeline and configure it according to your data transformation requirements. Use the AI Copilot to build your transformation logic using natural language.

Once your Pipeline is configured, save it to return to the Pipeline overview. Your new Output Pipeline will appear in the list with its name, last edited timestamp, and column match status.

When you have more than one Output Pipeline, one must be designated as Primary. The Primary Output Pipeline is the source of truth for your Workflow's output structure. It defines:

All other Output Pipelines are stacked on top of the Primary, meaning their rows are combined into a single unified output at execution time. Only Output Pipelines are stacked; Supporting Pipelines are not included in this behavior.

You can change which Output Pipeline is designated as Primary at any time. To do so, click the actions icon (⋮) to the right of the Pipeline you want to promote, then select Mark as Primary from the dropdown.

Once marked as Primary, that Pipeline's column structure becomes the new standard for all other Output Pipelines.

The Columns column in the Output Pipelines list gives you an at-a-glance status for each Pipeline:

When you open a non-conforming Pipeline, a warning banner appears in the Transformations panel that identifies the specific issue. For example, if a non-Primary Pipeline contains an extra column that does not exist in the Primary, the warning will display:

To resolve the warning, update the non-conforming Pipeline to align its columns with the Primary, either by removing extra columns or adjusting the transformation logic.

Alternatively, if the non-conforming Pipeline's structure is correct, use Mark as Primary to promote it and redefine the expected schema for all other Output Pipelines.

Supporting Pipelines are optional and must be explicitly enabled. From the Pipelines tab, click Enable Supporting Pipelines in the prompt at the bottom of the page.

Once enabled, a Supporting Pipelines section will appear above the Output Pipelines list. Click Add in that section to create your first Supporting Pipeline.

After clicking Add in the Supporting Pipelines section, a Pipeline setup panel will open. The setup experience is the same as creating an Output Pipeline (see Setting Up a Pipeline above). Unlike Output Pipelines, Supporting Pipelines do not need to conform to the Workflow's Output Column schema, giving you flexibility to shape intermediate data however your transformation logic requires.

Once saved, return to the Pipeline overview. Your Supporting Pipeline will appear at the top of the Pipelines tab, above your Output Pipelines, reflecting its earlier execution order in the Workflow.

Use Supporting Pipelines to organize complex transformations into manageable stages, especially when working with multiple input formats or enrichment layers.

Because Supporting Pipelines execute in order, sequence matters. A Supporting Pipeline that appears earlier in the list runs first, and its output can be used as an input by any Pipeline that follows it.

To reorder Supporting Pipelines, grab the drag handle (the equals icon on the left side of any Supporting Pipeline row) and drag it to the desired position.

One of the most powerful features of Supporting Pipelines is the ability to chain them together. There are two ways to use the output of a prior Supporting Pipeline in a subsequent Pipeline: as the primary input Dataset, or as the target of a Join transformation.

When creating or editing a Pipeline, the Select Dataset dropdown includes not only your source Datasets but also the outputs of any Supporting Pipelines that are ordered earlier in the list. Simply select the prior Supporting Pipeline's output as your starting Dataset and build your transformation logic from there.

You can also reference a prior Supporting Pipeline's output in a Join transformation within any subsequent Pipeline. When adding a Join transformation, the Dataset to Join dropdown will include both your source Datasets and the outputs of any Supporting Pipelines that precede the current one in the execution order.

This gives you two distinct ways to incorporate staged data into your transformation logic: starting from it, or joining against it. Both approaches allow you to break complex mapping and transformation logic into discrete, maintainable components rather than handling everything in a single Pipeline. Each stage has a clear purpose, the order is visible and adjustable, and individual steps can be modified or replaced without disrupting the rest of the Workflow.

To manage an existing Pipeline, click the actions icon (⋮) to the right of any Pipeline in the list. This opens a dropdown with the following options:

Once your Output Pipeline is configured, you can proceed to define your Workflow's outputs. Click Go to Outputs in the next step prompt at the bottom of the page, or select the Outputs tab from the top navigation within the Workflow.

Ready to configure your destinations? Continue to Workflow Outputs.