Create a Pipeline

Create a new pipeline within a pipeline service

POST /v1/pipelines

POST /v1/pipelines

Python

from metadata.sdk import configure
from metadata.sdk.entities import Pipelines
from metadata.generated.schema.api.data.createPipeline import CreatePipelineRequest

configure(
    host="https://your-company.open-metadata.org/api",
    jwt_token="your-jwt-token"
)

request = CreatePipelineRequest(
    name="dbt_analytics_customers",
    displayName="DBT Customer Analytics",
    service="sample_airflow",
    description="Analytics pipeline for customer data processing",
    sourceUrl="http://localhost:8080/tree?dag_id=dbt_analytics_customers",
    tasks=[
        {
            "name": "extract_customers",
            "taskType": "BatchTask",
            "description": "Extract customer data from source",
            "downstreamTasks": ["transform_customers"]
        },
        {
            "name": "transform_customers",
            "taskType": "BatchTask",
            "description": "Transform and clean customer data",
            "downstreamTasks": ["load_customers"]
        },
        {
            "name": "load_customers",
            "taskType": "BatchTask",
            "description": "Load transformed data into warehouse"
        }
    ]
)

pipeline = Pipelines.create(request)
print(f"Created: {pipeline.fullyQualifiedName}")

Response

{
  "id": "538faa63-d204-46ff-aead-d158d0401cac",
  "name": "dbt_analytics_customers",
  "displayName": "DBT Customer Analytics",
  "fullyQualifiedName": "sample_airflow.dbt_analytics_customers",
  "description": "Analytics pipeline for customer data processing",
  "version": 0.1,
  "updatedAt": 1769982668397,
  "updatedBy": "admin",
  "sourceUrl": "http://localhost:8080/tree?dag_id=dbt_analytics_customers",
  "service": {
    "id": "daa58a49-df05-48a3-a417-45dfd12eacf5",
    "type": "pipelineService",
    "name": "sample_airflow",
    "fullyQualifiedName": "sample_airflow",
    "deleted": false
  },
  "serviceType": "Airflow",
  "tasks": [
    {
      "name": "extract_customers",
      "taskType": "BatchTask",
      "description": "Extract customer data from source",
      "downstreamTasks": ["transform_customers"]
    },
    {
      "name": "transform_customers",
      "taskType": "BatchTask",
      "description": "Transform and clean customer data",
      "downstreamTasks": ["load_customers"]
    },
    {
      "name": "load_customers",
      "taskType": "BatchTask",
      "description": "Load transformed data into warehouse"
    }
  ],
  "href": "http://localhost:8585/api/v1/pipelines/538faa63-d204-46ff-aead-d158d0401cac",
  "deleted": false,
  "owners": [],
  "tags": [],
  "followers": [],
  "votes": {
    "upVotes": 0,
    "downVotes": 0
  },
  "domains": []
}

Create a Pipeline

Create a new pipeline within a pipeline service.

Body Parameters

name · string · required

Name of the pipeline. Must be unique within the parent pipeline service.

service · string · required

Fully qualified name of the parent PipelineService (e.g., sample_airflow).

displayName · string

Human-readable display name for the pipeline.

description · string

Description of the pipeline in Markdown format.

sourceUrl · string

URL to the pipeline definition in the source system (e.g., Airflow DAG URL).

tasks · array

Array of pipeline tasks representing individual steps in the pipeline.

Show properties

name · string

Name of the task.

displayUrl · string

URL to display for this task.

taskUrl · string

URL to the task in the source system.

taskType · string

Type of the task (e.g., BatchTask, StreamTask).

description · string

Description of the task.

downstreamTasks · array

Array of task names that depend on this task.

owners · array

Array of owner references (users or teams) to assign to the pipeline.

Show properties

id · string

UUID of the owner entity.

type · string

Type of owner entity (e.g., user, team).

name · string

Name of the owner entity.

domain · string

Fully qualified name of the domain to assign for governance purposes.

tags · array

Array of classification tags to apply to the pipeline.

Show properties

tagFQN · string · required

Fully qualified name of the tag.

labelType · string

Type of label (e.g., Manual, Derived, Propagated).

state · string

State of the tag (e.g., Suggested, Confirmed).

extension · object

Custom property values defined by your organization’s metadata schema.

POST /v1/pipelines

Python

from metadata.sdk import configure
from metadata.sdk.entities import Pipelines
from metadata.generated.schema.api.data.createPipeline import CreatePipelineRequest

configure(
    host="https://your-company.open-metadata.org/api",
    jwt_token="your-jwt-token"
)

request = CreatePipelineRequest(
    name="dbt_analytics_customers",
    displayName="DBT Customer Analytics",
    service="sample_airflow",
    description="Analytics pipeline for customer data processing",
    sourceUrl="http://localhost:8080/tree?dag_id=dbt_analytics_customers",
    tasks=[
        {
            "name": "extract_customers",
            "taskType": "BatchTask",
            "description": "Extract customer data from source",
            "downstreamTasks": ["transform_customers"]
        },
        {
            "name": "transform_customers",
            "taskType": "BatchTask",
            "description": "Transform and clean customer data",
            "downstreamTasks": ["load_customers"]
        },
        {
            "name": "load_customers",
            "taskType": "BatchTask",
            "description": "Load transformed data into warehouse"
        }
    ]
)

pipeline = Pipelines.create(request)
print(f"Created: {pipeline.fullyQualifiedName}")

Response

{
  "id": "538faa63-d204-46ff-aead-d158d0401cac",
  "name": "dbt_analytics_customers",
  "displayName": "DBT Customer Analytics",
  "fullyQualifiedName": "sample_airflow.dbt_analytics_customers",
  "description": "Analytics pipeline for customer data processing",
  "version": 0.1,
  "updatedAt": 1769982668397,
  "updatedBy": "admin",
  "sourceUrl": "http://localhost:8080/tree?dag_id=dbt_analytics_customers",
  "service": {
    "id": "daa58a49-df05-48a3-a417-45dfd12eacf5",
    "type": "pipelineService",
    "name": "sample_airflow",
    "fullyQualifiedName": "sample_airflow",
    "deleted": false
  },
  "serviceType": "Airflow",
  "tasks": [
    {
      "name": "extract_customers",
      "taskType": "BatchTask",
      "description": "Extract customer data from source",
      "downstreamTasks": ["transform_customers"]
    },
    {
      "name": "transform_customers",
      "taskType": "BatchTask",
      "description": "Transform and clean customer data",
      "downstreamTasks": ["load_customers"]
    },
    {
      "name": "load_customers",
      "taskType": "BatchTask",
      "description": "Load transformed data into warehouse"
    }
  ],
  "href": "http://localhost:8585/api/v1/pipelines/538faa63-d204-46ff-aead-d158d0401cac",
  "deleted": false,
  "owners": [],
  "tags": [],
  "followers": [],
  "votes": {
    "upVotes": 0,
    "downVotes": 0
  },
  "domains": []
}

---

Returns

Returns the created pipeline object with all specified properties and system-generated fields.

Response

id · string

Unique identifier for the pipeline (UUID format).

name · string

Pipeline name.

fullyQualifiedName · string

Fully qualified name in format service.pipelineName.

displayName · string

Human-readable display name.

description · string

Description of the pipeline in Markdown format.

sourceUrl · string

URL to the pipeline in the source system.

service · object

Reference to the parent pipeline service.

Show properties

id · string

UUID of the pipeline service.

type · string

Type of entity (always pipelineService).

name · string

Name of the pipeline service.

fullyQualifiedName · string

Fully qualified name of the pipeline service.

serviceType · string

Type of pipeline service (e.g., Airflow, Dagster, DBTCloud).

tasks · array

List of tasks in the pipeline.

Show properties

name · string

Name of the task.

taskType · string

Type of the task.

description · string

Description of the task.

downstreamTasks · array

Names of downstream dependent tasks.

owners · array

List of owners assigned to the pipeline.

Show properties

id · string

UUID of the owner entity.

type · string

Type of owner entity (e.g., user, team).

name · string

Name of the owner entity.

domains · array

Domain assignments for governance.

tags · array

Classification tags applied to the pipeline.

Show properties

tagFQN · string

Fully qualified name of the tag.

labelType · string

Type of label (e.g., Manual, Derived, Propagated).

state · string

State of the tag (e.g., Suggested, Confirmed).

extension · object

Custom property values defined by your organization’s metadata schema.

version · number

Version number for the entity (starts at 0.1).

---

Create or Update (PUT)

Use PUT /v1/pipelines instead of POST to perform an upsert. If a pipeline with the same fullyQualifiedName already exists, it will be updated; otherwise, a new pipeline is created. The request body is the same as POST.

curl -X PUT "{base_url}/api/v1/pipelines" \
  -H "Authorization: Bearer {access_token}" \
  -H "Content-Type: application/json" \
  -d '{ ... same body as POST ... }'

PUT will not return a 409 conflict error if the entity already exists — it will update the existing entity instead.

---

Bulk Create or Update (PUT)

Use PUT /v1/pipelines/bulk to create or update multiple pipelines in a single request. The request body is an array of create request objects.

curl -X PUT "{base_url}/api/v1/pipelines/bulk" \
  -H "Authorization: Bearer {access_token}" \
  -H "Content-Type: application/json" \
  -d '[
    { "name": "pipeline_one", "service": "sample_airflow" },
    { "name": "pipeline_two", "service": "sample_airflow" }
  ]'

---

Error Handling

Code	Error Type	Description
400	BAD_REQUEST	Invalid request body or missing required fields
401	UNAUTHORIZED	Invalid or missing authentication token
403	FORBIDDEN	User lacks permission to create pipelines
409	ENTITY_ALREADY_EXISTS	Pipeline with same name already exists in service (POST only)