Prefect - The Python Orchestrator

If you are in data engineering, you will have encoutered Airflow, an orchestration tool for ETL process based on the DAG architecture. A powerful alternative that is purely pythonic exists and it is Prefect. On this series, we focus on small snippets to get us started.

Setup and Installation

To begin we need to install prefect. This is rather straight forward with pip. You can use other tools like UV as well

pip install -U prefect

Let's now check if it is installed

import prefect
from prefect import get_client

print(f"Running Prefect {prefect.__version__}")

OUTPUTRunning Prefect 3.6.9

Prefect CLI

Once prefect is installed, it, also includes a command line interface that you can leverage for documentation, starting up servers and building your pipelines. For example, below we see how to get the same prefect profile information from the command line.

$ prefect version

OUTPUTVersion: 3.6.9 API version: 0.8.4 Python version: 3.13.5 Git commit: d5a15f55 Built: Wed, Dec 31, 2025 10:18 PM OS/Arch: darwin/arm64 Profile: ephemeral Server type: ephemeral Pydantic version: 2.11.7 Server: Database: sqlite SQLite version: 3.50.2

Prefect help

The --help flag returns a number of useful commands that are shipped with prefect.

$ prefect --help

OUTPUTUsage: prefect [OPTIONS] COMMAND [ARGS]... ╭─ Options ────────────────────────────────────────────────────────────────────╮ │ --version -v Display the current version. │ │ --profile -p TEXT Select a profile for this CLI │ │ run. │ │ --prompt --no-prompt Force toggle prompts for this │ │ CLI run. │ │ [default: (from │ │ PREFECT_CLI_PROMPT)] │ │ --install-completion Install completion for the │ │ current shell. │ │ --show-completion Show completion for the │ │ current shell, to copy it or │ │ customize the installation. │ │ --help Show this message and exit. │ ╰──────────────────────────────────────────────────────────────────────────────╯ ╭─ Commands ───────────────────────────────────────────────────────────────────╮ │ version Get the current Prefect version and integration │ │ information. │ │ api Make a direct request to the Prefect API. │ │ init │ │ deploy │ │ transfer Transfer resources from one Prefect profile to │ ... │ task-run View and inspect task runs. │ │ automation Manage automations. │ ╰──────────────────────────────────────────────────────────────────────────────╯

Checking Profiles

Another useful utility function is the profiles function, which allows you to identify and switch between different servers. For now, we have not setup any profiles but it is a handy tool to know.

$ prefect profile ls

OUTPUT┏━━━━━━━━━━━━━━━━━━━━━┓ ┃ Available Profiles: ┃ ┡━━━━━━━━━━━━━━━━━━━━━┩ └─────────────────────┘ * active profile

Understanding Your Configuration

config view

!prefect config view

OUTPUTPREFECT_PROFILE='ephemeral' PREFECT_SERVER_ALLOW_EPHEMERAL_MODE='true' (from profile)

This shows you're using the ephemeral profile, which means Prefect is running with an in-memory, temporary API server. This is great for quick development and testing, but data won't persist between sessions. The PREFECT_SERVER_ALLOW_EPHEMERAL_MODE setting enables this mode

profile inspect

This confirms the active profile is ephemeral and shows the specific settings applied. If you don't specify a profile name, Prefect defaults to 'ephemeral'. When you start a local Prefect server or connect to Prefect Cloud, you'll create and switch to different profiles with different API endpoints and settings.

$ prefect profile inspect

OUTPUTNo name provided, defaulting to 'ephemeral' PREFECT_SERVER_ALLOW_EPHEMERAL_MODE='true'