Prefect Server

In this tutorial, we'll explore the Prefect Server in detail - understanding what it does, how to run it, configure it, and interact with it programmatically.

What is the Prefect Server

The Prefect Server is a persistent API backend that tracks and manages your workflow orchestration. Unlike the ephemeral mode, which runs in-process and disappears when you your python Script ends, the Prefect Server:

  • Persists flow and task execution history in a database
  • Provides a web UI for monitoring and management
  • Enables scheduling and deployment of flows
  • Tracks work queues and work pools
  • Stores artifacts, logs, and results
  • Manages concurrent flow runs and task execution

Architecture Overview

The Prefect Server consists of several components:

OUTPUT┌─────────────────────────────────────────────┐ │ Prefect Server (Port 4200) │ ├─────────────────────────────────────────────┤ │ Web UI (Dashboard) │ │ REST API (/api) │ │ WebSocket (Real-time updates) │ └─────────────────┬───────────────────────────┘ │ ▼ ┌────────────────┐ │ Database │ │ (SQLite or │ │ PostgreSQL) │ └────────────────┘

Starting the Prefect Server

To start the Prefect Server with default setting, use the following commmands:

$ prefect server start
OUTPUT ___ ___ ___ ___ ___ ___ _____ | _ \ _ \ __| __| __/ __|_ _| | _/ / _|| _|| _| (__ | | |_| |_|_\___|_| |___\___| |_| Configure Prefect to communicate with the server with: prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api View the API reference documentation at http://127.0.0.1:4200/docs Check out the dashboard at http://127.0.0.1:4200

Configurating Server with Host and Port of your choosing

You can also customize the server start documentation by defining arguments such as port number, setting up analytics. The example below, we start a server with ort 9000 and we set analytics off.

prefect server start --host 0.0.0.0 --port 9000 --analytics-off
OUTPUT ___ ___ ___ ___ ___ ___ _____ | _ \ _ \ __| __| __/ __|_ _| | _/ / _|| _|| _| (__ | | |_| |_|_\___|_| |___\___| |_| Configure Prefect to communicate with the server with: prefect config set PREFECT_API_URL=http://0.0.0.0:9000/api View the API reference documentation at http://0.0.0.0:9000/docs Check out the dashboard at http://0.0.0.0:9000

Running Server in the Background - Linux/Mac Users

If you are using a mac/linux system, you can run the server in the background using the following code implementation.

$ nohup prefect server start > prefect-server.log 2>&1 &

You can check if the process is running using the following code. If it is running, you may see an output like below

$ ps aux | grep prefect
OUTPUTsifael 81057 0.0 0.0 435299760 1520 s000 S+ 7:50PM 0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox --exclude-dir=.venv --exclude-dir=venv prefect

We can also check the logs.

$ tail -f prefect-server.log
OUTPUT prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api View the API reference documentation at http://127.0.0.1:4200/docs Check out the dashboard at http://127.0.0.1:4200 Server stopped!