Back to articles
How to Create a Pipeline with Dotflow in Python
How-ToDevOps

How to Create a Pipeline with Dotflow in Python

via Dev.toFernando Celmer

In this tutorial, you'll learn how to build a complete data pipeline using Dotflow — a lightweight Python library that requires zero infrastructure. No Redis. No RabbitMQ. No Postgres. No Docker. Just pip install dotflow . What we'll build A pipeline that: Extracts user data from a source Transforms it by filtering active users and calculating stats Loads the results into storage Along the way, we'll add retry with backoff, parallel execution, checkpoint/resume, and cron scheduling. Step 1 — Install Dotflow pip install dotflow Step 2 — Create your first pipeline Create a file called pipeline.py : from dotflow import DotFlow , action @action def extract (): """ Simulate extracting data from a database or API. """ return { " users " : [ { " name " : " Alice " , " age " : 30 , " active " : True }, { " name " : " Bob " , " age " : 25 , " active " : False }, { " name " : " Charlie " , " age " : 35 , " active " : True }, { " name " : " Diana " , " age " : 28 , " active " : True }, ] } @actio

Continue reading on Dev.to

Opens in a new tab

Read Full Article
2 views

Related Articles