{"id":2165,"date":"2026-07-10T17:00:00","date_gmt":"2026-07-10T17:00:00","guid":{"rendered":"https:\/\/futurenews24.com\/index.php\/2026\/07\/10\/i-built-my-second-etl-pipeline-this-time-i-started-thinking-like-a-data-engineer\/"},"modified":"2026-07-11T04:59:10","modified_gmt":"2026-07-11T04:59:10","slug":"i-built-my-second-etl-pipeline-this-time-i-started-thinking-like-a-data-engineer","status":"publish","type":"post","link":"https:\/\/futurenews24.com\/index.php\/2026\/07\/10\/i-built-my-second-etl-pipeline-this-time-i-started-thinking-like-a-data-engineer\/","title":{"rendered":"I Constructed My Second ETL Pipeline. This Time, I Began Pondering Like a Knowledge Engineer"},"content":{"rendered":"
\n<\/p>\n
\n
, I made a decision I wished to transition from knowledge analyst to knowledge engineer.<\/p>\n
Like many individuals beginning out, I used to be overwhelmed by the sheer variety of issues I assumed I wanted to study. Knowledge warehouses, orchestration instruments, distributed processing, streaming programs, cloud platforms, infrastructure. The listing appeared limitless.<\/p>\n
As a substitute of attempting to study all the pieces without delay, I took a unique strategy.<\/p>\n
I created a 12-month self-study roadmap constructed round one easy concept: study by constructing.<\/p>\n
Somewhat than leaping from one tutorial to a different, I might construct a sequence of small tasks. Every venture would introduce just a few new ideas whereas reinforcing what I had realized earlier than. The aim wasn\u2019t to construct probably the most refined purposes doable. It was to develop the behavior of considering like an engineer by fixing one downside at a time.<\/p>\n
The primary venture in that journey was a GitHub ETL pipeline. It started as a easy Python script that fetched repository knowledge and exported it to a CSV file. As I realized extra, I steadily improved it. I changed CSV recordsdata with SQLite, made the pipeline idempotent to stop duplicate knowledge, and finally automated it with GitHub Actions.<\/p>\n
By the point I completed, I noticed one thing that hadn\u2019t been apparent once I began.<\/p>\n
Constructing the ETL logic was truly the straightforward half.<\/p>\n
The more durable questions appeared as soon as I finished fascinated about a script that runs as soon as and began fascinated about a system that should run over and over with out me watching it.<\/p>\n
How ought to it’s scheduled?<\/p>\n
What occurs if it fails midway by?<\/p>\n
The place ought to retry logic reside?<\/p>\n
How do you package deal the appliance so it runs the identical method in every single place?<\/p>\n
These questions taught me much more about knowledge engineering than parsing JSON or writing SQL ever did.<\/p>\n
That first venture left me with one other problem.<\/p>\n
GitHub Actions labored properly for automating a small ETL pipeline, however I wished to grasp what adjustments if you use a workflow orchestrator constructed particularly for knowledge engineering. I wished to find out how engineers separate orchestration from execution, how containerized workloads match into that image, and what a production-minded pipeline truly seems like, even on a small scale.<\/p>\n
So for my second venture, I made a decision to construct an automatic RSS ingestion pipeline.<\/p>\n
On the floor, it\u2019s a reasonably easy utility. It fetches articles from an RSS feed, parses them into structured objects, and shops them in PostgreSQL.<\/p>\n
However the aim was by no means to construct an RSS reader.<\/p>\n
The aim was to discover the engineering choices that rework a Python script right into a dependable knowledge pipeline.<\/p>\n
On this article, I\u2019ll stroll by these choices, the errors I made alongside the best way, and the teachings I realized constructing my first pipeline with Kestra.<\/p>\n
Why Construct One other ETL Pipeline?<\/h2>\n
After ending my first ETL venture, I thought of transferring on to one thing fully completely different.<\/p>\n
Possibly an information warehouse venture. Possibly Apache Spark. Possibly an API with a extra complicated transformation layer.<\/p>\n
As a substitute, I constructed\u2026 one other ETL pipeline.<\/p>\n
At first, that in all probability appears like a step backwards.<\/p>\n
In spite of everything, I\u2019d already constructed an extraction pipeline, made it idempotent, and scheduled it with GitHub Actions. Why repeat the identical train?<\/p>\n
As a result of I wasn\u2019t attempting to study a brand new dataset. I used to be attempting to study a brand new mind-set.<\/p>\n
One lesson from my first venture caught with me.<\/p>\n
Writing the ETL logic wasn\u2019t the troublesome half. The troublesome half was all the pieces surrounding it.<\/p>\n
How ought to the pipeline be executed?<\/p>\n
How ought to it get better from failures?<\/p>\n
How do you package deal it so it runs constantly on any machine?<\/p>\n
The place does scheduling belong?<\/p>\n
And maybe the most important query of all:<\/p>\n
\n
The place ought to the obligations of the appliance finish, and the place ought to the obligations of the orchestration layer start?<\/p>\n<\/blockquote>\n
These questions don\u2019t have a lot to do with RSS feeds or GitHub repositories. They\u2019re engineering questions, and I noticed I might discover them with nearly any knowledge supply.<\/p>\n
That\u2019s why I selected an RSS feed.<\/p>\n
Not as a result of RSS is especially thrilling, however as a result of it\u2019s deliberately easy.<\/p>\n
The extraction logic solely takes just a few traces of Python. That meant I might spend much less time worrying about enterprise logic and extra time fascinated about structure.<\/p>\n
For a similar motive, I made a decision to maneuver away from GitHub Actions for this venture.<\/p>\n
GitHub Actions was an awesome introduction to scheduling. It confirmed me the best way to automate a workflow and gave me my first style of working an ETL pipeline with out guide intervention.<\/p>\n
However GitHub Actions isn\u2019t designed particularly for orchestrating knowledge workflows.<\/p>\n
I wished to grasp what adjustments if you use a software that\u2019s constructed with knowledge pipelines in thoughts.<\/p>\n
That\u2019s what led me to Kestra.<\/p>\n
Somewhat than asking, \u201cHow do I run this Python script each hour?\u201d, I discovered myself asking a unique set of questions.<\/p>\n
How ought to retries be configured?<\/p>\n
How are workflow executions tracked?<\/p>\n
How ought to atmosphere variables be handed right into a container?<\/p>\n
What does a failed execution appear like?<\/p>\n
How do you separate the appliance from the infrastructure that runs it?<\/p>\n
These questions have been precisely what I wished to discover.<\/p>\n
By selecting a easy ETL pipeline, I might deal with the engineering choices as an alternative of getting distracted by difficult enterprise logic.<\/p>\n
Wanting again, I believe that was the correct determination.<\/p>\n
This venture isn\u2019t fascinating as a result of it processes RSS feeds.<\/p>\n
It\u2019s fascinating as a result of constructing it compelled me to consider reliability, repeatability, and orchestration in a method my first venture by no means did.<\/p>\n
The First Architectural Determination: Docker Earlier than Kestra<\/h2>\n
With the venture outlined, my first intuition was to leap straight into Kestra.<\/p>\n
In spite of everything, orchestration was one of many essential causes I selected this venture. Why not begin there?<\/p>\n
As a substitute, I did one thing that turned out to avoid wasting me loads of frustration later.<\/p>\n
I ignored Kestra fully.<\/p>\n
Which may sound counterintuitive, however I wished to keep away from introducing a number of transferring elements earlier than I knew the core utility truly labored.<\/p>\n
So I constructed the venture in layers.<\/p>\n
First, I wrote the Python ETL.<\/p>\n
Its job was deliberately small. Fetch the RSS feed, parse every entry into an Article object, and save the outcomes to PostgreSQL. Nothing extra.<\/p>\n
feed = feedparser.parse(RSS_URL)<\/p>\n
articles = parse_feed(feed)<\/p>\n
save_articles(articles)<\/p>\n
That\u2019s actually all of the ETL did. I deliberately saved the appliance small as a result of the main focus of this venture wasn\u2019t the transformation logic. It was all the pieces that occurred round it.<\/p>\n
As soon as that labored reliably, I turned my consideration to the database.<\/p>\n
I wished repeated executions to be protected, so I made the inserts idempotent utilizing PostgreSQL\u2019s ON CONFLICT DO NOTHING. That method, working the pipeline a number of occasions wouldn\u2019t create duplicate rows.<\/p>\n
INSERT INTO articles (…) \nVALUES (…) \nON CONFLICT (id) DO NOTHING;<\/p>\n
This one line made the pipeline protected to execute repeatedly. Whether or not Kestra ran the workflow as soon as or 100 occasions, PostgreSQL grew to become answerable for stopping duplicate data.<\/p>\n
Solely after the ETL and database labored collectively did I introduce Docker.<\/p>\n
That call modified how I assumed in regards to the venture.<\/p>\n
Initially, Docker felt like one other software I wanted to study.<\/p>\n
By the top of the venture, I noticed it had turn out to be one thing far more essential.<\/p>\n
It grew to become the unit of execution.<\/p>\n
FROM python:3.13-slim<\/p>\n
WORKDIR \/app<\/p>\n
COPY necessities.txt . \nRUN pip set up –no-cache-dir -r necessities.txt<\/p>\n
COPY . .<\/p>\n
CMD [“python”, “fetch_rss.py”]<\/p>\n
Packaging the ETL this manner meant the appliance grew to become self-contained. As a substitute of asking Kestra to grasp my Python venture, I might merely ask it to run a container that already knew the best way to execute the pipeline.<\/p>\n
As a substitute of considering, \u201cKestra will run my Python script,\u201d I began considering, \u201cKestra will run my Docker picture.\u201d<\/p>\n
That distinction might sound delicate, nevertheless it fully adjustments the connection between your utility and your orchestration layer.<\/p>\n
As soon as the ETL was packaged right into a container, it not mattered whether or not it ran on my laptop computer, inside Kestra, or on one other machine solely.<\/p>\n
The runtime atmosphere was all the time the identical.<\/p>\n
That consistency gave me one thing I didn\u2019t have earlier than: confidence.<\/p>\n
Earlier than introducing Kestra, I might run the container manually and confirm that it fetched the RSS feed, linked to PostgreSQL, and endured the anticipated data.<\/p>\n
If one thing failed, I knew the issue wasn\u2019t hidden behind one other layer of orchestration.<\/p>\n
That validation step turned out to be extremely helpful later.<\/p>\n
Throughout improvement, I bumped into points with networking, container configuration, and workflow execution. As a result of the Docker picture had already been validated independently, I might instantly rule out the ETL itself and deal with the orchestration layer.<\/p>\n
That made debugging dramatically simpler.<\/p>\n
Wanting again, I believe this was one of many largest classes from the venture.<\/p>\n
It\u2019s tempting to attach each element collectively as shortly as doable and hope all the pieces works.<\/p>\n
A greater strategy is to validate every layer earlier than introducing the following one.<\/p>\n
On this venture, the order seemed like this:<\/p>\n
Validate the Python ETL.<\/p>\n
Validate PostgreSQL persistence.<\/p>\n
Validate the Docker picture.<\/p>\n
Lastly, let Kestra orchestrate a container that I already trusted.<\/p>\n
Every layer constructed on the earlier one.<\/p>\n
By the point Kestra entered the image, I wasn\u2019t attempting to debug Python, PostgreSQL, Docker, and orchestration on the similar time.<\/p>\n
I used to be solely fixing one downside.<\/p>\n
That incremental strategy made all the venture really feel far more manageable, and it\u2019s a workflow I\u2019ll in all probability proceed utilizing on future knowledge engineering tasks.<\/p>\n
The Assumption That Turned Out to Be Unsuitable<\/h2>\n
With a working Docker picture, I used to be satisfied the laborious half was over.<\/p>\n
I had a Python ETL that labored.<\/p>\n
I had PostgreSQL working in Docker.<\/p>\n
I had verified that the container might fetch RSS articles and save them to the database.<\/p>\n
Now all Kestra needed to do was run it.<\/p>\n
Or so I assumed.<\/p>\n
My authentic assumption was easy.<\/p>\n
Kestra would level to my Python recordsdata, execute the script, and all the pieces would work precisely because it had from the command line.<\/p>\n
It didn\u2019t.<\/p>\n
I shortly found that there was an essential distinction I hadn\u2019t absolutely appreciated.<\/p>\n
Kestra is an orchestrator.<\/p>\n
It isn\u2019t answerable for constructing Python environments or managing utility dependencies. Its duty is deciding when and the way workloads ought to run.<\/p>\n
That realization modified the course of the venture.<\/p>\n
As a substitute of treating Kestra as one other place to execute Python code, I began treating it because the layer answerable for orchestrating a workload that already existed.<\/p>\n
That workload was my Docker picture.<\/p>\n
As soon as I made that psychological shift, the structure grew to become a lot cleaner.<\/p>\n
The ETL grew to become a self-contained utility.<\/p>\n
Docker grew to become the deployment artifact.<\/p>\n
Kestra grew to become the orchestrator.<\/p>\n<\/figure>\n
Every layer had a transparent duty, and none of them wanted to know the way the others labored internally.<\/p>\n
Apparently, reaching that time wasn\u2019t fully easy.<\/p>\n
My first intuition was to discover a method for Kestra to execute the Python venture straight. That strategy sounded less complicated, however the extra I experimented with it, the extra I noticed I used to be asking the orchestrator to take duty for one thing the appliance ought to already present.<\/p>\n
As soon as I embraced Docker because the execution artifact, the workflow grew to become a lot less complicated.<\/p>\n
Some approaches seemed promising at first however launched pointless complexity. Others labored, however didn\u2019t align with how Kestra encourages containerized workloads to be executed.<\/p>\n
Ultimately, I landed on a workflow that felt surprisingly easy.<\/p>\n
As a substitute of attempting to show Kestra the best way to run Python, I let Kestra do what it does finest.<\/p>\n
Wanting on the workflow now, what stands out isn\u2019t how a lot YAML it accommodates. It\u2019s how little Kestra truly must learn about my utility. Its solely duty is to launch a container that already is aware of the best way to execute the ETL.<\/p>\n
Inside that container, my utility already is aware of precisely what to do.<\/p>\n
That small architectural change solved extra than simply the rapid execution downside.<\/p>\n
It additionally strengthened an concept that\u2019s changing into a recurring theme in my studying journey.<\/p>\n
Good engineering usually isn\u2019t about including one other layer.<\/p>\n
It\u2019s about giving every layer a single duty and permitting it to try this job properly.<\/p>\n
Python shouldn\u2019t fear about orchestration.<\/p>\n
Kestra shouldn\u2019t fear about Python dependencies.<\/p>\n
Docker shouldn\u2019t know something about RSS feeds.<\/p>\n
Every element solves a unique downside.<\/p>\n
As soon as I finished asking one software to unravel each downside, all the system grew to become a lot simpler to motive about.<\/p>\n
Wanting again, this was in all probability the most important mindset shift in the entire venture.<\/p>\n
I didn\u2019t simply discover ways to use Kestra.<\/p>\n
I realized what orchestration truly means.<\/p>\n
As soon as a Pipeline Runs Mechanically, Every little thing Modifications<\/h2>\n
Up till this level, I had been working the pipeline manually.<\/p>\n
If one thing failed, I used to be sitting in entrance of my pc. I might learn the error, make a change, and check out once more.<\/p>\n
That security web disappears the second a pipeline begins working by itself.<\/p>\n
One of many first issues I configured in Kestra was a easy hourly schedule.<\/p>\n
In apply, it represented a a lot greater shift.<\/p>\n
The pipeline not trusted me remembering to run it.<\/p>\n
Each hour, Kestra would begin a brand new execution, launch the ETL container, and course of the newest articles from the RSS feed.<\/p>\n
That instantly raised one other query.<\/p>\n
What occurs if a kind of executions fails?<\/p>\n
Throughout improvement, I intentionally launched failures to reply that query. I pointed the pipeline at an invalid database host and watched what occurred.<\/p>\n
The primary execution failed, precisely as anticipated.<\/p>\n
Extra importantly, it didn\u2019t cease there.<\/p>\n
As a result of the workflow was configured with retries, Kestra routinely tried the execution once more after a brief delay. <\/p>\n
print(f”Feed title: {feed.feed.title}”) \nprint(f”Fetched articles: {len(articles)}”) \nprint(f”Saved {len(articles)} articles to the database.”)<\/p>\n
print(“=== RSS PIPELINE END ===”)<\/p>\n
Every run now tells a easy story.<\/p>\n
=== RSS PIPELINE START ===<\/p>\n
First entry preview: \nTitle: Christian Ledermann: Migrate From mypy To ty And pyrefly \nHyperlink: https:\/\/dev.to\/…<\/p>\n
Feed title: Planet Python \nFetched articles: 25 \nSaved 25 articles to the database.<\/p>\n
=== RSS PIPELINE END ===<\/p>\n
These adjustments didn\u2019t make the appliance smarter. They made it simpler to grasp. And that\u2019s an essential distinction.<\/p>\n
Good observability isn\u2019t about producing extra logs. It\u2019s about producing the correct logs.<\/p>\n
By the top of the venture, I noticed one thing fascinating. The Python code hadn\u2019t grown dramatically. Many of the work had occurred round it.<\/p>\n
Wanting again, a lot of the engineering effort wasn\u2019t spent making the ETL smarter. It was spent making it extra dependable.<\/p>\n
Scheduling ensured it ran with out me.<\/p>\n
Retries helped it get better from transient failures.<\/p>\n
Idempotency protected the database from duplicate writes.<\/p>\n
Logging made each execution simpler to grasp.<\/p>\n
Individually, none of those adjustments have been significantly complicated. Collectively, they reworked a easy Python script into one thing that behaved far more like a manufacturing system.<\/p>\n
The Last Structure<\/h2>\n
By the top of the venture, the structure had settled into one thing that felt surprisingly easy.<\/p>\n<\/figure>\n
Wanting on the last structure, it\u2019s tempting to suppose the venture was all the time heading on this course.<\/p>\n
It wasn\u2019t.<\/p>\n
Every layer was added solely after the earlier one had been validated.<\/p>\n
The ETL got here first.<\/p>\n
Then PostgreSQL.<\/p>\n
Then Docker.<\/p>\n
Lastly, Kestra.<\/p>\n
That order mattered.<\/p>\n
As a result of each element had already been examined independently, I by no means discovered myself debugging Python, Docker, PostgreSQL, and Kestra on the similar time. Every determination lowered the variety of unknowns as an alternative of accelerating them.<\/p>\n
Extra importantly, each element ended up with a transparent duty.<\/p>\n
Python is aware of the best way to fetch, parse, and persist RSS articles.<\/p>\n
PostgreSQL is aware of the best way to retailer knowledge safely and stop duplicates.<\/p>\n
Docker supplies a constant execution atmosphere.<\/p>\n
Kestra decides when the workload ought to run and what ought to occur if it fails.<\/p>\n
None of these parts are attempting to do one another\u2019s jobs.Mockingly, that\u2019s what made the completed system really feel a lot less complicated than I anticipated.<\/p>\n
What This Mission Modified In regards to the Method I Suppose<\/h2>\n
Once I began studying knowledge engineering, I assumed the troublesome half could be writing ETL code.<\/p>\n
That\u2019s what most newbie tutorials deal with.<\/p>\n
You discover ways to name an API.<\/p>\n
You rework the info.<\/p>\n
You reserve it someplace.<\/p>\n
Repeat.<\/p>\n
These are helpful abilities, however they\u2019re just one a part of the image.This venture taught me that the engineering begins after the script works.<\/p>\n
As soon as a pipeline is anticipated to run each hour, survive transient failures, keep away from duplicate knowledge, and produce logs that designate what occurred, the questions turn out to be far more fascinating.<\/p>\n
You\u2019re not fascinated about particular person capabilities. You\u2019re fascinated about programs.<\/p>\n
One of many largest mindset shifts for me was understanding the distinction between execution and orchestration.<\/p>\n
At first, these concepts felt nearly interchangeable. Now they really feel fully separate.<\/p>\n
The Python utility ought to deal with enterprise logic. The orchestrator ought to deal with when, the place, and the way that utility runs.<\/p>\n
Protecting these obligations separate made all the venture simpler to motive about.<\/p>\n
It additionally modified how I take into consideration Docker.<\/p>\n
Earlier than this venture, I noticed Docker primarily as a solution to package deal purposes.<\/p>\n
Now I see it as a deployment artifact.<\/p>\n
As soon as the ETL had been packaged right into a container and validated independently, I might cease worrying about whether or not it will behave in a different way inside Kestra.<\/p>\n
That confidence turned out to be one of many largest benefits of containerizing the appliance.<\/p>\n
Maybe an important lesson, although, had nothing to do with Kestra or Docker. It was the worth of constructing incrementally. <\/p>\n
Each main determination adopted the identical sample.<\/p>\n
Construct the smallest factor that works.<\/p>\n
Validate it.<\/p>\n
Solely then introduce the following layer.<\/p>\n
That strategy made the venture really feel a lot much less overwhelming than attempting to attach all the pieces collectively from the start.Wanting again, I believe that\u2019s a lesson I\u2019ll carry into each future venture, whatever the know-how.<\/p>\n
Wanting Forward<\/h2>\n
This RSS pipeline is just the second venture in my knowledge engineering studying journey. In comparison with my first ETL pipeline, the code itself isn\u2019t dramatically extra complicated. What modified was the best way I approached the issue.<\/p>\n
As a substitute of asking, \u201cHow do I write this script?\u201d, I discovered myself asking questions like:<\/p>\n
The place ought to this duty reside?<\/p>\n
What occurs if it fails?<\/p>\n
Can I run it repeatedly with out worrying about duplicate knowledge<\/p>\n
Can I belief it to run once I\u2019m not watching?<\/p>\n
These questions pushed me to suppose much less like somebody writing Python code and extra like somebody designing a system. And I think that\u2019s the actual worth of constructing tasks.<\/p>\n
Each venture teaches a brand new software. However the most effective tasks slowly change the best way you suppose. This one definitely did.<\/p>\n
I\u2019m positive the following venture will problem a totally completely different set of assumptions. Truthfully, I\u2019m wanting ahead to discovering out what they’re.<\/p>\n
That is a part of my ongoing sequence documenting my transition from programs analyst to knowledge engineer. In case you\u2019ve been following alongside, thanks.<\/p>\n
Join with me on\u00a0LinkedIn,\u00a0YouTube, and\u00a0Twitter.<\/p>\n<\/div>\n
<\/figure>\n
<\/figure>\n