, I made a decision I wished to transition from knowledge analyst to knowledge engineer.
Like many individuals beginning out, I used to be overwhelmed by the sheer variety of issues I assumed I wanted to study. Knowledge warehouses, orchestration instruments, distributed processing, streaming programs, cloud platforms, infrastructure. The listing appeared limitless.
As a substitute of attempting to study all the pieces without delay, I took a unique strategy.
I created a 12-month self-study roadmap constructed round one easy concept: study by constructing.
Somewhat than leaping from one tutorial to a different, I might construct a sequence of small tasks. Every venture would introduce just a few new ideas whereas reinforcing what I had realized earlier than. The aim wasn’t to construct probably the most refined purposes doable. It was to develop the behavior of considering like an engineer by fixing one downside at a time.
The primary venture in that journey was a GitHub ETL pipeline. It started as a easy Python script that fetched repository knowledge and exported it to a CSV file. As I realized extra, I steadily improved it. I changed CSV recordsdata with SQLite, made the pipeline idempotent to stop duplicate knowledge, and finally automated it with GitHub Actions.
By the point I completed, I noticed one thing that hadn’t been apparent once I began.
Constructing the ETL logic was truly the straightforward half.
The more durable questions appeared as soon as I finished fascinated about a script that runs as soon as and began fascinated about a system that should run over and over with out me watching it.
How ought to it’s scheduled?
What occurs if it fails midway by?
The place ought to retry logic reside?
How do you package deal the appliance so it runs the identical method in every single place?
These questions taught me much more about knowledge engineering than parsing JSON or writing SQL ever did.
That first venture left me with one other problem.
GitHub Actions labored properly for automating a small ETL pipeline, however I wished to grasp what adjustments if you use a workflow orchestrator constructed particularly for knowledge engineering. I wished to find out how engineers separate orchestration from execution, how containerized workloads match into that image, and what a production-minded pipeline truly seems like, even on a small scale.
So for my second venture, I made a decision to construct an automatic RSS ingestion pipeline.
On the floor, it’s a reasonably easy utility. It fetches articles from an RSS feed, parses them into structured objects, and shops them in PostgreSQL.
However the aim was by no means to construct an RSS reader.
The aim was to discover the engineering choices that rework a Python script right into a dependable knowledge pipeline.
On this article, I’ll stroll by these choices, the errors I made alongside the best way, and the teachings I realized constructing my first pipeline with Kestra.
Why Construct One other ETL Pipeline?
After ending my first ETL venture, I thought of transferring on to one thing fully completely different.
Possibly an information warehouse venture. Possibly Apache Spark. Possibly an API with a extra complicated transformation layer.
As a substitute, I constructed… one other ETL pipeline.
At first, that in all probability appears like a step backwards.
In spite of everything, I’d already constructed an extraction pipeline, made it idempotent, and scheduled it with GitHub Actions. Why repeat the identical train?
As a result of I wasn’t attempting to study a brand new dataset. I used to be attempting to study a brand new mind-set.
One lesson from my first venture caught with me.
Writing the ETL logic wasn’t the troublesome half. The troublesome half was all the pieces surrounding it.
How ought to the pipeline be executed?
How ought to it get better from failures?
How do you package deal it so it runs constantly on any machine?
The place does scheduling belong?
And maybe the most important query of all:
The place ought to the obligations of the appliance finish, and the place ought to the obligations of the orchestration layer start?
These questions don’t have a lot to do with RSS feeds or GitHub repositories. They’re engineering questions, and I noticed I might discover them with nearly any knowledge supply.
That’s why I selected an RSS feed.
Not as a result of RSS is especially thrilling, however as a result of it’s deliberately easy.
The extraction logic solely takes just a few traces of Python. That meant I might spend much less time worrying about enterprise logic and extra time fascinated about structure.
For a similar motive, I made a decision to maneuver away from GitHub Actions for this venture.
GitHub Actions was an awesome introduction to scheduling. It confirmed me the best way to automate a workflow and gave me my first style of working an ETL pipeline with out guide intervention.
However GitHub Actions isn’t designed particularly for orchestrating knowledge workflows.
I wished to grasp what adjustments if you use a software that’s constructed with knowledge pipelines in thoughts.
That’s what led me to Kestra.
Somewhat than asking, “How do I run this Python script each hour?”, I discovered myself asking a unique set of questions.
How ought to retries be configured?
How are workflow executions tracked?
How ought to atmosphere variables be handed right into a container?
What does a failed execution appear like?
How do you separate the appliance from the infrastructure that runs it?
These questions have been precisely what I wished to discover.
By selecting a easy ETL pipeline, I might deal with the engineering choices as an alternative of getting distracted by difficult enterprise logic.
Wanting again, I believe that was the correct determination.
This venture isn’t fascinating as a result of it processes RSS feeds.
It’s fascinating as a result of constructing it compelled me to consider reliability, repeatability, and orchestration in a method my first venture by no means did.
The First Architectural Determination: Docker Earlier than Kestra
With the venture outlined, my first intuition was to leap straight into Kestra.
In spite of everything, orchestration was one of many essential causes I selected this venture. Why not begin there?
As a substitute, I did one thing that turned out to avoid wasting me loads of frustration later.
I ignored Kestra fully.
Which may sound counterintuitive, however I wished to keep away from introducing a number of transferring elements earlier than I knew the core utility truly labored.
So I constructed the venture in layers.
First, I wrote the Python ETL.
Its job was deliberately small. Fetch the RSS feed, parse every entry into an Article object, and save the outcomes to PostgreSQL. Nothing extra.
feed = feedparser.parse(RSS_URL)
articles = parse_feed(feed)
save_articles(articles)
That’s actually all of the ETL did. I deliberately saved the appliance small as a result of the main focus of this venture wasn’t the transformation logic. It was all the pieces that occurred round it.
As soon as that labored reliably, I turned my consideration to the database.
I wished repeated executions to be protected, so I made the inserts idempotent utilizing PostgreSQL’s ON CONFLICT DO NOTHING. That method, working the pipeline a number of occasions wouldn’t create duplicate rows.
INSERT INTO articles (…)
VALUES (…)
ON CONFLICT (id) DO NOTHING;
This one line made the pipeline protected to execute repeatedly. Whether or not Kestra ran the workflow as soon as or 100 occasions, PostgreSQL grew to become answerable for stopping duplicate data.
Solely after the ETL and database labored collectively did I introduce Docker.
That call modified how I assumed in regards to the venture.
Initially, Docker felt like one other software I wanted to study.
By the top of the venture, I noticed it had turn out to be one thing far more essential.
It grew to become the unit of execution.
FROM python:3.13-slim
WORKDIR /app
COPY necessities.txt .
RUN pip set up –no-cache-dir -r necessities.txt
COPY . .
CMD [“python”, “fetch_rss.py”]
Packaging the ETL this manner meant the appliance grew to become self-contained. As a substitute of asking Kestra to grasp my Python venture, I might merely ask it to run a container that already knew the best way to execute the pipeline.
As a substitute of considering, “Kestra will run my Python script,” I began considering, “Kestra will run my Docker picture.”
That distinction might sound delicate, nevertheless it fully adjustments the connection between your utility and your orchestration layer.
As soon as the ETL was packaged right into a container, it not mattered whether or not it ran on my laptop computer, inside Kestra, or on one other machine solely.
The runtime atmosphere was all the time the identical.
That consistency gave me one thing I didn’t have earlier than: confidence.
Earlier than introducing Kestra, I might run the container manually and confirm that it fetched the RSS feed, linked to PostgreSQL, and endured the anticipated data.
If one thing failed, I knew the issue wasn’t hidden behind one other layer of orchestration.
That validation step turned out to be extremely helpful later.
Throughout improvement, I bumped into points with networking, container configuration, and workflow execution. As a result of the Docker picture had already been validated independently, I might instantly rule out the ETL itself and deal with the orchestration layer.
That made debugging dramatically simpler.
Wanting again, I believe this was one of many largest classes from the venture.
It’s tempting to attach each element collectively as shortly as doable and hope all the pieces works.
A greater strategy is to validate every layer earlier than introducing the following one.
On this venture, the order seemed like this:
Validate the Python ETL.
Validate PostgreSQL persistence.
Validate the Docker picture.
Lastly, let Kestra orchestrate a container that I already trusted.
Every layer constructed on the earlier one.
By the point Kestra entered the image, I wasn’t attempting to debug Python, PostgreSQL, Docker, and orchestration on the similar time.
I used to be solely fixing one downside.
That incremental strategy made all the venture really feel far more manageable, and it’s a workflow I’ll in all probability proceed utilizing on future knowledge engineering tasks.
The Assumption That Turned Out to Be Unsuitable
With a working Docker picture, I used to be satisfied the laborious half was over.
I had a Python ETL that labored.
I had PostgreSQL working in Docker.
I had verified that the container might fetch RSS articles and save them to the database.
Now all Kestra needed to do was run it.
Or so I assumed.
My authentic assumption was easy.
Kestra would level to my Python recordsdata, execute the script, and all the pieces would work precisely because it had from the command line.
It didn’t.
I shortly found that there was an essential distinction I hadn’t absolutely appreciated.
Kestra is an orchestrator.
It isn’t answerable for constructing Python environments or managing utility dependencies. Its duty is deciding when and the way workloads ought to run.
That realization modified the course of the venture.
As a substitute of treating Kestra as one other place to execute Python code, I began treating it because the layer answerable for orchestrating a workload that already existed.
That workload was my Docker picture.
As soon as I made that psychological shift, the structure grew to become a lot cleaner.
The ETL grew to become a self-contained utility.
Docker grew to become the deployment artifact.
Kestra grew to become the orchestrator.
Every layer had a transparent duty, and none of them wanted to know the way the others labored internally.
Apparently, reaching that time wasn’t fully easy.
My first intuition was to discover a method for Kestra to execute the Python venture straight. That strategy sounded less complicated, however the extra I experimented with it, the extra I noticed I used to be asking the orchestrator to take duty for one thing the appliance ought to already present.
As soon as I embraced Docker because the execution artifact, the workflow grew to become a lot less complicated.
Some approaches seemed promising at first however launched pointless complexity. Others labored, however didn’t align with how Kestra encourages containerized workloads to be executed.
Ultimately, I landed on a workflow that felt surprisingly easy.
As a substitute of attempting to show Kestra the best way to run Python, I let Kestra do what it does finest.
It launches a container.
duties:
– id: run_etl
sort: io.kestra.plugin.scripts.shell.Instructions
containerImage: rss-pipeline-etl:newest
taskRunner:
sort: io.kestra.plugin.scripts.runner.docker.Docker
instructions:
– python /app/fetch_rss.py
Wanting on the workflow now, what stands out isn’t how a lot YAML it accommodates. It’s how little Kestra truly must learn about my utility. Its solely duty is to launch a container that already is aware of the best way to execute the ETL.
Inside that container, my utility already is aware of precisely what to do.
That small architectural change solved extra than simply the rapid execution downside.
It additionally strengthened an concept that’s changing into a recurring theme in my studying journey.
Good engineering usually isn’t about including one other layer.
It’s about giving every layer a single duty and permitting it to try this job properly.
Python shouldn’t fear about orchestration.
Kestra shouldn’t fear about Python dependencies.
Docker shouldn’t know something about RSS feeds.
Every element solves a unique downside.
As soon as I finished asking one software to unravel each downside, all the system grew to become a lot simpler to motive about.
Wanting again, this was in all probability the most important mindset shift in the entire venture.
I didn’t simply discover ways to use Kestra.
I realized what orchestration truly means.
As soon as a Pipeline Runs Mechanically, Every little thing Modifications
Up till this level, I had been working the pipeline manually.
If one thing failed, I used to be sitting in entrance of my pc. I might learn the error, make a change, and check out once more.
That security web disappears the second a pipeline begins working by itself.
One of many first issues I configured in Kestra was a easy hourly schedule.
triggers:
– id: hourly_schedule
sort: io.kestra.plugin.core.set off.Schedule
cron: “0 * * * *”
On paper, it was only a cron expression.
In apply, it represented a a lot greater shift.
The pipeline not trusted me remembering to run it.
Each hour, Kestra would begin a brand new execution, launch the ETL container, and course of the newest articles from the RSS feed.
That instantly raised one other query.
What occurs if a kind of executions fails?
Throughout improvement, I intentionally launched failures to reply that query. I pointed the pipeline at an invalid database host and watched what occurred.
The primary execution failed, precisely as anticipated.
Extra importantly, it didn’t cease there.
As a result of the workflow was configured with retries, Kestra routinely tried the execution once more after a brief delay.
retry:
sort: fixed
maxAttempts: 3
interval: PT30S
That was one in every of my favourite moments within the venture.
As soon as I corrected the configuration, the workflow accomplished efficiently with out requiring any adjustments to the appliance itself.
That was one in every of my favourite moments within the venture.
Not as a result of retries are significantly difficult, however as a result of they highlighted one other separation of obligations.
The ETL shouldn’t resolve whether or not it deserves one other likelihood.
That’s an orchestration concern.
By transferring retry logic into Kestra, the Python utility remained centered on a single duty: course of the feed and persist the outcomes.
The orchestration layer dealt with resilience.
Scheduling launched one other problem that I’d already encountered in my first ETL venture.
Repeated executions imply repeated makes an attempt to put in writing knowledge.
If the identical RSS article seems in a number of hourly runs, the pipeline shouldn’t insert it twice.
Thankfully, I had already solved the same downside earlier than.
The database layer was designed to be idempotent utilizing PostgreSQL’s ON CONFLICT DO NOTHING.
INSERT INTO articles (…)
VALUES (…)
ON CONFLICT (id) DO NOTHING;
That meant each execution might safely try and insert the identical data with out creating duplicates.
The mix of scheduling, retries, and idempotent writes made the pipeline far more forgiving.
If a run failed, Kestra might retry it.
If a profitable retry encountered knowledge that had already been written, PostgreSQL would merely ignore the duplicates.
Neither layer wanted to know what the opposite was doing.
They every dealt with their very own duty.
The final enchancment was visibility.
Early in improvement, my logs seemed precisely such as you’d count on from a venture that was nonetheless being debugged.
I printed complete RSS objects to the console simply to ensure the parser was working.
It wasn’t fairly, nevertheless it served its objective.
Because the pipeline grew to become extra secure, these debug statements grew to become much less helpful.
I changed them with logs that described the execution as an alternative of dumping uncooked knowledge.
Earlier than
print(feed.entries[0])
After
print(“=== RSS PIPELINE START ===”)
first = feed.entries[0]
print(“First entry preview:”)
print(f”Title: {first.get(‘title’)}”)
print(f”Hyperlink: {first.get(‘hyperlink’)}”)
print(f”Feed title: {feed.feed.title}”)
print(f”Fetched articles: {len(articles)}”)
print(f”Saved {len(articles)} articles to the database.”)
print(“=== RSS PIPELINE END ===”)
Every run now tells a easy story.
=== RSS PIPELINE START ===
First entry preview:
Title: Christian Ledermann: Migrate From mypy To ty And pyrefly
Hyperlink: https://dev.to/…
Feed title: Planet Python
Fetched articles: 25
Saved 25 articles to the database.
=== RSS PIPELINE END ===
These adjustments didn’t make the appliance smarter. They made it simpler to grasp. And that’s an essential distinction.
Good observability isn’t about producing extra logs. It’s about producing the correct logs.
By the top of the venture, I noticed one thing fascinating. The Python code hadn’t grown dramatically. Many of the work had occurred round it.
Wanting again, a lot of the engineering effort wasn’t spent making the ETL smarter. It was spent making it extra dependable.
Scheduling ensured it ran with out me.
Retries helped it get better from transient failures.
Idempotency protected the database from duplicate writes.
Logging made each execution simpler to grasp.
Individually, none of those adjustments have been significantly complicated. Collectively, they reworked a easy Python script into one thing that behaved far more like a manufacturing system.
The Last Structure
By the top of the venture, the structure had settled into one thing that felt surprisingly easy.
Wanting on the last structure, it’s tempting to suppose the venture was all the time heading on this course.
It wasn’t.
Every layer was added solely after the earlier one had been validated.
The ETL got here first.
Then PostgreSQL.
Then Docker.
Lastly, Kestra.
That order mattered.
As a result of each element had already been examined independently, I by no means discovered myself debugging Python, Docker, PostgreSQL, and Kestra on the similar time. Every determination lowered the variety of unknowns as an alternative of accelerating them.
Extra importantly, each element ended up with a transparent duty.
Python is aware of the best way to fetch, parse, and persist RSS articles.
PostgreSQL is aware of the best way to retailer knowledge safely and stop duplicates.
Docker supplies a constant execution atmosphere.
Kestra decides when the workload ought to run and what ought to occur if it fails.
None of these parts are attempting to do one another’s jobs.Mockingly, that’s what made the completed system really feel a lot less complicated than I anticipated.
What This Mission Modified In regards to the Method I Suppose
Once I began studying knowledge engineering, I assumed the troublesome half could be writing ETL code.
That’s what most newbie tutorials deal with.
You discover ways to name an API.
You rework the info.
You reserve it someplace.
Repeat.
These are helpful abilities, however they’re just one a part of the image.This venture taught me that the engineering begins after the script works.
As soon as a pipeline is anticipated to run each hour, survive transient failures, keep away from duplicate knowledge, and produce logs that designate what occurred, the questions turn out to be far more fascinating.
You’re not fascinated about particular person capabilities. You’re fascinated about programs.
One of many largest mindset shifts for me was understanding the distinction between execution and orchestration.
At first, these concepts felt nearly interchangeable. Now they really feel fully separate.
The Python utility ought to deal with enterprise logic. The orchestrator ought to deal with when, the place, and the way that utility runs.
Protecting these obligations separate made all the venture simpler to motive about.
It additionally modified how I take into consideration Docker.
Earlier than this venture, I noticed Docker primarily as a solution to package deal purposes.
Now I see it as a deployment artifact.
As soon as the ETL had been packaged right into a container and validated independently, I might cease worrying about whether or not it will behave in a different way inside Kestra.
That confidence turned out to be one of many largest benefits of containerizing the appliance.
Maybe an important lesson, although, had nothing to do with Kestra or Docker. It was the worth of constructing incrementally.
Each main determination adopted the identical sample.
Construct the smallest factor that works.
Validate it.
Solely then introduce the following layer.
That strategy made the venture really feel a lot much less overwhelming than attempting to attach all the pieces collectively from the start.Wanting again, I believe that’s a lesson I’ll carry into each future venture, whatever the know-how.
Wanting Forward
This RSS pipeline is just the second venture in my knowledge engineering studying journey. In comparison with my first ETL pipeline, the code itself isn’t dramatically extra complicated. What modified was the best way I approached the issue.
As a substitute of asking, “How do I write this script?”, I discovered myself asking questions like:
The place ought to this duty reside?
What occurs if it fails?
Can I run it repeatedly with out worrying about duplicate knowledge
Can I belief it to run once I’m not watching?
These questions pushed me to suppose much less like somebody writing Python code and extra like somebody designing a system. And I think that’s the actual worth of constructing tasks.
Each venture teaches a brand new software. However the most effective tasks slowly change the best way you suppose. This one definitely did.
I’m positive the following venture will problem a totally completely different set of assumptions. Truthfully, I’m wanting ahead to discovering out what they’re.
That is a part of my ongoing sequence documenting my transition from programs analyst to knowledge engineer. In case you’ve been following alongside, thanks.
Join with me on LinkedIn, YouTube, and Twitter.
