I Thought My Data Engineering Career Was Over in 2017

In 2017, I sat in a cubicle staring at a legacy ETL screen and felt convinced that my data engineering career had already reached a dead end.

I had spent four months studying the Big Data stack: Hadoop, Spark, and Hive. I cleared the internal interviews required to join a Big Data project. I was ready to make the move.

Then the organisation moved in a different direction. I was not assigned to the Big Data team. I joined a legacy project using Informatica PowerCenter instead.

At the time, it felt like a career failure. Friends were discussing distributed processing and Spark clusters while I was writing SQL and shell scripts around technology I considered old.

Years later, I can see that this “stagnant” project became one of the most useful periods of my career.

Myth 1: Your assigned project defines your future

As a junior engineer, project allocation can feel permanent. I thought working with a legacy tool meant I would become a legacy developer forever.

The project was only the environment. The capabilities I practised inside it were much broader:

• complex SQL and data transformation;
• Unix and shell scripting;
• deployment and CI/CD fundamentals;
• incident investigation;
• dependency management;
• data-quality failures; and
• communication with people who depended on the pipelines.

When I eventually moved into Big Data systems, those skills transferred. The execution engine changed. The need to understand data movement, failure, correctness, and operations did not.

A project can constrain your current tasks. It does not have to define your professional identity.

Myth 2: Success requires the newest tool

In 2017, Hadoop was the exciting destination. Then Spark became the default answer. Later the industry focused on cloud warehouses, dbt, serverless processing, and now AI-assisted engineering.

The tools matter. They change what is possible and which problems are economical to solve.

But a career built only on tool familiarity has a short shelf life. The more durable questions are:

• How does data enter the system?
• Where can it become incorrect?
• How do we recover from partial failure?
• Which workload requires batch, streaming, or interactive access?
• How do consumers discover and trust the output?
• What trade-off are we making between cost, latency, and complexity?

Informatica taught me to reason about pipelines. That reasoning later transferred to Spark, cloud platforms, and modern analytics tooling.

Do not ignore tools. Learn the engineering ideas underneath them.

Myth 3: Someone else controls your learning path

For a while, I waited for a manager or staffing process to fix my situation. I thought I needed permission before I could become a Big Data engineer.

Eventually, I reached out to a manager in another team and offered to contribute as a shadow resource.

I completed the responsibilities of my assigned project and then worked on real tasks with the Big Data team. Running both paths was exhausting, and it is not a sustainable model for everyone. But it gave me the practical experience that an internal course alone could not provide.

The important lesson was not “work every evening.” It was:

Take ownership of creating evidence that you can do the work you want next.

That evidence might come from an internal collaboration, an open-source contribution, a small production improvement, a certification with a real project, or a portfolio that demonstrates your thinking.

Legacy work can become an advantage

Engineers who have seen several generations of technology develop a useful perspective. They recognise when a new tool solves an old operational problem and when it merely changes the interface.

Working with legacy systems also teaches constraints:

• migrations cannot happen instantly;
• business-critical pipelines may outlive their original designers;
• documentation and lineage are not optional;
• compatibility matters; and
• “replace everything” is rarely a complete plan.

That perspective now helps me evaluate modern data and AI systems with more realism.

Your trajectory is larger than one assignment

A disappointing project can slow a plan without ending it. Use the environment to build transferable skills, keep learning toward the next role, and look for small ways to practise beyond your current job description.

Your employer chooses the project you are assigned. You still have influence over the engineer you become.

That journey from visual ETL to modern analytics also shaped how I think about transformation tooling today. Read From Informatica to dbt for the technical side of that evolution.