5 Bad Habits Killing Your Potential as a Data Engineer
Every data engineer wants to feel like they are constantly evolving as a professional and growing their technical skills. As data engineers we like to be challenged and feel we are progressing towards our end goal.
This is the nature of the work we do—improving systems, implementing processes to increase data quality, and finding innovative ways to cut down costs. Everything is about taking steps forward rather than back.
Three years into my role as a data engineer, I started to feel that my growth was plateauing. I was no longer excited about the work I was doing. I didn't feel as if I was learning from those around me. There were no new technical skills that I was able to learn through my job.
Now, looking back, I can see it was the right time for me to move on to my next role. However, there were a few specific habits of mine that I can pinpoint as reasons for why my success began to stall at that company.
In this article, I am going to share the bad habits that may also be killing your potential as a data engineer, and how you can fix them.
Avoiding what's hard
This is one that data engineers tend to embrace when first getting started. In fact, I don't think you can become a successful data engineer without embracing the suck.
You need to be able to push through debugging something that went wrong and working on difficult projects because that is the only thing that is going to make you better.
As we progress in our data engineering careers, we almost don't want to look like we don't know something. We become insecure about the skills we may be lacking, avoiding taking on projects where we struggle.
However, this is a huge mistake.
You can't improve your Python scripting, for example, if you never volunteer to write a Python script. You won't learn the latest modern data stack orchestration tool if you let someone with more experience build out the data pipeline.
You need to ask your manager for hard tasks that you may not know how to do, to broaden your skill set and stretch your mind.
Not asking for feedback or code reviews
Writing your code and building out projects successfully is one thing. Improving your code and projects is another thing. The former won't make you better than you already are. The ladder will shape you into a better data engineer.
Without feedback, we can't improve.
While it may be tempting to finish your tasks and merge your code changes straight to production, code reviews are necessary to become a better data engineer. Someone else reviewing your code can point out things you may have never known otherwise.
For example, someone on your team may ask why you coded out a longer SQL function when you could have used a pre-built one that you didn't even know existed. By having them point this out, you then learn the pre-built function and apply it to your future code.
Little things like this can go a long way in your Data Engineering journey.
If you don't currently have a code review process, ask your manager if you can implement one. This will involve setting new standards for the way your team creates PRs and merges code to production.
Here are a few things I found helpful when creating a code review process:
- add PR templates
- enabled branch protection (forces code to be approved before being merged)
- add code review guidelines to your repo's ReadMe file
These three things will allow you to create an effective process that ensures only the best code is being merged to production, allowing you to grow in your coding skills as well.
Not volunteering for tasks you don't know how to do
This goes hand-in-hand with avoiding what's hard. Many times, we only volunteer to work on the tasks that we know how to do, avoiding the things we don't know how to do.
However, how can we expect to expand our skillset if we don't take on the tasks that we are unfamiliar with?
The truth is, we can't. We either expand the range of things we feel comfortable working on, or we become complacent.
If you are a data engineer, volunteer for the more analytics-heavy tasks that can teach you a new tool like dbt or Prefect.
If you are a highly technical engineer, volunteer to interact more with the business or optimize team processes. This will help develop your soft skills.
Lean towards the things that you don't know and you will always be growing and evolving. Working on things you know nothing about may lead you down a slightly different career path that you didn't even know was possible!
Working with dbt as a data engineer helped me to realize how much I loved the tool and wanted to work closer with the business, leading me to transition to analytics engineering.
Not practicing SQL every day
When you are in a role where you use SQL every day, this seems like a no-brainer. Of course, you are going to get practice in!
However, I've been in a few different data engineering roles where I don't use SQL at all. While it may not have been important to the projects I was working on, it's a skill I believe every data engineer needs to know.
In fact, they shouldn't just know it, but they need to be sharp in it! Almost every technical interview will involve SQL coding. If you stop practicing your SQL, it could hurt you when you are ready to change jobs.
Job hunting aside, SQL is the core of many data engineering roles. Just because you stop using it in a particular role doesn't mean you won't need it again.
Build a habit of practicing SQL every day through platforms like Leetcode which gives you real-life practice problems.
And, if you are in a specialized data engineering role, there may be another skill that you feel you need to practice every day. If you're an avid Python developer but not using Python in your current role, don't lose that skill. Find an extra 30 minutes each day to challenge yourself to strengthen it.
Not surrounding yourself with engineers smarter than you
This one may be the most important out of all the bad Habits killing your potential as a data engineer. While it's not necessarily a habit, it can become a habit to surround yourself with people you are comfortable with.
It's comfortable to stay in a job where you know your colleagues and how they work. However, a lot of times, we can outgrow the people around us. Maybe they are great work friends, but are they helping you grow into the data engineer you want to be?
In my last role, I was the only analytics engineer on the data team. While this was great because I got to learn many new tools and skills that I may not have learned otherwise, I lacked someone to learn from.
I wasn't truly improving because there was no one to review my code.
I wasn't growing into a better data engineer because there was no one to critique my work or show me a better way of doing things.
Although this was comfortable, I could feel myself once again plateauing. I knew it was time to find a role where the people around me knew more about analytics engineering than I did.
My growth ever since has been incredible. I've learned more about data modeling, databases, SQL code, and run optimization than I ever thought I would. Working with someone smarter than me has shown me all of the areas I can improve on and has made me a better engineer.
Conclusion
While these are all things that may feel uncomfortable at first, it is more uncomfortable to stay in the same spot for 5–10 years without any growth.
You don't want to look back at the last 10 years of your career as a data engineer, thinking you wasted your time because you didn't do any of these things.
I can't think of a worse place to be than the same place years from now. To become the data engineer you want to be, you need to:
- lean into difficult tasks
- ask for feedback and code reviews
- volunteer for projects you don't know how to do
- practice SQL every day
- surround yourself with people smarter than you
If you can do these 5 things, it's impossible to not improve as a data engineer. Your growth will exponentiate and lead you to places you never even thought you could go.
Good luck! I'm rooting for you.
