How to Build a Competency Framework for Data Science Teams
In 2021, 365 DataScience carried out a study of 1,000s of Linkedin profiles to understand trends in the data science discipline. A couple of points that really stood out were that "Very few individuals (less than 2%) have stayed at the same job for more than 5 years" [1] and "_The median ** time spent on the job by a data scientist in our study was 1.7 years" [1_]. Fortunately, I haven't seen this turnover in my teams, but I know many data scientists, and most argue that ‘lack of role clarity‘ is one of their top 3 challenges. If you feel this is happening to you or in your team, I hope this article helps you build a competency framework which is adaptable, fair and robust to provide answers for this ‘lack of role clarity'.
PS: All images are authored by me, unless otherwise specified.
First of all, why do we need a competency framework?
A competency framework is a means by which teams communicate which behaviours are required, valued, recognised and rewarded with respect to specific roles or levels. For example, the expectations for a junior data scientist should be different to a principal data scientist. The same applies for two similar senior data scientists, where one is an individual contributor and another is a manager.
If you are leading a Data Science team, I can guarantee you will get questions such as ‘How do I get to the next level?', ‘What do I need to do to get a higher bonus?' or ‘I did everything I was told to do, why am I not getting rewarded for it?' When you leads teams, you must be able to answer these in a fair and robust way. You can't be saying one thing in Q1 and something different in Q3. What worked for one direct report, has to work for the rest. The vertical and horizontal roles need to be defined and be transparent to everyone. In summary, you need a solid competency framework.
Let me guide you through my thought process on building a competency framework. Naturally, I will focus on the Data Science side of things, but hopefully, this could even be useful for other disciplines.
The "easy" step: define your roles
The screenshot below shows our roles at Skyscanner. Whilst other industries might have more levels of seniority (consultancies here with their 10 level hierarchies) or have specialised DS roles (product data scientists in Facebook), I will stick with the career ladder below, as I feel many companies actually relate to this.
At Skyscanner we have 5 hierarchy levels in our Data Science discipline. In reality, we rarely hire Jr. Data Scientists, so our seniority mix starts from Data Scientists and above. It is also interesting to highlight that people Management roles start at a certain seniority level. It is rare to see a jump from Data Scientist IC role directly to a DS manager role.
In the next section, we will try to capture what we expect from each of these roles.
The "eagled eyed view" step: what are you looking for in your Data Science discipline?
We have established that any member on your team could be a Junior DS, DS, Senior DS and Principal DS. Now we need to understand what core competencies do we value for our discipline.
As mentioned in the section header, this exercise requires an eagled eyed view. We are looking to distill the main themes that can describe both the day to day work of a data scientist and their possible future growth. Each of these areas will be later broken down into more specific requirements, but for the moment, we need to focus on the building the "foundation pillars".
Taking inspiration from other disciplines
If what I have just written above is too vague, let's review what core competencies Monzo looks for their engineering teams [2] (this framework is public as mentioned in this article from Monzo [3]) or core competencies for product managers [4]. I have purposely chosen the engineering and product management disciplines as these are teams which data scientists work the most with. I thought it would be interesting to understand how they would value themselves, and come up with the equivalent for the Data Science discipline.
Monzo
Monzo establishes 4 core competencies: Scope, Impact, Technical skills and Behaviours. For each competency, you can read how they differ at different levels too, but remember that we at the "eagle eyed view" step. Each competency outlines what it is trying to measure. For example, under Behaviours, Monzo will want to assess expectations such as influence, communication and leadership. In addition, looking at the Technicall skills competency, you can see that they are not listing specific skills; it's about "how you apply your technical skills" [directly quoted from the document]. In fact, they mention that previous iterations were too detailed which "led to engineers check boxing their progression, and was often overly specific to certain areas of the business" [directly quoted from the document].
Product manager roles
Intercom shows a possible breakdown for a PM competency framework. Looking at their foundational pillars, 5 areas are suggested: Insight Driven, Strategy, Execution, Driving Outcomes and Leadership Behaviours. I really like how they break down each of these 5 competencies into smaller and more focused expectations for the job. For example, if you look at Execution, they break it down into (1) start with the problem (2) think big, start small (3) ship to learn and (4) ship the whole customer experience.
Data Science at Skyscanner looks at 6 key pillars + manager track
Having read a couple of external examples, let's talk about the data science discipline. At Skyscanner, we evaluate individuals based on 6 core competencies. The following list also describes the goal for each competency.
- Scope. How well defined are the problems you are tackling? What degree of leadership are you taking? How far ahead are you planning?
- Expertise. Are you learning or are you a master of a skill / area? How are you ensuring that others benefit from this knowledge?
- Delivery. What is your degree of responsibility for each task? Think not only on your own delivery, but also how you are delivering through others.
- Build the right thing. Understanding business context and translating it into value. Complexity may or may not be aligned with value.
- Build it right. How robust, scalable, cost effective and understandable is my work? Are you following engineering, modelling and statistical best practices?
- Run it right. How do you ensure that your solution is as good as it was yesterday? What processes do you have in place to account for system failures?
- The manager track. How you develop your direct reports and help them grow and achieve their career goals. How you build high performance teams. Communication is a key aspect here both from the company to the team and from the team to the company.
I really like this breakdown. It's slightly more detailed than the Monzo example but less restrictive than the PM example. Think about your day to day work as a data scientist and I am sure that anything you do can be pinned to one of these core competencies.
The "clear definitions" step: setting clear boundaries between seniority levels for each core competency.
Now it's time to clearly define the expectations of each competency for different levels of seniority. The way I define success for this step is with 2 dimensions:
- There is no overlap between seniority levels. Reading your competency framework in ascending order of seniority (junior DS to principal DS) should clearly show the progression jumps between levels. Setting boundaries such as leading 1 vs multiple projects or influencing the squad vs other teams is super important. We don't want confusion in the team.
- You don't create a checkbox list of everything that can fall into each competency. Strike a balance between being detailed and general. Clearly defined competencies are crucial, but being overly specific can make your framework less adaptable or outdated as time goes on. For example, regarding expertise, do you really want to list all possible ML solutions such as time series forecasting, recommender systems, geospatial models or classical classification predictions? My feeling is that this is too specific.
Let's dive into how we defined our progression steps in Skyscanner's Data Science discipline. Unfortunately, from a presentation point of view, Medium doesn't natively support tables, so I will have to paste screenshots for specific examples.
Scope – the art of being comfortable with uncertainty
How well defined are the problems you are tackling? What degree of leadership are you taking?
Expertise – mentor what you know you are good at
Are you learning or are you a master of a skill / area? How are you ensuring that others benefit from this knowledge?
Delivery – make it happen, no excuses
What is your degree of responsibility for each task? Think not only on your own delivery, but also how you are delivering through others.
Build the right thing – focus on adding value.
Understanding business context and translating it into value. Complexity may or may not be aligned with value.
Build it right – following best practices
How robust, scalable, cost effective and understandable is my work? Are you following engineering, modelling and statistical best practices?
Run it right – how much would you bet that your system works in 3 months time?
How do you ensure that your solution is as good as it was yesterday? What processes do you have in place to account for system failures?
The manager track – a different set of skills
How you develop your direct reports and help them grow and achieve their career goals. How you build high performance teams. Communication is a key aspect here both from the company to the team and from the team to the company.
Summary – wrapping up highlights
The reasons I really like this competency framework for data scientists are:
- We have achieved our goal of "There is no overlap between seniority levels". I feel that for each core competency, it is clear what is expected at each level. The assumption is that, once you master the competencies of a level, you bring them with you to next. This is why we are asking you to demonstrate extra responsibilities.
- We have achieved our goal of "You don't create a checkbox list of everything that can fall into each competency". Throughout the framework, there is no mentioning of tooling or specialties. I don't care if you do MLOps with Databricks or AWS, this tooling will be defined by the company. I am not an expert on time series, but you might be. It doesn't matter if you have built 10 deep learning models if they don't add value.
- It is easy to remember 6 competencies (plus the manager track). I feel it is important for team members to easily point out what are the core competencies they will be measured on. Not the specific detail, but the themes. Unconsciously, this helps frame any type of work. For example, in stand ups or in planning meetings, I tend to ask a lot about "Are we building the right thing?" or "Are we running it right?"
- But it also has enough details to be a transparent guide for individuals. The idea of providing a long list of examples is to make sure there are little confusions between roles.
- Flexibility. It is true that the levels presented in the tables might be more representative of the tech world, but flavours of these dimensions could be used to define a competency framework for data science consultancy firm too.
Acknowledgments
- [1] 365 Data Science, article: Who is a Data Scientist in 2021? – A Research on 1,001 Data Scientists
- [2] Monzo engineering competency framework
- [3] Monzo blog talking about their competency framework
- [4] Intercom product manager competency framework
Further reading
Thanks for reading the article! If you are interested in more of my written content, here is an article capturing all of my other blogs posts organised by themes: Data Science team and project management, Data storytelling, Marketing & bidding science and Machine Learning & modelling.
Stay tuned!
If you want to get notified when I release new written content, feel free to follow me on Medium or subscribe to my Substack newsletter. In addition, I would be very happy to chat on Linkedin!
Get notified with my latest written content about Data Science!
