The Art of Stress Management as a Data Scientist
The Art of Stress Management as a Data Scientist
Data scientists' work can be very demanding. A couple of weeks ago, I experienced the busiest week that happens every quarter in my role. During this "planning week", I needed to make forecasts for four quarters, over 10,000 products, using the latest data.
Although all models are already pre-trained offline, maintaining different models at various product hierarchies and layers of complicated post-processing procedures makes the execution no easy task, and the tight deadline allows little room for errors.
This video records the mental state I was in throughout the week:
Like many of those who are pursuing dreams elsewhere, I video chat with my Mom every week to catch up and vent about what life throws at me. Whenever I share my work stress with her, she always tells me to be grateful.
Appreciation would be a surprising reaction to stressful workloads, but my Mom has her own reasoning. As the eldest in her family, she had to give up high school to work in a factory's assembly lines. Her biggest wish at the early stage of her career was not to have to work the night shift anymore.
To her, physical exhaustion, extended working hours, and minimal payouts were all insignificant compared to the disruptive sleeping schedule.
Thus, when my Mom hears me talking about stress coming from work done during normal business hours, using a computer on top of a standing desk, connected to oversized monitors, and at my own house saving the hustle of commuting to the office, of course, her first reaction is "Clearly, you don't see how lucky you are."
Indeed, I am fortunate enough not to have to wake up in the middle of the night for a shift or lift a finger to perform at work. However, these mental stresses that sometimes keep me sleepless at night are also real. The question is, where do these stresses come from?
Data scientists in different companies and industries, with different seniority levels and job descriptions, may have very different answers. However, I'm quite certain that we have all encountered being bombarded by endless error messages while running a program, receiving ambiguous and unreasonable requests from stakeholders, and experiencing self-doubt while working on deliverables. The list can go on.
Our stress is indeed different from physical exhaustion. Given a couple of days of rest, muscles can recover from physical stress. Are minds taking longer, or do they ever recover from "mental exhaustion?"
The fact that we are not required to work in the office next to certain machines seems to give us freedom, but if you think further, it's really a restriction in disguise.
Unlike those who can't work unless they are by the assembly line, whose work officially ends once they clock out, our work doesn't vanish when we close the computer.
The hours may end, the computers may be closed, but work can continuously linger around. It may appear as a message or an email in your phone that begins with "No need to address this now, but…" It takes over your mind whenever there is some empty space available: during showers, before falling asleep, while working out. Your mind is still preoccupied with unresolved work matters and reflections about mistakes.
Drawing a line between work and life takes active effort. That is why I am writing this article. For all the 50+ articles I published here at Medium, I wrote a lot about how to be an excellent data scientist – – preparing the interviews to enter the field, continuously growing in the role, and various articles exploring different technical fields. I realized I was missing a critical piece:
What do you do when you are not a data scientist?
Do you know well enough what to do when you are not a data scientist to be a better data scientist?
Honestly, I wouldn't have so much to write about stress management if it weren't for the busy planning weeks I experience every quarter. Initially, I was unaware of what I was dealing with and would come out of it feeling like a zombie. Viewing the upcoming planning week approaching as an unclimbable mountain, I would start to panic a month in advance, anticipating the stress I would face.
Now, if you watch the video, you can see I am more prepared. Although it's still chaotic, and unexpected errors keep showing up, I have the confidence to navigate the uncertainties and manage to deliver. Here are a few things I think I did right to meet tight deadlines while maintaining a sane mind state.
Plan Ahead but be flexible
Fear and anxiety stem from unknowing. The workload usually sounds more overwhelming until an action plan is laid out. No matter how far the destination sounds, there is a way to get there, and with good planning, we can get there faster.
In my case, I usually create the execution plans ahead of time and keep a log of the progress. Besides complexities, there are sequential orders embedded in the process. For example, model A's input is model B's output, thus setting priority is very important.
Even with detailed plans, unexpected situations happen every time. Debugging with a tight deadline is my worst nightmare. After encountering several unfortunate experiences, I learned to be more flexible with my plans. Depending on the situation, I would shift priorities, give up certain goals, or settle for plan B. I've come to realize that sometimes done is better than perfection.
Keep Others Accountable
As project complexities or scope increase, it is less likely that a deliverable will be completed solely by one person. Collaboration and utilization of others' work are often necessary. In this case, having strong hard skills alone is no longer sufficient to ensure successful delivery. It is essential to also cultivate soft skills in effective communication and to hold others accountable for their deliverables.
It's a learning process to transition from independent contributor to team player. Whenever you think that teaching someone how to do something is much slower than simply doing it by yourself, get rid of that thought because it is not true in the long run.
Walking alone is faster, but walking together can get us further. Teaching or collaborating with others gives us more capabilities and bandwidth to approach perfection. By cultivating trust in collaboration, we can get more things done efficiently. Plus, stress always feels light when shared with others.
Develop Self-care routines
Self-care is the most important aspect of putting yourself first. Knowing how to rest is as important as knowing how to work.
Although it's tempting to unwind by watching mindless TV or scrolling through social media after a busy work day, in my experience, it's not as relaxing as one might expect. In fact, it makes me more tired and less energetic for the work the next day because of the information overload, blue light exposure, passive engagement, etc.
Believe it or not, physical exhaustion can help relieve mental overload. Personally, I find it beneficial to combine mental exhaustion with physical exertion, such as engaging in heavy-weight training at the gym or going for a long-distance run outside, followed by a relaxing stretching session. It feels as though the stress is being sweated out of my body. However, if you have been lacking sleep, it is best not to engage in a high-intensity workout until you are fully rested and recovered.
Besides this day-to-day relaxation, I always take some days off after a few stressful working weeks to recharge. I take some vacations to travel or visit my family. Otherwise, I go to the local bookstores to check out what's new and what the staff recommends. "I need to recharge my battery" is the cue for me to visit the bookstore. Even though I may not buy books on every visit, it takes hours and hours to just wander around different aisles.
There is no one-size-fits-all self-care routine. You'll need to experiment to find what works best for you.
Manage Expectations
Managing expectations has two aspects: your own expectations and others' expectations.
For your own expectations, I am referring to the extended working hours needed to deliver on time. During the busiest week, I always set my mindset to keep an eye on work messages, jump in meetings at any time, and even work overtime to keep an efficient workflow. At least for me, it is manageable as long as it's planned, and I can ensure WLB other than this one week every quarter.
For others' expectations, I mean you need to know your limit and not overpromise. Be brave to communicate your boundaries and push back if it's too overwhelming. Knowing how and when to say no is critical. It is better to say no in advance than to say yes and then be unable to commit.
Although uncertainties and bugs are unavoidable. It is very important to reflect and improve afterward. If you always find your work overwhelming, it is also the cue to ask yourself and others, what's a more sustainable solution? Do we need more headcount? Do we need to cut down on the complexities? Can we improve the inefficiencies inside each process? The right answers to these questions will spare you from unnecessary pain and stress.
Accept imperfection and don't internalize
This is my biggest takeaway. Although everyone wants to shoot for perfection, we must accept and make peace with imperfections.
When I work on my deliverables every quarter, I want to ensure that every step is done correctly and submit the best forecasts I can. However, I have to admit and accept that I cannot control everything, and thus, done is better than perfection.
Even though my work occasionally contains flaws and unexpected errors, and I would have done better if I had more time, I need to understand that the flaws are in the forecasts, not in myself and my capabilities. I shouldn't internalize the imperfection. It is very important to give ourselves affirmation like this, especially when we have tried our best.
Feline Friends
Last but not least, I want to thank my cats. I can't emphasize enough how much they have helped me through my ups and downs with their warm love (disguised by their careless faces), not to mention how they have inspired me to illustrate the importance of causality.
We have talked enough about stress management. The last point I want to touch upon is why we are exposing ourselves to stress in the first place. Is stress avoidable, not through stress management but through eliminating the source?
In my mom's case, she had to take on a lot of responsibilities at a young age to support her family, and her choices were limited. In many ways, I am much more fortunate. Every quarter after the work is done, I ask myself this question: Is it worth putting myself in a stressful situation over and over again?
Indeed, I consider myself relatively very tolerant of stress. After all, it's nothing compared to the years when I was pursuing a PhD, surviving on minimum wage, handling multiple responsibilities, and having no clear future.
Stress at work can come from many different sources: technical challenges, collaboration frictions, communication barriers, office politics, etc. However, most of the time, in essence, stress just comes from being an adult who is starting to take responsibilities in different roles.
We grow out of stress. Every time when I push through a stressful project, I feel very proud of myself for being mentally stronger. Observing myself survive uncertainties and stressful situations makes me more confident. In general, to be tolerant and grow out of stress, we are motivated by at least one of the following:
- Personal growth goals: Goals may change, but your guiding North Star remains constant. You only need to find it, even if it takes a lifetime.
- Deliver impact: Fulfillment gives us happiness, but work is not the only place to find it.
- Personal interests: Passion is the best fuel.
- Decent compensation: Money cannot buy happiness, but it can alleviate certain unhappiness and buy you freedom.
It is crucial to regularly reflect on whether the amount and source of stress are beneficial, helping you grow stronger, or harmful, potentially causing permanent damage. Be mindful not to be trapped by stress.
After years of working night shifts at an assembly line, my Mom found her way out through writing. It may sound surprising if you don't know the fact that no matter how tired she was after her shifts, she never stopped reading and writing.
When you enjoy doing something, it doesn't burden you but only brings satisfaction and fulfillment, and more to my Mom. She obtained a degree in adult education while working. After winning several internal writing contests and publishing some articles, she got the chance to be a writer/editor of her company's newspaper, a 9 to 5 office job. I can't believe I only noticed the resemblance between us after writing this article.
Thank you for reading. Let me know if you also experience stress at work. What are the sources, and how do you deal with them?
Also, my work schedule is not always this packed. Check out this video for one of a normal workday as a WFH data scientist:
Lastly, don't forget to:
- Check my other articles on different topics like data science interview preparation; causal inference;
- Follow me on LinkedIn or YouTube
- Subscribe to my email list
