It’s been a depressing job market for tech workers. How depressing? According to a recent poll, 25% of engineers said it took them a year to find a new job.
I know many of you think this hiring slowdown won’t last, but what if AI is causing a long term shift in demand for certain skills? What if AI is partially responsible for all these layoffs?
I’m a really curious person and I needed answers. So I decided to dig into the cold, hard data and analyze 20 million tech job openings, from November 1, 2022 to Feb 21, 2024 to understand how AI has impacted the demand for certain tech skills and engineers. And to answer the question: Are these tech layoffs driven by AI in any way?
Here’s what I’ll analyze:
- Which engineering roles (ie. machine learning, data science, frontend, backend) have grown and declined the most in demand
- How salaries for certain engineering roles have changed in the past year
- Which skills/technologies/languages have grown and declined the most in demand
- Whether companies are hiring much more AI talent after layoffs than before
The source of my data and my detailed methodology are in the very end of this post, but my main dataset was 20M job openings crawled directly from over 50,000 company websites, provided by Revealera (a data provider for job postings) and the list of companies that did layoffs were from this list from Techcrunch.
Without further to do, let’s jump into the data!
Which engineering roles have grown and declined the most in demand?
First off, how has demand changed for different engineers?
To find out, I decided to compare the raw number of new job openings for each type of engineer from Nov 1, 2022 to Feb 21, 2023 to the # of new job openings from Nov 1, 2023 to Feb 21, 2024.
My conclusion? The job market for engineers is a tale of two very different worlds.
On one end, the demand for AI research scientists and machine learning engineers has boomed. The # of job openings grew 80% for AI scientists and 70% for machine learning engineers. There’s no secret that these people are highly coveted right now.
On the other hand, the demand for every other type of engineer has slowed down. Job openings for mobile engineers, frontend engineers and data engineers all dropped more than 20% from a year ago. They’re the ugly duckling nobody wants to hire right now.
I don’t think anyone is surprised that AI engineers and scientists are in huge demand, but what’s interesting is the decline in demand for other engineers isn’t uniform. There are differences in demand , which suggests that it isn’t all due to tech layoffs. AI might have some impact here.
For instance, job openings for backend engineers declined just 14% vs 24% for frontend engineers. I think there may be an “AI effect” here, as companies need stable and scalable backend infrastructure to deploy machine learning models like LLMs. On the other hand, there’s no need for strong frontend skills if you’re investing in AI. Whether you’re using Angular or React has little impact on how well your machine learning models perform.
Secondly, the fact that hiring for data scientists is more resilient could be attributed to the fact that they complement AI efforts by providing data preparation, cleaning and analysis. But at the same time, they aren’t as high in demand as ML engineers because data scientists tend to focus on traditional models that aren’t based on deep learning or LLMs.
How salaries for certain engineering roles have changed in the past years
How have salaries changed for each role changed in the past year?
To find out, I decided to analyze the salaries in these job postings, focused ONLY on high cost of living cities like SF, NYC and Seattle. It’s important to note that these are public salary ranges listed in job postings, *not* actual salaries of people currently employed in these roles. And I’m looking at only base salaries, *not* total compensation.
Good news: Salaries haven’t declined. But they haven’t gone up much either. In fact, if we adjust for inflation, salaries have been flat, even among AI scientists and ML engineers.
What this suggests is that, yes there’s been a huge spike in demand for AI talent, but there isn’t a lack of supply of machine learning engineers and scientists either. To give you some context, the # of job openings for machine learning engineers in the past 4 months is still around 15% lower than what it was during the hiring craze from 2020-2021. And AI and ML engineers + scientists make up just 5% out of all engineering related job postings.
So the bad news is if you’re thinking of switching gears and transitioning to becoming a machine learning engineer, there’s lots of competition for relatively few job openings, compared to the rest of software engineering.
It’ll be interesting to see how salaries changed in the next few years, but it might be possible that salaries for engineers might remain stagnant for awhile.
Which skills and languages have grown and declined the most in demand
Next, I decided to analyze which machine learning skills have increased in demand the most.
NLP (natural language processing) had by far the largest growth in demand with a 155% increase in jobs that mention “NLP”. On the surface, this makes sense because the #1 killer use case for LLMs are in creating chatbots for customer service. Mentions of computer vision, on the other hand grew half as much, and that’s likely because the use cases for computer vision are much more specialized (ie self-driving vehicles) and not as popular.
Of course, the one “skill” missing here is LLMs, which I couldn’t include in the chart because it would simply overwhelm everything else. Mentions of LLMs in job postings increased a whopping 3000% year over year, as evidenced by this chart.
OK, let’s put aside AI and ML skills for a second. Which traditional engineering skills such as programming languages have declined the most in demand? I looked at 4 popular backend languages and 4 popular frontend languages, to see which languages were the biggest losers.
A couple of takeaways: First, Rust is a huge winner. The number of job openings mentioning Rust went up 32% from a year ago, which is quite amazing considering job openings for backend engineers went down. Maybe there’s a supply/demand imbalance worth digging into here.
2nd, though demand for React went down, it’s still clearly stealing market share from Angular and Vue. 3rd, Ruby on Rails’ popularity continues to wane and was the biggest loser. Finally, Python’s strong resiliency is mostly due to the fact it’s the de-facto language for machine learning.
Any relationship between companies doing layoffs and their hiring for AI talent
Now, for the ultimate question: Is there any relationship between all these tech layoffs and AI? Are companies laying off people so they can spend more money on hiring AI talent instead?
I decided a reasonable way of finding this answer was to compile a list of companies that did mass layoffs in 2023, and compare the # of AI jobs they posted in the 3 months before the layoffs and the 3 months after the layoffs. If there was a noticeable increase after the layoffs (compared to the change in overall jobs post-layoffs), it seemed reasonable there was at least a casual relationship.
I looked at 50 companies (big and small) that did layoffs and what did the data say? There’s simply no relationship between layoffs and hiring for more AI talent. Companies that did layoffs in 2023 had on average 20% more job postings related to AI in the 3 months after the layoff announcement vs pre-layoffs, but they also had 24% more job postings overall (for any role) in the 3 months afterwards.
Here’s a sample of some Big Tech companies that did layoffs last year, and their AI job openings pre and post layoffs.
This doesn’t kill the narrative that companies are laying off people to spend more on AI (after all, they could be using that $$$ to buy GPUs), but it does make it weaker. Yes, companies are focusing on AI, and yes a few companies are likely laying off low level workers that can be replaced by AI. But the narrative that most companies are laying off people so they can hire a lot more AI talent isn’t backed by any data at all. It’s disingenuous.
What’s more likely is that companies are shedding headcount because they over-hired since the pandemic, interest rates are still high, and Wall Street is rewarding cost-efficiency measures. Not because they need to dramatically shift their corporate strategy to AI.
What’s interesting is that companies are putting up more job postings after they announce layoffs than before, which might lead credence to the idea that companies may want to layoff low performers and replace them with high performers. But that’s a story for another article.
Conclusion
It remains to be seen whether demand for other types of engineers besides AI experts picks up, or whether this is the “new normal”. Yes, hiring is way down from late 2020-2021, but tech job openings are basically on par with what it was pre-2020. The hiring surge that occurred when interest rates was 0% may simply be a mirage that will never happen again.
The good news is it doesn’t seem like companies are laying off non-AI engineers so they can use the money to hire more AI engineers. At the same time, AI is changing the demand for certain types of engineers, and decreasing the demand for others. The more related your skill set is to AI, the more in demand you are. And this could be a long term trend.
But what should you do if you’re a software engineer that can’t find a job, or worried about being laid off? While I wouldn’t recommend you shift gears immediately and rebrand yourself as an AI engineer, it wouldn’t hurt to learn some machine learning on the side, so you can start contributing to machine learning and AI projects.
For example, if you’re a devops engineer, you can learn how to build CI/CD pipelines for ML projects. Likewise, if you’re a security engineer, you can learn how to build threat modeling systems with machine learning, or ways to safeguard AI models from tampering. The key is to integrate principles of ML into your skillset naturally. So you can become a more complete well-rounded engineer. I daresay the definition of a “full stack engineer” might change to include AI and ML skills.
Maybe the AI hype dies down. Maybe AI is mostly a solution in search of a problem. Who knows. But it doesn’t hurt to learn the skills needed in case AI becomes a disruptive technology (as mobile and cloud was a decade ago)
Subscribe to get more data-driven research on hiring, labor and AI trends
Future articles will include: outsourcing trends, our best attempt to use data to find out which careers are least likely to be replaced by AI, and more! No fluff, no armchair predictions, just refreshing research backed by DATA
Methodology (the boring stuff)
I analyzed 20M job postings from 50,000 companies (5000 enterprises and 45,000 smaller ones, with almost every public company > 1B market cap). These job postings were provided by Revealera.com, a data provider of job postings data.
I built a machine learning classifier that read the job description/title for each of these job openings and classified them to an engineering job title (ML engineer, AI scientist, backend engineer, frontend engineer, etc). I also used an entity extraction algorithm to extract the skills present in each job posting. I indexed all of this data into an Elasticsearch cluster where I performed simple time period aggregations to find out the volume of job openings for each type of engineer and skill.
Salaries were taken from the midpoint of the salary range listed in each job with a location in either Seattle, New York, or San Francisco. If there were multiple salary ranges, the min was the min of all of them, likewise for the max. Each company was counted just once so if there were multiple jobs for a company, I took the average of all of them for that company. Only salaries with a USD currency were analyzed.
