Automate Your Job Search with AI Agents

User: AI find the best jobs that fit my profile. AI: On it..

---

Table of Content

Intro

1. Common challenges impacting every job seeker (The Why..)
2. AI Agent's crucial role in optimizing the job search process (The What..)
3. Introducing AI Agents capabilities to the process (The How..)

Hands-on: Implementing AI-powered job search engine

1. Project Structure
2. Framework
3. Tools
4. Data
5. Tasks
6. Agents
7. LLM
8. Output Model
9. Putting Everything Together: The Crew
10. Result Analysis

Source Code Summary Potential Improvements References

---

Job search can be difficult and time-consuming. It can take weeks and up to several months to navigate through job listings until eventually landing a job.

The amount of time depends on many factors, such as expertise, role demand, markets, etc.. but according to unemployment data (including those passively scouting) from the U.S. Bureau of Labor Statistics shows that the median duration in March 2024 was 21.6 weeks ~ 5 months.

On a human age scale, and considering the amount of time an average job seeker takes to search for new jobs, that's a lot of unretrievable lifetime just spent on screening 100s of applications.

The number of applications submitted is one of the success factors for landing a job. A study result positioned the average number of applications it takes to land the right position as 100–200+ applications, which nonetheless can have outliers depending on the market, economic situation, and an applicant's expertise. The sheer amount of applications and effort one puts in searching for a job justifies our motivation behind building efficient and scalable solutions saving individuals thousands of hours.

---

Common challenges facing every job seeker (The Why..)

Finding the role that best fits one's skills and requirements in a market flooded with job listings is challenging. If you've been in such a position you know how stressful and tiring it can be going through hundreds of listings trying to crunch and match skills, salary ranges, prerequisite requirements, etc… The longer the process takes the lower the motivation one has and the higher their chances to quit the search and settle for less.

The time it takes to find the right role varies depending on one's experience, application timing, and how in-demand their skills are in the job market. Other uncontrollable factors like those impacting the industry and organizations they're applying for.

The challenges we tackle in building an AI-powered job searching engine in this article are mainly focused on optimizing the job search and the matching process, thus resolving key challenges like the following:

Key Challenges

• Job Advertisements Search: Traditionally, screening jobs begins with choosing the right source platform where jobs are advertised. Filtering through jobs with specific criteria and evaluating the final result by digesting all their content. For better reachability, many choose to work on more than one platform. Hardships start when one has to juggle between multiple sources and keeping track of where is what.
• Job Advertisements Evaluation: Does the job posting fit my previous experience and skill set? Do I have the years of experience required? Is it in the right location? What languages do I need to speak? Does the salary range meet my needs? When is the role's start date? Those are some of the questions that arise when evaluating a job advertisement's fit to one's requirements.
• Organization Evaluation: A common step during the job screening is researching the organization. Some common criteria for evaluating an organization are employee reviews, market performance, reputation, etc.. More benchmarks or criteria can be introduced depending on the market the organization operates in. If you're searching for cross-market opportunities this step can be very time-consuming considering all the factors that need to be taken into consideration.
• Job Shortlisting: Before coming to a final decision and after screening tens and hundreds of job listings you decide to shortlist the few that made it through. Whichever approach one chooses to take, will require some kind of rating strategy based on some criteria to sort the listings by. These are subjective ratings based on one's preferences and priorities. You can imagine the amount of mixing and matching between job listings required to complete this task.

⚠️ Note: The challenges that one experiences during searching for a job are much greater than the ones mentioned in the list above. The list discussed is directed towards the article content and solution scope.

The challenges listed incorporate repetitive tasks, require strong reasoning capabilities, and a large amount of content to digest to arrive at a clear actionable list.

These tasks perfectly meet the key characteristics of employing LLM-powered agents.

---

AI Agent's crucial role in optimizing the job search process (The What..)

Imagine the end result of hiring 100s of individuals who can screen job advertisements, deeply evaluate the advertising companies and their job postings, and finally tailor their recommendations that best fit your needs. A full end-2-end process that's built on automation and reasoning that significantly reduces the amount of time anyone would ever spend screening online job advertisements.

Let's analyze AI agent's capabilities in terms of automating and personalizing job searches and recommendations. We will further dive into building the engine in subsequent sections. For now, we focus on What AI agents bring to the table.

1. Agents can use tools

Agents are capable of taking advantage of defined tools to fulfill their tasks. This involves tools enabling better research capabilities, such as providing access to search the internet and request data using pre-defined APIs. This extends the capabilities of agents towards interfacing with a large selection of tools far beyond the data they were trained on and additionally provides the ability for users to orchestrate how agents solve specific tasks with a selected toolkit.

2. Agents can digest a substantial amount of information

With the increasing size of context windows (1ml+) for AI models, and even reaching infinite context windows, the amount of information agents can digest and reason about is ever-growing. Distributed correctly, agents can analyze an infinite amount of information and provide users with a final summarized version.

3. Agents can reason

While there can be limitations to the reasoning capabilities of AI Agents, the new, improved, and optimized models being published more & more often are closing in on that gap. Considering the task at hand of analyzing job advertisements and comparing those to a user query based on some parameters, the task is potentially considered a straightforward analysis task for a highly trained AI agent.

4. Agents can summarize and structure results

A Tool that can process a large amount of information is valuable but loses its value when it is incapable of delivering that information in an efficient and structured manner. AI Agents are suspectable to hallucinations which can impact how information is summarized and reported. Providing a well-engineered prompt and additional validation steps ensures that Agents fulfill their tasks with correct and reliable results.

5. Agents can collaborate

The difference in background, roles, and tasks agents are assigned to work with, provide a layer of variability that supports approaching problem-solving from different perspectives. Collaboration between agents utilizes this capability to optimize the final results delivered to users.

6. Agents are scalable

Dynamic allocation of large amounts of agents to parallel execution of tasks is extremely powerful. The approach shines when problems can be divided into smaller chunks and worked on by separate agents. A large number of agents can be allocated to a batch of job postings, each responsible for tackling a step in the process and passing the result to the next for the next phase. Finally, agents can gather the results for a final summary and structure. That's one approach one can take to build a fast and scalable distributed handling of job advertisements across multiple channels or sources.

How can we put together all those capabilities for our use case?

---

Introducing AI Agents capabilities to the process (The How..)

To ensure a solution that utilizes all AI capabilities, we introduce agents to every step of the process.

1. Agents will use tools to retrieve information about job advertisements. The initial job posting will be retrieved based on the user query.
2. Agents will provide a rating and reasoning behind the rating for the job postings based on the user's resume and query.
3. Agents will use internet access tools to research information about the source organization for the job advertisement and based on that provide the organizations with a rating score.
4. Agents will finally summarize the results and structure the results based on a predefined model.

The next section will dive into utilizing the listed AI Agents' capabilities for building a job search engine.

---

Hands-on Tutorial: Building an AI-Powered Job Search Engine

In the current section, we will walk through the complete implementation step by step and end with analyzing the final results and potential improvements.

Project Structure

The project structure ensures a clear separation between functionalities and components, providing an easier approach to modifying and adapting the code to your requirements. It consists of:

• A configs directory that contains all needed configs and parameters to configure agents (role, background, backstory) and tasks (description and expected output)
• A data directory containing all needed data to test the job search engine
• A models directory containing defined models for the expected output schema
• A utils directory containing needed support functions
• agents_factory.py and tasks_factory.py are used to dynamically generate instances of agents and tasks based on configurations defined.

project/
├── configs
│ └── agents.yml # Agents configuration
│ └── tasks.yml # Tasks configurations
│
├── data
│ ├── sample_jobs.json # JSON file containing job listings
│ └── sample_resume.txt # Text file containing a resume
│
├── models
│ └── models.py #  ORM models
│
├── utils
│ └── utils.py # Utility functions and helpers
│
├── .env # containing all needed environment variables
│
├── agents_factory.py # Factory class for creating agent instances
├── tasks_factory.py # Factory class for creating task instances
│
└── main.py # Main

Framework

The project will utilize the crewAI framework to build the end-to-end application. It provides a straightforward interface to build AI Agents with assigned tasks and specific tools. It integrates very well with other available AI frameworks, thus fitting perfectly to the scope of this article.

Ensure first that a Python environment (in my case using python-3.11.9) is available and install the framework with all the tools needed.

pip install 'crewai[tools]'

Installation of crewai[tools] package should cover all the needed packages to run the application.

Tools

The Agents will require a set of tools to fulfill their assigned tasks. Assignment of tools to agents will depend on how we define the request flow architecture and how information is transmitted from one agent to the other.

Two tools are needed to test our use case, both of which are already available within crewAI.

FileReadTool

On a production scale, where one can manage to accumulate a large list of dependable APIs providing job data, the go-to agent tools are those that can interface with multiple providers and fetch job information based on selected parameters.

For the scope of our application, we will work with an already retrieved JSON response _sample_jobs.json_ having a list of job details to keep the focus on showcasing the solution.

A generated resume _sample_resume.txt_ is also available to test ratings provided by Agents.

SerperDevTool

Agents will be tasked to gather information about organizations and provide rating feedback to users. This tool supports the process by enabling agents to search the internet. To use the tool Make sure that you create an account on serper.dev to acquire the API KEY needed and load it into the environment.

Ensure that the API key is defined in .env file.

SERPER_API_KEY=<>

Tools can be directly imported

from crewai_tools import FileReadTool, SerperDevTool

Adding tools and assigning them to agents makes extending the application scope with new features very fast and simple.

Data

A _sample_jobs.json_ contains a JSON response of a list of sample job advertisements synthetically AI-generated as testing data for our use case.

While the job data presented in the article is in JSON format, it could be easily in any other format for agents to parse, such as personally collected job description data documents stored as simple text, PDF, or Excel files.

Fully utilizing the search process requires agents to use tools to interface with job platforms APIs for full automation and scalability of the data ingestion process. Examples of such official job search API providers are Glassdoor or Jooble.

Tags: Artificial Intelligence Hands On Tutorials Job Hunting Large Language Models Python

Add Fav