VIETNAM NATIONAL UNIVERSITY, HANOI INTERNATIONAL SCHOOL PROGRAM: BUSINESS DATA ANALYST GRADUATE PROJECT REPORT Using matrix and nature language processing techniques to provide job advice STUDENT’S NAME : HOANG MANH LINH SUPERVISOR : Assoc. TRAN THI OANH CLASS : BDA2020A Hanoi - Year 2024 VIETNAM NATIONAL UNIVERSITY, HANOI INTERNATIONAL SCHOOL PROGRAM: BUSINESS DATA ANALYST GRADUATE PROJECT REPORT Using matrix and nature language processing techniques to provide job advice SUPERVISOR : Assoc. TRAN THI OANH STUDENT : HOANG MANH LINH STUDENT ID : 20070947 COHORT : BUSINESS DATA ANALYST CLASS : BDA2020A Hanoi - Year 2024 TABLE OF CONTENTS ACKNOWLEDGEMENT. 2 LIST OF TABLES.
3 LIST OF FIGURES. Flowchart for the problem. Aims, Objectives and Contributions of the thesis. Objectives and Contributions.
Significance and Motivation of the thesis. 10 CHAPTER 2: THEORETICAL BACKGROUND. Nature Language Processing (Text Processing). Term Frequency - Inverse Document Frequency (TF-IDF).
Part-of-Speech tagging (POS tagging). Named Entity Recognition (NER). Content-based Filtering. 15 CHAPTER 3: DATASETS & EXPLORE DATA ANALYSIS.
Explore Data Analysis. About the cv. About the job. 24 CHAPTER 4: RESEARCH METHODOLOGY.
Using NLP to handle long text. Latent Dirichlet Allocation (LDA). Using POS Tagging and NER from spaCy. Using Content-based filtering to build matching model.
39 CHAPTER 5: EXPERIMENTS AND EVALUATION. Data Pre-processing. Recommendation model and Evaluation. Recommend is based on Text String Processing.
Content-based Filtering. 57 ACKNOWLEDGEMENT I would like to express my heartfelt appreciation to Associate Professor Dr. Tran Thi Oanh for her exceptional guidance and support during my research on "Using Machine Learning and Deep Learning Techniques to Provide Job Advice to New Graduates." Her expertise and insights have greatly improved the quality of my research and sparked my passion for using machine learning and deep learning to assist new graduates in their job search. I am also grateful to Mr.
Nguyen Trung Hieu and CiaoLINK for providing invaluable data for my study. Their support has been crucial in conducting comprehensive and data- driven research. Additionally, I want to acknowledge the International School - Vietnam National University, Hanoi, for providing valuable education and knowledge that allowed me to apply theoretical concepts to practical research. I am aware that my research paper may have limitations due to the complexity of the topic and time constraints.
Therefore, I kindly request the readers of my paper to provide their valuable contributions and feedback, which will undoubtedly enhance the quality and impact of my work. Lastly, I would like to express my heartfelt gratitude to my family and friends. Their constant support and encouragement have played a vital role in helping me overcome challenges throughout my studies and research. While I have strived to ensure the thoroughness of my thesis, I acknowledge that there may still be areas for improvement.
I welcome and value any constructive feedback from my teachers to enhance and refine my work. Thanks for everyone sincerely! Hanoi, June 2024 Hoang Manh Linh 1 ABSTRACT This thesis aims to utilize matrix and Natural Language Processing (NLP) techniques to provide job advice to new graduates in the USA. The motivation for this work arises from the significant layoffs that occurred in the US this past April, creating substantial challenges for recent graduates in securing employment. By focusing on Natural Language Processing (NLP), the system can analyze lengthy text passages, which facilitates the development of a more effective matching model.
This approach allows for the extraction and understanding of intricate details from job descriptions and applicant profiles, ultimately leading to more accurate job recommendations. The system employs content-based filtering as its primary model, leveraging this method to analyze and match data from job descriptions and applicant profiles based on specific skills and experiences. One of the key metrics used in this content-based filtering approach is cosine similarity, which measures the cosine of the angle between two vectors—in this case, the vector representations of job descriptions and applicant profiles. The cosine similarity scores in the system range from 0.
These scores indicate that the system is capable of identifying highly relevant job opportunities for some CVs, with higher scores reflecting better matches. However, for some profiles with lower scores, the recommendations might be less accurate due to insufficient or less detailed information available in the CVs. The integration of advanced NLP techniques enhances the system's ability to understand complex job descriptions and applicant profiles, leading to precise recommendations. This approach improves job suggestion relevance and accuracy, meeting diverse job seekers' needs.
In summary, this job recommendation system provides recent graduates with an effective tool to navigate the U. job market, showcasing the potential of machine learning and NLP in addressing job recommendation challenges. 2 LIST OF TABLES Table 2.1: About the cv.2: About the job. 18 3 LIST OF FIGURES Figure 1.1: Flowchart of the project .1: Tag of POS .2: Top 20 Most Common Job Titles .3: Top 20 Locations of Individuals.4: Top 10 Jobs in the United Kingdom.5: Top 10 Jobs in the United States .7: Distribution of Seniority Level .8: Distribution of Employment Type.9: Distribution of Job Function .10: Distribution of Title .11: Distribution of Location .12: Distribution of Industries .13: Heatmap of Top Job Functions by Top Locations .14: Job Functions by Employment Type .15: Top 10 Jobs in New York .1: Data Pre-processing flowchart .3: CV Data after translate .4: CV Data after using unidecode .5: Using NLTK for Summary data .6: Using LDA and TfidfVectorizer for Summary data .7: Named all the topics .8: Apply POS and NER for CV .9: Apply POS and NER for JOB .11: Result if CV matches Job .12: Result if CV does not match Job .13: Result for CV1 .14: Result for CV2 .15: Result for CV3.
52 5 ABBREVIATIONS Abbreviations Full Form CV Curriculum Vitae LDA Latent Dirichlet Allocation TD-IDF Term Frequency – Inverse Document Frequency NLTK Natural Language Toolkit NLP Nature Language Processing AI Artificial Intelligence IT Information Technology ML Machine Learning POS tagging Part-of-Speech tagging NER Named Entity Recognition 6 CHAPTER 1: INTRODUCTION 1. Problem In April, U.-based employers reported 64,789 job cuts, representing a 28% decrease from the 90,309 cuts recorded in March and a 3.3% drop from the 66,995 cuts reported in April 2023. This data comes from a report by Challenger, Gray & Christmas, Inc., a global outplacement and executive coaching firm. So far this year, companies have announced 322,043 job cuts, marking a 4.6% decrease compared to the 337,411 job cuts announced by the end of April last year.[1][2] Despite this overall decline, the job market remains challenging, particularly for recent graduates who are entering a competitive landscape with fewer available positions.
The competitive nature of the job market highlights the importance of innovative solutions for job seekers, especially new graduates who may lack experience and professional networks. This underscores the significance of my research on "Using Machine Learning and Deep Learning Techniques to Provide Job Advice to New Graduates." By leveraging advanced technologies, this study aims to offer personalized job recommendations. The goal is to help new graduates navigate the job market more effectively and find suitable employment opportunities in a difficult environment. Maxtrix as Content-based Filtering algorithms can analyze vast amounts of data from job postings, resumes, and industry trends to identify patterns and match candidates with positions that align with their skills, interests, and career goals.
This personalized approach can significantly enhance the job search process, making it more efficient and tailored to the unique needs of each graduate. Furthermore, by utilizing these technologies, the study aims to reduce the time and effort required for new graduates to secure employment, ultimately improving their chances of success in a competitive job market. The findings of this research could have far-reaching implications, offering 7 valuable insights and tools for educators, career counselors, and job placement services to support new graduates in their career endeavors. Flowchart for the problem Figure 1.1: Flowchart of the project The process diagram is presented with directional arrows, illustrating each step from data processing of CVs and JOBs to recommending jobs for CVs based on matching.
Below is a detailed analysis of each step in the process: Raw Data: The first step is collecting data from applicants' CVs and job descriptions from companies. This is a crucial step to ensure that the input data is complete and accurate. The data collection should include detailed information such as the candidate's skills, experience, education, and specific requirements from the job descriptions. Data Processing: Processing raw data from CVs and job descriptions to prepare for the matching process.
Using NLP methods such as Part-of-Speech (POS) tagging, Named Entity Recognition (NER), Tokenization, Keyword Extraction, Topic Modeling, this includes cleaning the data, standardizing formats, and extracting important keywords and phrases. This enhances the efficiency and accuracy of the matching algorithms. Matching CV and JOB: Matching applicants' CVs with job descriptions based on predefined criteria. This is the core step of the system, using content-based filtering to determine the compatibility between applicants and jobs.
The accuracy of the matching depends on the quality of the data and the capability of the algorithms. 8 Recommend 5 JOB for CV based on matching: Recommending suitable jobs for applicants based on the matching results. The final step is generating job recommendations for applicants, which should be presented clearly, including information about the position, company, seniority level, etc. Aims, Objectives and Contributions of the thesis 1.
Aims The primary objectives of this thesis are to develop a comprehensive job recommendation system for people in the United States, with a particular focus on recent graduates seeking employment. By leveraging LinkedIn data, the system will analyze the profiles of individuals who are currently living in the US, as well as job descriptions from companies that are actively hiring. The system will integrate content-based filtering to analyze textual similarities between job descriptions and candidates' profiles, ensuring relevant matches, and collaborative filtering to consider user behavior and preferences, providing personalized job recommendations. Ultimately, this thesis aims to create a robust career reference system that not only matches job seekers with suitable opportunities but also provides valuable insights into industry trends, essential skills, and career development pathways.
Objectives and Contributions The primary objectives of this thesis are to develop a comprehensive job recommendation system tailored for individuals seeking employment in the United States, with a particular emphasis on recent graduates entering the job market. The system will leverage LinkedIn data to analyze profiles of individuals residing in the US and job descriptions from companies that are actively recruiting. By integrating content- based filtering, the system will assess textual similarities between job descriptions and candidate profiles to ensure highly relevant matches. 9 Additionally, collaborative filtering will be utilized to incorporate user behavior and preferences, providing personalized job recommendations.
Ultimately, this thesis aims to create a robust career guidance system that not only aligns job seekers with suitable opportunities but also delivers valuable insights into industry trends, essential skills, and career development pathways. The primary contributions of this thesis are highlighted through the following points: Utilize extractive techniques to process raw data supplied by CiaoLink company. Develop a matching algorithm to offer the most appropriate job recommendations for job seekers. Significance and Motivation of the thesis The significance and motivation of this thesis lie in addressing the challenges faced by job seekers, particularly recent graduates, in navigating the competitive job market in the United States.
With the dynamic and often overwhelming nature of the job search process, there is a critical need for a system that can streamline and enhance the matching of candidates with suitable job opportunities. By leveraging LinkedIn data, this thesis seeks to provide a data-driven approach to job recommendations, ensuring that candidates receive relevant and personalized job matches.