19 hours ago
Man scrapes 4.1 million jobs with ChatGPT, turns data into new hiring platform
If you've been job hunting recently, you've probably come across the same problem: endless 'ghost jobs' on platforms like LinkedIn and Indeed. These are postings that are either outdated, fake, or placed by third-party recruiters to collect resumes, and they waste everyone's a Reddit user, it was the last straw. 'I got sick and tired of how LinkedIn & Indeed is contaminated with ghost jobs and 3rd party offshore agencies, making it nearly impossible to navigate,' he he did what frustrated coders often do: he built a solution.
Using the ChatGPT API, he managed to scrape 4.1 million jobs directly from company websites, which were not even on Indeed or AS A DATA CLEANERThe problem with scraping job listings is that every company has a different format on their website. One might list the title and salary upfront, another hides it at the bottom, and others use completely custom layouts. That makes it hard to collect large amounts of jobs in a structured experimenting with the ChatGPT API, the Reddit user realised he could dump messy, raw job postings into the model and ask it to return neatly formatted information in JSON, a standardised format designed to simplify and streamline job descriptions. This included details like job title, years of experience, salary, location, and whether the role was his words: 'After playing with ChatGPT's API, I realised that you can effectively dump raw job descriptions and ask it to give you formatted information back in JSON (ex salary, yoe, etc).'This was the breakthrough that made large-scale scraping possible.4.1 MILLION JOBS, 220K REMOTEUsing this method, he scraped 4.1 million jobs directly from company websites. Out of these, more than 220,000 were remote positions -- something jobseekers are increasingly then built a platform called where users can search these listings with filters far more powerful than what LinkedIn or Indeed usually offer. You can filter by job title, exclude irrelevant keywords, and even slice by years of experience.'Update: I've now used this technique to scrape 4.1 million jobs (with over 220k remote jobs) and built powerful filters. I made it publicly available here in case you're interested ( he announced on THIS REPEATABLE FOR OTHERS?The big question many asked in the Reddit thread was: can others do the same thing?The answer is yes -- but with caveats. The method itself is straightforward:Scrape job listings from company the raw text into ChatGPT's back structured data (JSON) with clean labels like salary, years of experience, that data in a searchable the Reddit user also pointed out that the process is computationally heavy and expensive at scale. While it's easy to try on a small sample of job postings, running millions of listings through ChatGPT's API requires both infrastructure and make sure the jobs were legitimate, he also cross-referenced company data using and Dun & Bradstreet, filtering out shady agencies or duplicates. He noted in his post that this step made his database more reliable than simply scraping his Reddit post, he even shared the link to his ChatGPT prompt that he used to scrape millions of job NOTICE THE DIFFERENCEEarly users of said the difference was obvious: fewer ghost jobs, fewer spammy recruiter listings, and more real roles posted directly by are still occasional errors. Sometimes a job gets wrongly tagged as remote, or salary details are missed. But compared to the bigger problem of ghost postings, these are the Reddit user summed it up: 'The jobs themselves are real and posted directly by the companies. That's what matters most.'A GLIMPSE INTO AI-POWERED JOB SEARCHThis project shows how tools like ChatGPT can do more than just write essays or code snippets. With the right workflow, they can structure messy human data at scale -- turning the chaos of job listings into something for jobseekers who've wasted hours scrolling through fake postings, this AI-powered shortcut might feel like a breath of fresh air.- EndsMust Watch
