If you’re tired of sifting through ghost jobs and third-party offshore agencies on LinkedIn and Indeed, you’re not alone. I was frustrated with the noise and decided to take matters into my own hands.
I discovered that most companies post jobs directly on their websites, but until recently, there was no way to scrape them at scale because each job posting has different structures and formats. That was until I started playing with ChatGPT’s API.
I realized that I could use ChatGPT to dump raw job descriptions and ask it to give me formatted information back in JSON, including salary, years of experience, and more. And that’s exactly what I did.
## The Result
I’ve now used this technique to scrape an astonishing 4.1 million jobs, with over 220,000 remote jobs. And the best part? I made it publicly available at Hiring.Cafe for anyone to use.
## Powerful Filters
The platform comes with powerful filters that allow you to:
– Select multiple job titles and job functions (and even exclude them)
– Filter out or restrict to particular industries and sectors
– Select IC vs Management roles, and for each option, select your desired years of experience
– And much more
## Open Sourcing the ChatGPT Prompt
I’ve also open-sourced my ChatGPT prompt, so if you’re curious and want to contribute, you can check it out here.
## The Future of Job Search
My goal is ambitious: to scrape every job on earth and put it online before I graduate from my PhD. And with ChatGPT by my side, I’m confident that I can make it happen.
—