In this case, the RSS feed would allow me to pick up new job listings whenever they were posted. An RSS feed is essentially a computer-readable summary of updates that a website sends out. To my dismay, they didn’t.Ĭraigslist had an RSS feed that was publicly available for personal use. I looked up Craigslist to find out if they had a publicly available REST API. I was trying various things out to see what stuck, and my first approach was to find out how I could access Craigslist data easily. I was using Python 2.4 and an older version of Sublime text, yet the process of writing an application from scratch was truly an exhilarating experience. It wasn’t the best development environment by any measure. I had a used BenQ laptop my brother had given me when I left for college that I used for development.
The journey to build a (working) prototype So I put two and two together, and decided to build a small application using this new programming language. I didn’t know much Python, but I wanted to get my hands dirty and learn more about it. That said, I had always heard about this new “hot” programming language called Python. Most of the things I worked on in college were academic projects that involved building and parsing binary trees, computer graphics, and simple language processing models. CraigslistĪt that point in time, I had never built a fully fledged application. Craigslist is essentially the marketplace of the Internet, where you can go and find things for sale, services, community posts, and so on.
Webscraper job descriptions software#
I was in my room, furiously building some software that would help me collect, and respond to, people who were looking for software engineers on Craigslist. I tried to build an app to do it for me, and this article is about how I did it.
I didn’t like to socialize much back then, so I decided to tackle this job-finding problem the best way I knew how. I could fit everything I owned in the trunk of my 2002 Honda Civic.
Webscraper job descriptions full#
When I left for Seattle, I had a backpack full of college textbooks and some clothes. I was right out of college, and I needed a job. I was working on something that I thought was genuinely interesting and awesome. It was midnight on a Friday, my friends were out having a good time, and yet I was nailed to my computer screen typing away.