Scrape web using Scrapy FrameWork

less than 1 minute read

Scrape job using Scrapy


An open source and collaborative framework for extracting the data you need from websites.

Using scrapy you can scrape data and save it in the any format you wish to. For demo, i have scraped and searched python jobs in Hyderabad. Xpath understanding will be handy while defining the selectors. You can setup a path to save scraped files. Scrapy allows you to deploy the job easily using scrapyd, it is very intutive and userfriendly.

Please refer demo github repo for details. In this demo, i have scraped for python job and saved the scraped file in json and html format. I have also included steps for deploying using scrapyd. Feel free to add comments or suggestions.

Happy learning.

Leave a comment