Raj Nath Sah

Raj Nath Sah

Another developer with interest in Database, Python, Analytics, Automation and Big Data.

Scrape web using Scrapy FrameWork

less than 1 minute read

Scrape Naukri.com job using Scrapy

Scrapy

An open source and collaborative framework for extracting the data you need from websites.

Using scrapy you can scrape data and save it in the any format you wish to. For demo, i have scraped naukri.com and searched python jobs in Hyderabad. Xpath understanding will be handy while defining the selectors. You can setup a path to save scraped files. Scrapy allows you to deploy the job easily using scrapyd, it is very intutive and userfriendly.

Please refer demo github repo for details. In this demo, i have scraped naukri.com for python job and saved the scraped file in json and html format. I have also included steps for deploying using scrapyd. Feel free to add comments or suggestions.

Happy learning.

Leave a comment

You may also enjoy

Bulk Data Load in Redshift using Pentaho


less than 1 minute read

Redshift database is a cloud based, data warehouse solution offered by Amazon. It provides storage system that stores petabytes of data in easy-to-access clu...

Q&A for Python interview


20 minute read

Important topics in Q&A format is based on my experience. Feel free to update me in case of any observation or you want to add anything.

Automate airflow scheduler log cleanup


less than 1 minute read

Airflow Airflow is a platform created by the community to programmatically author, schedule and monitor workflows.

Remote command execution over SSH using Python


3 minute read

SSH SSH or Secure Shell is a cryptographic network protocol for operating network services securely over an unsecured network.[1] Typical applications includ...