Extract data from Website using Selenium

Extract data from website using selenium

Extracting data from website is termed as web scrapping.we can use selenium and python for web scrapping.

1.First we have to import all required modules from selenium as written below

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
from selenium.webdriver.support.select import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.action_chains import ActionChains
from selenium.common.exceptions import NoSuchElementException

2. Download chrome webdriver or write code as given below anew chrome window will open.

driver = webdriver.Chrome('location of driver')
or
driver = webdriver.Chrome()

3.Mention the url you want to scrape

driver.get('url')

4.Declare list in which data need to store.

DaylengthMin=[]
DaylengthMax=[]
Tilt=[]

5. Inspect element which data we have to capture by right click on that element and store in given list.Simmilarly we can do for all elements we want to extract from web page

  f1=driver.find_elements(By.ID,'min')
for e in f1:
DaylengthMin.append(e.text)

6.Then store in data frame and export in csv or excel

df=pd.DataFrame({'DaylengthMax':DaylengthMax,'DaylengthMin':DaylengthMin,'Tilt':Tilt})
df.to_csv('filename.csv',index=False)