Web scraping with python

  1. Scrapy
  2. Requests
  3. urllib
  4. Beautiful soup
  5. Selenium etc.

Beautiful soup

How to install it?

pip install beautifulsoup4

How to use it?

  1. Get the HTML/XML page first
  2. Then use beautiful soup
from bs4 import BeautifulSoup
import requests
pageget = requests.get("samplesite.com/….")
bsoup = BeautifulSoup(pageget.text, 'html.parser')
links = bsoup.find_all('a')
list = []
for x in links:
list.append( x.get('href'))
print(list)

Reference:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store