爬蟲工作的三個(gè)基本步驟： 爬取網(wǎng)頁、解析內(nèi)容、存儲數(shù)據(jù)

準(zhǔn)備

先安裝爬取網(wǎng)頁需要用到的第三方庫：?requests 和 bs4

pip install requests

pip install bs4

爬取網(wǎng)頁

            
              # coding: UTF-8
import requests

link = "http://www.santostang.com/"
headers = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'}
r = requests.get(link, headers=headers)
print(r.text)
            
          

程序運(yùn)行后輸出 網(wǎng)頁的html代碼

解析網(wǎng)頁內(nèi)容

            
              # coding: UTF-8
import requests
from bs4 import BeautifulSoup

link = "http://www.santostang.com/"
headers = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'}
r = requests.get(link, headers=headers)

soup = BeautifulSoup(r.text, "lxml")
title = soup.find("h1", class_="post-title").a.text.strip()
print(title)
            
          

獲取到了網(wǎng)頁第一篇文章的 title，輸出內(nèi)容為：

第四章 – 4.3 通過selenium 模擬瀏覽器抓取

存儲數(shù)據(jù)

            
              # coding: UTF-8
import requests
from bs4 import BeautifulSoup

link = "http://www.santostang.com/"
headers = {'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'}
r = requests.get(link, headers=headers)

soup = BeautifulSoup(r.text, "lxml")
title = soup.find("h1", class_="post-title").a.text.strip()

with open('d:/title.txt', 'w') as f:
	f.write(title)
            
          

運(yùn)行程序后找到d:/title.txt 文件，發(fā)現(xiàn)文件的內(nèi)容就是網(wǎng)頁第一篇文章的title，即 “第四章 – 4.3 通過selenium 模擬瀏覽器抓取”

至此，講解完了python爬蟲的三個(gè)基本步驟和代碼實(shí)現(xiàn)

本文內(nèi)容到此結(jié)束，更多內(nèi)容可關(guān)注公眾號和個(gè)人微信號：