爬虫入门到进阶(四)

xpath解析库

1.xpath解析库

Xpath解析库介绍: 数据解析的过程中使用过正则表达式, 但正则表达式想要进准匹配难度较高, 一旦正则表达式书写错误, 匹配的数据也会出错. 网页由三部分组成: HTML, Css, JavaScript, HTML页面标签存在层级关系, 即DOM树, 在获取目标数据时可以根据网页层次关系定位标签, 再获取标签的文本或属性.

xpath解析库解析数据原理:

根据网页DOM树定位节点标签

获取节点标签的正文文本或属性值

# xpath安装, 初体验 --> 使用步骤: 
1.xpath安装: pip install lxml 
2.requests模块爬取糗事百科热门的标题: 
import requests 
from lxml import etree 
url = 'https://www.qiushibaike.com/' 
headers = { "User-Agent":'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36' }
res = requests.get(url=url, headers=headers) tree = etree.HTML(res.text) 
title_lst = tree.xpath('//ul/li/div/a/text()') for item in title_lst: 
	print(item) 3.xpath使用步骤: 
from lxml import etree 

tree = etree.HTML(res.text) 
tree = etree.parse(res.html,etree.HTMLParse()) 

# 示例如下, 了解内容 
tag_or_attr = tree.xpath('xpath表达式')

# xpath解析本地文件 
import requests 
from lxml import etree
url = 'https://www.qiushibaike.com/' 
headers = { "User-Agent":'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36' }

res = requests.get(url=url, headers=headers) with open('qb.html', 'w', encoding='utf-8') as f: 
	f.write(res.text) 
tree = etree.parse('./qb.html', etree.HTMLParser()) 
title_lst = tree.xpath('//ul/li/div/a/text()') for item in title_lst: 
	print(item)

# xpath语法: 
1.常用规则: 
	1.1 nodename: 节点名定位 
	1.2 //: 从当前节点选取子孙节点 
	1.3 /: 从当前节点选取直接子节点 
	4. nodename[@attribute="..."] 根据属性定位标签 '//div[@class="ui-main"]' 
	5. @attributename: 获取属性 
	6. text(): 获取文本 
2.属性匹配两种情况: 多属性匹配 & 单属性多值匹配 
	2.2 多属性匹配 
	示例: tree.xpath('//div[@class="item" and @name="test"]/text()') 
	2.1 单属性多值匹配 示例:
	tree.xpath('//div[contains(@class, "dc")]/text()') 
3.按序选择: 
	3.1 索引定位: 从1开始(牢记， 牢记， 牢记) 
	3.2 last()函数 
	3.3 position()函数

解析示例: 示例解析的是本地文件 
<!DOCTYPE html> 
	<html lang="en"> 
		<head><meta charset="UTF-8"> 
		<title>Xpath练习文件</title> 
		</head> 
	<body> 
		<div id="007"> "我是div标签的文字内容, 和下面的p标签还有div标签是同级的哦" 
		<p>这是p标签内的文字内容</p> 
		<div>这是p标签同级的div标签</div> 
		</div> <div class="divtag"> 
		<ul>
		<li>第1个li标签</li>
		<li>第2个li标签</li> 
		<li>第3个li标签</li> 
		<li>第4个li标签</li> 
		<li>第5个li标签</li> </ul> 
		<a href="https://www.baidu.com">这是百度的跳转连接</a> 
		</div> 
		<div class="c1" name="laoda">老大在此</div> 
		<div class="c1 c3" name="laoer">老二任性, class有两个值</div> 
		<div class="c1" name="laosan">我是老三</div> 
		</body> 
		</html>

from lxml import etree 
	tree = etree.parse('./x.html', etree.HTMLParser()) 
# 1.根据节点名, 即nodename定位title标签, 获取标签内文字 
title_text = tree.xpath('//title/text()') print(title_text) 
# 2.根据节点属性定位: 定位id为007的div标签 
div_007 = tree.xpath('//div[@id="007"]') print(div_007) 
# 3.示例直接子节点与子孙节点:/, // 
div_007_one = tree.xpath('//div[@id="007"]/text()') print(div_007_one) div_007_two = tree.xpath('//div[@id="007"]//text()') print(div_007_two) 
# 4.获取a标签的href属性 
a_href =tree.xpath('//div[@class="divtag"]/a/@href') print(a_href) 
# 4.多属性定位: 根据class属性和name属性定位div标签
 div_two_attr = tree.xpath('//div[@class="c1" and @name="laoda"]/text()') print(div_two_attr) 
 # 5.属性多值定位: 定位所有class中有c1的div标签
  div_c1 = tree.xpath('//div[contains(@class, "c1")]') 
  # 6.按序定位 
  li_first = tree.xpath('//div[@class="divtag"]/ul/li[1]/text()') 
  # 定位第一个li标 签, 获取其文本 
  print(li_first) 
  li_last = tree.xpath('//div[@class="divtag"]/ul/li[last()]/text()') 
  # 定位最后一 个li标签 
  print(li_last) li_daotwo = tree.xpath('//div[@class="divtag"]/ul/li[last()-1]/text()') 
  # 定位倒 数第二个li标签 
  print(li_daotwo)

li_qianthree = tree.xpath('//div[@class="divtag"]/ul/li[position()<4]/text()') 
# 定位前三个li标签 
print(li_qianthree)

2.案例: 豆瓣电影的相关信息

import requests 
from lxml import etree 
url = 'https://movie.douban.com/chart' headers = { 'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36' }
res = requests.get(url=url, headers=headers) tree = etree.HTML(res.text) 
td_list = tree.xpath('//tr[@class="item"]/td[2]') 
# print(td_list) 
for td in td_list: 
	title = td.xpath('./div/a//text()') 	   produce = td.xpath('./div/p/text()') 
	score =td.xpath('./div/div/span[2]/text()') 	comment_num = td.xpath('./div/div/span[3]/text()') 	print(1111111111111111111111111111111111111111111) 
	print(title) 
	print(produce) 
	print(score) 
	print(comment_num) 
print(2222222222222222222222222222222222222222222)

3.案例: 糗事百科实现多页爬取

爬取糗事百科, 热门前两页的每一项标题, 详情页链接, 好笑
指数及评论数, 多页爬取

1）.xpath常用规则: 
	1. nodename: 节点名定位 
	2. //: 从当前节点选取子孙节点 
	3. /: 从当前节点选取直接子节点 
	4. nodename[@attribute="..."] 根据属性定位标签 
	5. @attributename: 获取属性 
	6. text(): 获取文本 
2）.属性匹配两种情况: 
多属性匹配 & 单属性多值匹配 
2.2 多属性匹配 示例: tree.xpath('//div[@class="item" and @name="test"]/text()') 
2.1 单属性多值匹配 示例: tree.xpath('//div[contains(@class, "dc")]/text()') 
3）.按序选择:
3.1 索引定位: 从1开始(牢记， 牢记， 牢记) 
3.2 last()函数 
3.3 position()函数