大佬教程收集整理的这篇文章主要介绍了从网页中抓取“隐藏”表格,大佬教程大佬觉得挺不错的,现在分享给大家,也给大家做个参考。
我正在尝试通过以下网址获取表格:https://www.agenas.gov.it/covid19/web/index.php?r=site%2Ftab2。 我尝试阅读 qith requests 和 BeautifulSoup:
from bs4 import BeautifulSoup as bs
import requests
s = requests.session()
req = s.get('https://www.agenas.gov.it/covID19/web/index.PHP?r=site%2Ftab2',headers={
"User-Agent" : "Mozilla/5.0 (X11; linux x86_64) AppleWebKit/537.36 (KHTML,like Gecko) "
"Chrome/51.0.2704.103 Safari/537.36"})
soup = bs(req.content)
table = soup.find('table')
然而,我只得到表格的标题。
<table class="table">
<caption class="pl8">Ricoverati e posti letto in area non critica e terapia intensiva.</caption>
<thead>
<tr>
<th class="cella-tabella-sm align-mIDdle text-center" scope="col">Regioni</th>
<th class="cella-tabella-sm bg-blu align-mIDdle text-center" scope="col">Ricoverati in Area Non Critica</th>
<th class="cella-tabella-sm bg-blu align-mIDdle text-center" scope="col">PL in Area Non Critica</th>
<th class="cella-tabella-sm bg-blu align-mIDdle text-center" scope="col">Ricoverati in terapia intensiva</th>
<th class="cella-tabella-sm bg-blu align-mIDdle text-center" scope="col">PL in terapia Intensiva</th>
<th class="cella-tabella-sm bg-blu align-mIDdle text-center" scope="col">PL terapia Intensiva attivabili</th>
</tr>
</thead>
<tbody ID="tab2_body">
</tbody>
</table>
所以我尝试使用我认为表格所在的 URL:https://Agenas:tab2-19@www.agenas.gov.it/covID19/web/index.PHP?r =Json%2Ftab2 。 但在这种情况下,我总是得到 401 状态代码,甚至如先前请求中所示添加标头用户名和密码。例如:
requests.get('https://Agenas:tab2-19@www.agenas.gov.it/covID19/web/index.PHP?r=Json%2Ftab2',headers={'username':'Agenas','password':'tab2-19'
'user-agent': 'Mozilla/5.0 (windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/89.0.4389.90 Safari/537.36'})
知道如何解决这个问题吗? 谢谢。
headers
所需的那些“秘密”实际上嵌入在 <script>
标签中。因此,您可以将它们找出来,将它们解析为 JSON
并在 request headers
中使用。
方法如下:
import json
import re
import requests
from bs4 import BeautifulSoup
headers = {
"user-agent": "Mozilla/5.0 (X11; Linux x86_64) "
"AppleWebKit/537.36 (KHTML,like Gecko) "
"Chrome/89.0.4389.90 Safari/537.36","x-requested-with": "XMLHttpRequest",}
with requests.Session() as s:
end_point = "https://Agenas:tab2-19@www.agenas.gov.it/covid19/web/index.php?r=json%2Ftab2"
regular_page = "https://www.agenas.gov.it/covid19/web/index.php?r=site%2Ftab2"
html = s.get(regular_page,headers=headers).text
soup = BeautifulSoup(html,"html.parser").find_all("script")[-1].string
hacked_payload = json.loads(
re.search(r"headers:\s({.*}),",soup,re.S).group(1).strip()
)
headers.update(hacked_payload)
print(json.dumps(s.get(end_point,headers=headers).json(),indent=2))
输出:
[
{
"regione": "Abruzzo","dato1": "667","dato2": "1495","dato3": "89","dato4": "215","dato5": "0"
},{
"regione": "Basilicata","dato1": "164","dato2": "426","dato3": "12","dato4": "88","dato5": "13"
},and so on ...
以上是大佬教程为你收集整理的从网页中抓取“隐藏”表格全部内容,希望文章能够帮你解决从网页中抓取“隐藏”表格所遇到的程序开发问题。
如果觉得大佬教程网站内容还不错,欢迎将大佬教程推荐给程序员好友。
本图文内容来源于网友网络收集整理提供,作为学习参考使用,版权属于原作者。
如您有任何意见或建议可联系处理。小编QQ:384754419,请注明来意。