web-crawler – scrapy如何用于提取网站的链接图？

我不知道任何工具或贡献正是产生你想要的东西.你必须建立一个scrapy蜘蛛.我可以在这里解释一下必要的步骤：

>创建一个scrapy项目并生成一个默认的蜘蛛

$scrapy startproject sitegraph
$cd sitegraph
$scrapy genspider graphspider mydomain.com

>这将创建一个包含items.py文件的目录.在此文件中添加以下行

from scrapy.item import Item,Field

class SitegraphItem(Item):
     url=Field()
     linkedurls=Field()

>在spiders目录中你会发现graphspider.py替换它(当然mydomain.com需要替换)：

from scrapy.SELEctor import HtmlXPathSELEctor
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.contrib.spiders import CrawlSpider,Rule
from scrapy.utilS.Url import urljoin_rfc
from sitegraph.items import SitegraphItem

class GraphspiderSpider(CrawlSpider):
    name = 'graphspider'
    allowed_domains = ['mydomain.com']
    start_urls = ['http://mydomain/index.html']

    rules = (
        Rule(SgmlLinkExtractor(allow=r'/'),callBACk='parse_item',follow=TruE),)

    def parse_item(self,responsE):
        hxs = HtmlXPathSELEctor(responsE)
        i = SitegraphItem()
        I['url'] = response.url
        I['http_status'] = response.status
        llinks=[]
        for anchor in hxs.SELEct('//a[@href]'):
            href=anchor.SELEct('@href').extract()[0]
            if not href.lower().startswith("javascript"):
        llinks.append(urljoin_rfc(response.url,href))
        I['linkedurls'] = llinks
        return i

>然后编辑setTings.py文件并添加(相应地更改文件名)：

FEED_FORMAT="jsonlines"
FEED_URI="file:///tmp/sitegraph.json"

>现在你可以运行：

$scrapy crawl graphspider

>这将生成一个json文件,您可以使用它来构建图形.

您可以使用像networkx这样的软件包来分析它的pygraphviz来绘制它(不建议用于大型网站)

import json
import pygraphviz as pg

def loadgraph(fName):
        G=pg.AGraph(directed=TruE)
        for line in open(fName):
            j=json.loads(linE)
            url=j["url"]
            G.add_node(url)
            for linked_url in j["linkedurls"]:
                G.add_edge(url,linked_url)
        return G

if __name__=='__main__':
        G=loadgraph("/tmp/sitegraph.json")
        G.layout(prog='dot')
        G.draw("sitegraph.png")