您现在的位置是:首页 > 技术笔记 网站首页技术笔记

No72:Scrapy多个spider时指定pipeline

爬虫  
简介Scrapy存在多个爬虫的时候如何指定对应的管道呢。

1、在pipeline里判断爬虫

  • settings.py
ITEM_PIPELINES = {
        "xxxx.pipelines.MyPipeline": 300,
}
  • OneSpider.py
class OneSpider(scrapy.spiders.Spider):
    name = "one"
  • TwoSpider.py
class TwoSpider(scrapy.spiders.Spider):
    name = "two"
  • pipelines.py
class MyPipeline(object):
    def process_item(self, item, spider):

        if spider.name == "one":
            print("one")
        elif spider.name == "two":
            print("two")

        return item

2、在爬虫里设置pipeline(scrapy版本必须是1.1以上)

  • settings.py
ITEM_PIPELINES = {
        "xxxx.pipelines.OneSpiderPipeline": 300,
        "xxxx.pipelines.TwoSpiderPipeline": 400,
}
  • OneSpider.py
class OneSpider(scrapy.Spider):
    name = "one"
    custom_settings = {
            "ITEM_PIPELINES": {"xxxx.pipelines.OneSpiderPipeline": 300},
    }
  • TwoSpider.py
class TwoSpider(scrapy.Spider):
    name = "two"
    custom_settings = {
            "ITEM_PIPELINES": {"xxxx.pipelines.TwoSpiderPipeline": 400},
    }
  • pipelines.py
class OneSpiderPipeline(object):
    def process_item(self, item, spider):
        print("one")

        return item


class TwoSpiderPipeline(object):
    def process_item(self, item, spider):
        print("two")

        return item