Scrapy at a glance

1
2
3
4
5
6
7
8
9
10
11
$ systeminfo | findstr /B /C:"OS 名称" /C:"OS 版本"
OS 名称:          Microsoft Windows 10 企业版
OS 版本:          10.0.17763 暂缺 Build 17763
$ conda env list
# conda environments:
#
base                     D:\Anaconda3
bunnies               *  D:\Anaconda3\envs\bunnies
crawler                  D:\Anaconda3\envs\crawler
$ (crawler) G:\Python\jupyter>scrapy version
Scrapy 1.6.0

Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival.

In order to show you what Scrapy brings to the table, we’ll walk you through an example of a Scrapy Spider using the simplest way to run a spider.

Here’s the code for a spider that scrapes famous quotes from website http://quotes.toscrape.com, following the pagination:

1
2
3
4
5
6
7
8
9
10
11
12
13
<div class="quote" itemscope="" itemtype="http://schema.org/CreativeWork">
        <span class="text" itemprop="text">“A day without sunshine is like, you know, night.”</span>
        <span>by <small class="author" itemprop="author">Steve Martin</small>
        <a href="/author/Steve-Martin">(about)</a>
        </span>
        <div class="tags">
            Tags:
            <meta class="keywords" itemprop="keywords" content="humor,obvious,simile"> 
            <a class="tag" href="/tag/humor/page/1/">humor</a>
            <a class="tag" href="/tag/obvious/page/1/">obvious</a>
            <a class="tag" href="/tag/simile/page/1/">simile</a>
        </div>
</div>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/tag/humor/',
    ]

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').extract_first(),
                'author': quote.xpath('span/small/text()').extract_first(),
            }

        next_page = response.css('li.next a::attr("href")').extract_first()
        if next_page is not None:
            yield response.follow(next_page, self.parse)

Put this in a text file, name it to something like quotes_spider.py and run the spider using the runspider command:

1
scrapy runspider quotes_spider.py -o quotes.json

When this finishes you will have in the quotes.json file a list of the quotes in JSON format, containing text and author, looking like this (reformatted here for better readability):

That will generate an quotes.json file containing all scraped items, serialized in JSON.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[
{"text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d", "author": "Jane Austen"},
{"text": "\u201cA day without sunshine is like, you know, night.\u201d", "author": "Steve Martin"},
{"text": "\u201cAnyone who thinks sitting in church can make you a Christian must also think that sitting in a garage can make you a car.\u201d", "author": "Garrison Keillor"},
{"text": "\u201cBeauty is in the eye of the beholder and it may be necessary from time to time to give a stupid or misinformed beholder a black eye.\u201d", "author": "Jim Henson"},
{"text": "\u201cAll you need is love. But a little chocolate now and then doesn't hurt.\u201d", "author": "Charles M. Schulz"},
{"text": "\u201cRemember, we're madly in love, so it's all right to kiss me anytime you feel like it.\u201d", "author": "Suzanne Collins"},
{"text": "\u201cSome people never go crazy. What truly horrible lives they must lead.\u201d", "author": "Charles Bukowski"},
{"text": "\u201cThe trouble with having an open mind, of course, is that people will insist on coming along and trying to put things in it.\u201d", "author": "Terry Pratchett"},
{"text": "\u201cThink left and think right and think low and think high. Oh, the thinks you can think up if only you try!\u201d", "author": "Dr. Seuss"},
{"text": "\u201cThe reason I talk to myself is because I\u2019m the only one whose answers I accept.\u201d", "author": "George Carlin"},
{"text": "\u201cI am free of all prejudice. I hate everyone equally. \u201d", "author": "W.C. Fields"},
{"text": "\u201cA lady's imagination is very rapid; it jumps from admiration to love, from love to matrimony in a moment.\u201d", "author": "Jane Austen"}
]

For historic reasons, Scrapy appends to a given file instead of overwriting its contents. If you run this command twice without removing the file before the second time, you’ll end up with a broken JSON file.

You can also use other formats, like JSON Lines:

1
scrapy crawl quotes -o quotes.jl

The JSON Lines format is useful because it’s stream-like, you can easily append new records to it. It doesn’t have the same problem of JSON when you run twice. Also, as each record is a separate line, you can process big files without having to fit everything in memory, there are tools like JQ to help doing that at the command-line.

Things that are good to know

Scrapy is written in pure Python and depends on a few key Python packages (among others):

  • lxml, an efficient XML and HTML parser
  • parsel, an HTML/XML data extraction library written on top of lxml,
  • w3lib, a multi-purpose helper for dealing with URLs and web page encodings
  • twisted, an asynchronous networking framework
  • cryptography and pyOpenSSL, to deal with various network-level security needs

Scrapy Tutorial

Creating 1st project

Before you start scraping, you will have to set up a new Scrapy project.

Enter a directory where you’d like to store your code and run:

1
scrapy startproject rebirth

This will create a rebirth directory with the following contents:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ cd rebirth
$ tree /F
G:.
│  scrapy.cfg			# deploy configuration file
│
└─rebirth				# project's Python module, you'll import your code from here
    │  items.py			# project items definition file
    │  middlewares.py	# project middlewares file
    │  pipelines.py 	# project pipelines file
    │  settings.py		# project settings file
    │  __init__.py		
    │
    ├─spiders			# a directory where you'll later put your spiders
    │  │  __init__.py
    │  │
    │  └─__pycache__
    └─__pycache__

Our 1st Spider

Spiders are classes that you define and that Scrapy uses to scrape information from a website (or a group of websites). They must subclass scrapy.Spider and define the initial requests to make, optionally how to follow links in the pages, and how to parse the downloaded page content to extract data.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# rebirth/spiders
import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def start_requests(self):
        urls = [
            'http://quotes.toscrape.com/page/1/',
            'http://quotes.toscrape.com/page/2/',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        page = response.url.split("/")[-2]
        filename = 'quotes-%s.html' % page
        with open(filename, 'wb') as f:
            f.write(response.body)
        self.log('Saved file %s' % filename)

As you can see, our Spider subclasses scrapy.Spider and defines some attributes and methods:

  • name: identifies the Spider. It must be unique within a project, that is, you can’t set the same name for different Spiders.

  • start_requests(): must return an iterable of Requests (you can return a list of requests or write a generator function) which the Spider will begin to crawl from. Subsequent requests will be generated successively from these initial requests.

  • parse(): a method that will be called to handle the response downloaded for each of the requests made. The response parameter is an instance of TextResponse that holds the page content and has further helpful methods to handle it.

    The parse() method usually parses the response, extracting the scraped data as dicts and also finding new URLs to follow and creating new requests (Request) from them.

Run the spider using scrapy crawl quotes,this command runs the spider with name quotes that we’ve just added, that will send some requests for the quotes.toscrape.com domain. You will get a file tree similar to this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
G:.
│  quotes-1.html
│  quotes-2.html
│  scrapy.cfg
│
└─rebirth
    │  items.py
    │  middlewares.py
    │  pipelines.py
    │  settings.py
    │  __init__.py
    │
    ├─spiders
    │  │  quotes_spider.py
    │  │  __init__.py
    │  │
    │  └─__pycache__
    │          quotes_spider.cpython-37.pyc
    │          __init__.cpython-37.pyc
    │
    └─__pycache__
            settings.cpython-37.pyc
            __init__.cpython-37.pyc

Instead of implementing a start_requests() method that generates scrapy.Request objects from URLs, you can just define a start_urls class attribute with a list of URLs. This list will then be used by the default implementation of start_requests() to create the initial requests for your spider:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
        'http://quotes.toscrape.com/page/2/',
    ]

    def parse(self, response):
        page = response.url.split("/")[-2]
        filename = 'quotes-%s.html' % page
        with open(filename, 'wb') as f:
            f.write(response.body)

The parse() method will be called to handle each of the requests for those URLs, even though we haven’t explicitly told Scrapy to do so. This happens because parse() is Scrapy’s default callback method, which is called for requests without an explicitly assigned callback.

Extracting data

The best way to learn how to extract data with Scrapy is trying selectors using the shell Scrapy shell. Run:

1
scrapy shell "http://quotes.toscrape.com/page/1/" # For Win

You will get:

1
2
3
4
5
6
7
8
9
10
11
12
13
[s] Available Scrapy objects:
[s]   scrapy     scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s]   crawler    <scrapy.crawler.Crawler object at 0x0000024E0CDE0C50>
[s]   item       {}
[s]   request    <GET http://quotes.toscrape.com/page/1/>
[s]   response   <200 http://quotes.toscrape.com/page/1/>
[s]   settings   <scrapy.settings.Settings object at 0x0000024E0CDE0A20>
[s]   spider     <DefaultSpider 'default' at 0x24e0d081278>
[s] Useful shortcuts:
[s]   fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
[s]   fetch(req)                  Fetch a scrapy.Request and update local objects
[s]   shelp()           Shell help (print this help)
[s]   view(response)    View response in a browser

Using the shell, you can try selecting elements using CSS with the response object:

1
2
3
4
5
6
7
8
In [1]: response.css('title')
Out[1]: [<Selector xpath='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>]
In [2]: response.css('title').extract()
Out[2]: ['<title>Quotes to Scrape</title>']
In [3]: response.css('title::text')
Out[3]: [<Selector xpath='descendant-or-self::title/text()' data='Quotes to Scrape'>]
In [4]: response.css('title::text').extract()
Out[4]: ['Quotes to Scrape']

As an alternative, you could’ve written:

1
2
3
4
In [5]: response.css('title::text')[0].extract()
Out[5]: 'Quotes to Scrape'
In [6]: response.css('title::text').extract_first()
Out[6]: 'Quotes to Scrape'

Using .extract_first() avoids an IndexError and returns None when it doesn’t find any element matching the selection.

You can also use the re()method to extract using regular expressions:

1
2
3
4
5
6
In [7]: response.css('title::text').re(r'Quotes.*')
Out[7]: ['Quotes to Scrape']
In [8]: response.css('title::text').re(r'Q\w+')
Out[8]: ['Quotes']
In [9]: response.css('title::text').re(r'(\w+) to (\w+)')
Out[9]: ['Quotes', 'Scrape']
1
2
3
4
5
for quote in response.css("div.quote"):
...     text = quote.css("span.text::text").extract_first()
...     author = quote.css("small.author::text").extract_first()
...     tags = quote.css("div.tags a.tag::text").extract()
...     print(dict(text=text, author=author, tags=tags))

You can also use the xpath method to extract using regular expressions:

Each quote in http://quotes.toscrape.com is represented by HTML elements that look like this:

Emmet:

1
div.quote>span.text{"The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking."}+(span{by}>small.author+a[href="#"]{(about)})+div.tags{Tags:}>a.tag[href="#"]{$$$}*4
1
2
3
4
5
6
7
8
9
10
<div class="quote">
	<span class="text">"The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking."</span>
	<span>by<small class="author"></small><a href="#">(about)</a></span>
	<div class="tags">Tags:
		<a href="#" class="tag">001</a>
		<a href="#" class="tag">002</a>
		<a href="#" class="tag">003</a>
		<a href="#" class="tag">004</a>
	</div>
</div>
1
2
3
4
5
6
In [10]: response.xpath('//div[@class="quote"]//small/text()').extract_first()
Out[10]: 'Albert Einstein'
In [11]: response.xpath('//span[@class="text"]/text()').extract_first()
Out[11]: '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'
In [12]: response.xpath('//div[@class="quote"][1]//a[@class="tag"]/text()').extract()
Out[12]: ['change', 'deep-thoughts', 'thinking', 'world']

Let’s say, instead of just scraping the stuff from the first two pages from http://quotes.toscrape.com, you want quotes from all the pages in the website.

For that, Scrapy supports a CSS extension that let’s you select the attribute contents, like this:

1
2
3
4
5
6
7
8
In [13]: response.css('li.next a::attr(href)').extract_first()
Out[13]: '/page/2/'
In [14]: response.xpath('//li[@class="next"]/a/@href').extract_first()
Out[14]: '/page/2/'
In [15]: response.css('li.next a')[0]
Out[15]: <Selector xpath="descendant-or-self::li[@class and contains(concat(' ', normalize-space(@class), ' '), ' next ')]/descendant-or-self::*/a" data='<a href="/page/2/">Next <span aria-hidde'>
In [16]: response.follow(response.css('li.next a')[0])
Out[16]: <GET http://quotes.toscrape.com/page/2/>

Let’s see now our spider modified to recursively follow the link to the next page, extracting data from it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
    ]

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').extract_first(),
                'author': quote.css('small.author::text').extract_first(),
                'tags': quote.css('div.tags a.tag::text').extract(),
            }

        next_page = response.css('li.next a::attr(href)').extract_first()
        if next_page is not None:
            next_page = response.urljoin(next_page)
            yield scrapy.Request(next_page, callback=self.parse)

As a shortcut for creating Request objects you can use response.follow:

1
2
3
next_page = response.css('li.next a::attr(href)').extract_first()
if next_page is not None:
	yield response.follow(next_page, callback=self.parse)

Unlike scrapy.Request, response.follow supports relative URLs directly - no need to call urljoin. Note that response.follow just returns a Request instance; you still have to yield this Request.

You can also pass a selector to response.follow instead of a string; this selector should extract necessary attributes:

1
2
for href in response.css('li.next a::attr(href)'):
    yield response.follow(href, callback=self.parse)

For <a> elements there is a shortcut: response.follow uses their href attribute automatically. So the code can be shortened further:

1
2
for a in response.css('li.next a'):
	yield response.follow(a, callback=self.parse)

More simplicity:

1
response.follow(response.css('li.next a')[0],callback=self.parse)

In this example, it creates a sort of loop, following all the links to the next page until it doesn’t find one – handy for crawling blogs, forums and other sites with pagination.

Scraping author information

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import scrapy

class AuthorSpider(scrapy.Spider):
    name = 'author'

    start_urls = ['http://quotes.toscrape.com/']

    def parse(self, response):
        # follow links to author pages
        for href in response.css('.author + a::attr(href)'):
            yield response.follow(href, self.parse_author)

        # follow pagination links
        for href in response.css('li.next a::attr(href)'):
            yield response.follow(href, self.parse)

    def parse_author(self, response):
        def extract_with_css(query):
            return response.css(query).extract_first().strip()

        yield {
            'name': extract_with_css('h3.author-title::text'),
            'birthdate': extract_with_css('.author-born-date::text'),
            'bio': extract_with_css('.author-description::text'),
        }

scrapy crawl author -o author.json:

1
2
3
[
{"name": "Jane Austen", "birthdate": "December 16, 1775", "bio": "Jane Austen was an English novelist whose works of romantic fiction, set among the landed gentry, earned her a place as one of the most widely read writers in English literature, her realism and biting social commentary cementing her historical importance among scholars and critics.Austen lived her entire life as part of a close-knit family located on the lower fringes of the English landed gentry. She was educated primarily by her father and older brothers as well as through her own reading. The steadfast support of her family was critical to her development as a professional writer. Her artistic apprenticeship lasted from her teenage years until she was about 35 years old. During this period, she experimented with various literary forms, including the epistolary novel which she tried then abandoned, and wrote and extensively revised three major novels and began a fourth. From 1811 until 1816, with the release of Sense and Sensibility (1811), Pride and Prejudice (1813), Mansfield Park (1814) and Emma (1815), she achieved success as a published writer. She wrote two additional novels, Northanger Abbey and Persuasion, both published posthumously in 1818, and began a third, which was eventually titled Sanditon, but died before completing it.Austen's works critique the novels of sensibility of the second half of the 18th century and are part of the transition to 19th-century realism. Her plots, though fundamentally comic, highlight the dependence of women on marriage to secure social standing and economic security. Her work brought her little personal fame and only a few positive reviews during her lifetime, but the publication in 1869 of her nephew's A Memoir of Jane Austen introduced her to a wider public, and by the 1940s she had become widely accepted in academia as a great English writer. The second half of the 20th century saw a proliferation of Austen scholarship and the emergence of a Janeite fan culture."},
....]

The interesting thing this spider demonstrates is that, even if there are many quotes from the same author, we don’t need to worry about visiting the same author page multiple times. By default, Scrapy filters out duplicated requests to URLs already visited, avoiding the problem of hitting servers too much because of a programming mistake. This can be configured by the settingDUPEFILTER_CLASS.

Using spider arguments

You can provide command line arguments to your spiders by using the -a option when running them:

1
scrapy crawl quotes -o quotes-humor.json -a tag=humor

These arguments are passed to the Spider’s __init__ method and become spider attributes by default.

In this example, the value provided for the tag argument will be available via self.tag. You can use this to make your spider fetch only quotes with a specific tag, building the URL based on the argument:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def start_requests(self):
        url = 'http://quotes.toscrape.com/'
        tag = getattr(self, 'tag', None)
        if tag is not None:
            url = url + 'tag/' + tag
        yield scrapy.Request(url, self.parse)

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').extract_first(),
                'author': quote.css('small.author::text').extract_first(),
            }

        next_page = response.css('li.next a::attr(href)').extract_first()
        if next_page is not None:
            yield response.follow(next_page, self.parse)

If you pass the tag=humor argument to this spider, you’ll notice that it will only visit URLs from the humor tag, such as http://quotes.toscrape.com/tag/humor.

1
2
3
[{"text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d", "author": "Albert Einstein"},
{"text": "\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d", "author": "J.K. Rowling"},
...

By now, we have two spiders:

1
2
3
$ scrapy list
author
quotes

Command line tool

Scrapy is controlled through the scrapy command-line tool, to be referred here as the “Scrapy tool” to differentiate it from the sub-commands, which we just call “commands” or “Scrapy commands”.

Configuration settings

Scrapy will look for configuration parameters in ini-style scrapy.cfg files in standard locations:

  1. /etc/scrapy.cfg or c:\scrapy\scrapy.cfg (system-wide),
  2. ~/.config/scrapy.cfg ($XDG_CONFIG_HOME) and ~/.scrapy.cfg ($HOME) for global (user-wide) settings, and
  3. scrapy.cfg inside a scrapy project’s root

Settings from these files are merged in the listed order of preference: user-defined values have higher priority than system-wide defaults and project-wide settings will override all others, when defined.

Available tool commands

startproject

  • Syntax: scrapy startproject <project_name> [project_dir]

Creates a new Scrapy project named project_name, under the project_dir directory. If project_dirwasn’t specified, project_dir will be the same as project_name.

Usage example:

1
$ scrapy startproject myproject

genspider

  • Syntax: scrapy genspider [-t template] <name> <domain>

Create a new spider in the current folder or in the current project’s spiders folder, if called from inside a project. The <name> parameter is set as the spider’s name, while <domain> is used to generate the allowed_domains and start_urls spider’s attributes.

Usage example:

1
2
3
4
5
6
7
8
9
10
11
12
$ scrapy genspider -l
Available templates:
  basic
  crawl
  csvfeed
  xmlfeed

$ scrapy genspider example example.com
Created spider 'example' using template 'basic'

$ scrapy genspider -t crawl scrapyorg scrapy.org
Created spider 'scrapyorg' using template 'crawl'

This is just a convenience shortcut command for creating spiders based on pre-defined templates, but certainly not the only way to create spiders. You can just create the spider source code files yourself, instead of using this command.

crawl

  • Syntax: scrapy crawl <spider>
  • Requires project: yes

Start crawling using a spider.

Usage examples:

1
2
$ scrapy crawl myspider
[ ... myspider starts crawling ... ]

check

  • Syntax: scrapy check [-l] <spider>
  • Requires project: yes

Run contract checks.

Usage examples:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ scrapy check -l
first_spider
  * parse
  * parse_item
second_spider
  * parse
  * parse_item

$ scrapy check
[FAILED] first_spider:parse_item
>>> 'RetailPricex' field is missing

[FAILED] first_spider:parse
>>> Returned 92 requests, expected 0..4

list

  • Syntax: scrapy list
  • Requires project: yes

List all available spiders in the current project. The output is one spider per line.

Usage example:

1
2
3
$ scrapy list
spider1
spider2

fetch

  • Syntax: scrapy fetch <url>
  • Requires project: no

Downloads the given URL using the Scrapy downloader and writes the contents to standard output.

The interesting thing about this command is that it fetches the page how the spider would download it. For example, if the spider has a USER_AGENT attribute which overrides the User Agent, it will use that one.

So this command can be used to “see” how your spider would fetch a certain page.

Supported options:

  • --spider=SPIDER: bypass spider autodetection and force use of specific spider
  • --headers: print the response’s HTTP headers instead of the response’s body
  • --no-redirect: do not follow HTTP 3xx redirects (default is to follow them)

Usage examples:

1
2
3
4
5
6
7
8
9
10
11
12
13
$ scrapy fetch --nolog http://www.example.com/some/page.html
[ ... html content here ... ]

$ scrapy fetch --nolog --headers http://www.example.com/
{'Accept-Ranges': ['bytes'],
 'Age': ['1263   '],
 'Connection': ['close     '],
 'Content-Length': ['596'],
 'Content-Type': ['text/html; charset=UTF-8'],
 'Date': ['Wed, 18 Aug 2010 23:59:46 GMT'],
 'Etag': ['"573c1-254-48c9c87349680"'],
 'Last-Modified': ['Fri, 30 Jul 2010 15:30:18 GMT'],
 'Server': ['Apache/2.2.3 (CentOS)']}

view

  • Syntax: scrapy view <url>
  • Requires project: no

Opens the given URL in a browser, as your Scrapy spider would “see” it. Sometimes spiders see pages differently from regular users, so this can be used to check what the spider “sees” and confirm it’s what you expect.

Supported options:

  • --spider=SPIDER: bypass spider autodetection and force use of specific spider
  • --no-redirect: do not follow HTTP 3xx redirects (default is to follow them)

Usage example:

1
2
$ scrapy view http://www.example.com/some/page.html
[ ... browser starts ... ]

shell

  • Syntax: scrapy shell [url]
  • Requires project: no

Starts the Scrapy shell for the given URL (if given) or empty if no URL is given. Also supports UNIX-style local file paths, either relative with ./ or ../ prefixes or absolute file paths.

Supported options:

  • --spider=SPIDER: bypass spider autodetection and force use of specific spider
  • -c code: evaluate the code in the shell, print the result and exit
  • --no-redirect: do not follow HTTP 3xx redirects (default is to follow them); this only affects the URL you may pass as argument on the command line; once you are inside the shell, fetch(url)will still follow HTTP redirects by default.

Usage example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ scrapy shell http://www.example.com/some/page.html
[ ... scrapy shell starts ... ]

$ scrapy shell --nolog http://www.example.com/ -c '(response.status, response.url)'
(200, 'http://www.example.com/')

# shell follows HTTP redirects by default
$ scrapy shell --nolog http://httpbin.org/redirect-to?url=http%3A%2F%2Fexample.com%2F -c '(response.status, response.url)'
(200, 'http://example.com/')

# you can disable this with --no-redirect
# (only for the URL passed as command line argument)
$ scrapy shell --no-redirect --nolog http://httpbin.org/redirect-to?url=http%3A%2F%2Fexample.com%2F -c '(response.status, response.url)'
(302, 'http://httpbin.org/redirect-to?url=http%3A%2F%2Fexample.com%2F')

parse

  • Syntax: scrapy parse <url> [options]
  • Requires project: yes

Fetches the given URL and parses it with the spider that handles it, using the method passed with the --callback option, or parse if not given.

Supported options:

  • --spider=SPIDER: bypass spider autodetection and force use of specific spider
  • --a NAME=VALUE: set spider argument (may be repeated)
  • --callback or -c: spider method to use as callback for parsing the response
  • --meta or -m: additional request meta that will be passed to the callback request. This must be a valid json string. Example: –meta=’{“foo” : “bar”}’
  • --pipelines: process items through pipelines
  • --rules or -r: use CrawlSpider rules to discover the callback (i.e. spider method) to use for parsing the response
  • --noitems: don’t show scraped items
  • --nolinks: don’t show extracted links
  • --nocolour: avoid using pygments to colorize the output
  • --depth or -d: depth level for which the requests should be followed recursively (default: 1)
  • --verbose or -v: display information for each depth level

Usage example:

1
2
3
4
5
6
7
8
9
10
11
$ scrapy parse http://www.example.com/ -c parse_item
[ ... scrapy log lines crawling example.com spider ... ]

>>> STATUS DEPTH LEVEL 1 <<<
# Scraped Items  ------------------------------------------------------------
[{'name': u'Example item',
 'category': u'Furniture',
 'length': u'12 cm'}]

# Requests  -----------------------------------------------------------------
[]

settings

  • Syntax: scrapy settings [options]
  • Requires project: no

Get the value of a Scrapy setting.

If used inside a project it’ll show the project setting value, otherwise it’ll show the default Scrapy value for that setting.

Example usage:

1
2
3
4
$ scrapy settings --get BOT_NAME
scrapybot
$ scrapy settings --get DOWNLOAD_DELAY
0

runspider

  • Syntax: scrapy runspider <spider_file.py>
  • Requires project: no

Run a spider self-contained in a Python file, without having to create a project.

Example usage:

1
2
$ scrapy runspider myspider.py
[ ... spider starts crawling ... ]

version

  • Syntax: scrapy version [-v]
  • Requires project: no

Prints the Scrapy version. If used with -v it also prints Python, Twisted and Platform info, which is useful for bug reports.

REFERENCES