Metadata-Version: 2.4
Name: Protego
Version: 0.7.0
Summary: Pure-Python robots.txt parser with support for modern conventions
Project-URL: Homepage, https://github.com/scrapy/protego
Project-URL: Source, https://github.com/scrapy/protego
Project-URL: Tracker, https://github.com/scrapy/protego/issues
Project-URL: Release notes, https://github.com/scrapy/protego/blob/master/CHANGELOG.rst
Author-email: Anubhav Patel <anubhavp28@gmail.com>
License-Expression: BSD-3-Clause
License-File: LICENSE
Keywords: parser,rep,robots,robots.txt
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/x-rst

=======
Protego
=======

.. image:: https://img.shields.io/pypi/pyversions/protego.svg
   :target: https://pypi.python.org/pypi/protego
   :alt: Supported Python Versions

.. image:: https://github.com/scrapy/protego/actions/workflows/tests-ubuntu.yml/badge.svg
   :target: https://github.com/scrapy/protego/actions/workflows/tests-ubuntu.yml
   :alt: CI

Protego is a pure-Python ``robots.txt`` parser. It implements the parsing and
URL matching rules of `RFC 9309`_, and additionally supports the
``Crawl-delay``, ``Request-rate``, ``Visit-time`` and ``Host`` extensions.

Fetching ``robots.txt`` is up to you, and so are the parts of `RFC 9309`_ that
govern it, such as the handling of HTTP status codes and redirects, caching, and
imposing a parsing limit.

.. _RFC 9309: https://www.rfc-editor.org/rfc/rfc9309.html


Install
=======

To install Protego, simply use pip:

.. code-block:: none

    pip install protego


Usage
=====

.. code-block:: pycon

   >>> from protego import Protego
   >>> robotstxt = """
   ... User-agent: *
   ... Disallow: /
   ... Allow: /about
   ... Allow: /account
   ... Disallow: /account/contact$
   ... Disallow: /account/*/profile
   ... Crawl-delay: 4
   ... Request-rate: 10/1m                 # 10 requests every 1 minute
   ...
   ... Sitemap: http://example.com/sitemap-index.xml
   ... Host: http://example.co.in
   ... """
   >>> rp = Protego.parse(robotstxt)
   >>> rp.can_fetch("http://example.com/profiles", "mybot")
   False
   >>> rp.can_fetch("http://example.com/about", "mybot")
   True
   >>> rp.can_fetch("http://example.com/account", "mybot")
   True
   >>> rp.can_fetch("http://example.com/account/myuser/profile", "mybot")
   False
   >>> rp.can_fetch("http://example.com/account/contact", "mybot")
   False
   >>> rp.crawl_delay("mybot")
   4.0
   >>> rp.request_rate("mybot")
   RequestRate(requests=10, seconds=60, start_time=None, end_time=None)
   >>> list(rp.sitemaps)
   ['http://example.com/sitemap-index.xml']
   >>> rp.preferred_host
   'http://example.co.in'


Using Protego with Requests_:

.. code-block:: pycon

   >>> from protego import Protego
   >>> import requests
   >>> r = requests.get("https://google.com/robots.txt")
   >>> rp = Protego.parse(r.text)
   >>> rp.can_fetch("https://google.com/search", "mybot")
   False
   >>> rp.can_fetch("https://google.com/search/about", "mybot")
   True
   >>> list(rp.sitemaps)
   ['https://www.google.com/sitemap.xml']

.. _Requests: https://3.python-requests.org/


Comparison
==========

The following table compares Protego to the most popular ``robots.txt`` parsers
implemented in Python. Performance is the speed difference against Protego, so
a positive value means faster than Protego. It is measured over the
``robots.txt`` of 100 of the most visited websites: the time taken to check the
URLs their homepages link to, and the time taken to parse the files themselves.

.. The table below is generated by benchmarks/compare.py. Regenerate it
   rather than editing it by hand.

   Its feature rows are probe results against the exact releases named in
   "Version tested", not claims about the parsers in general. RobotFileParser
   is the one that moves: CPython taught urllib.robotparser the RFC 9309
   wildcards and longest-match precedence during the 3.14 series, after
   3.14.3. Generated on 3.14.3 or older, its wildcard and length-based
   precedence cells come out empty; generated on a later 3.14 they carry a
   checkmark. Both are what the probes measured on that interpreter, so do not
   flip those two cells by hand in either direction.

.. comparison-table-start

.. list-table::
   :header-rows: 1
   :stub-columns: 1

   * -
     - Protego
     - RobotFileParser
     - robotspy
     - Robotexclusionrulesparser
   * - Version tested
     -
     - Python 3.14.7
     - 0.13.0
     - 1.7.1
   * - Reference specification
     - `RFC 9309`_
     - `Martijn Koster's 1996 draft`_
     - `RFC 9309`_
     - `Martijn Koster's 1996 draft`_
   * - `Wildcard support`_
     - ✓
     - ✓
     - ✓
     - ✓
   * - `Length-based precedence`_
     - ✓
     - ✓
     - ✓
     -
   * - Crawl-delay
     - ✓
     - ✓
     -
     -
   * - Request-rate
     - ✓
     -
     -
     -
   * - Visit-time
     - ✓
     -
     -
     -
   * - Sitemaps
     - ✓
     - ✓
     - ✓
     - ✓
   * - Host
     - ✓
     -
     -
     -
   * - Matching performance
     -
     - -62%
     - -74%
     - -96%
   * - Parsing performance
     -
     - -71%
     - +39%
     - +56%

.. comparison-table-end

.. _Length-based precedence: https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.2
.. _Martijn Koster's 1996 draft: https://www.robotstxt.org/norobots-rfc.txt
.. _Wildcard support: https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.3


API Reference
=============

Class ``protego.Protego``:

Properties
----------

*   ``sitemaps`` {``list_iterator``} A list of sitemaps specified in
    ``robots.txt``.

*   ``preferred_host`` {string} Preferred host specified in ``robots.txt``.


Methods
-------

*   ``parse(robotstxt_body)`` Parse ``robots.txt`` and return a new instance of
    ``protego.Protego``.

*   ``can_fetch(url, user_agent)`` Return True if the user agent can fetch the
    URL, otherwise return ``False``.

    *user_agent* may be a product token, such as ``"mybot"``, or a whole
    ``User-Agent`` header value, such as ``"Mozilla/5.0 (compatible;
    mybot/1.0)"``; a group applies when its product token appears in
    *user_agent* at a token boundary.

*   ``crawl_delay(user_agent)`` Return the crawl delay specified for the user
    agent as a float. If nothing is specified, return ``None``.

*   ``request_rate(user_agent)`` Return the request rate specified for the user
    agent as a named tuple ``RequestRate(requests, seconds, start_time,
    end_time)``. If nothing is specified, return ``None``.

*   ``visit_time(user_agent)`` Return the visit time specified for the user
    agent as a named tuple ``VisitTime(start_time, end_time)``.
    If nothing is specified, return ``None``.
