Add extraction filter

[MaxDall]

This PR implements an extraction filter not to be confused with article filters #181.

Problem statement:
With #155 the completed attribute of Article got removed leaving Fundus without the functionality to filter on extraction results.

Solution:
This PR addresses this issue by adding a new parameter to the Crawler.crawl() method and the Scraper called only_complete respectively extraction_filter.

• only_complete acts as a boolean flag and filter parameter at the same time giving users the possibility to only retrieve articles where all or partial extracted attributes' boolean value evaluates to True. The default value is False.
• extraction_fiilter being a parameter for a slightly more low level object only excepts callables as a value specified through an ExtractionFilter protocol.

Example usage:

from typing import Dict, Any

from fundus import PublisherCollection, Crawler, Requires

crawler = Crawler(PublisherCollection.de.MDR)

# extracted values for all retrieved articles evaluate to True
for article in crawler.crawl(max_articles=2, only_complete=True):
    print(article)

# values named `body` and `title` are required in the extraction and have to evaluate to True
for article in crawler.crawl(max_articles=2, only_complete=Requires("body", "title")):
    assert article.body and article.title

# default case, no restrictions given, all articles are retrieved
for article in crawler.crawl(max_articles=2, only_complete=False):
    assert article.authors


# custom filter function
def custom_filter(extracted: Dict[str, Any]) -> bool:
    keys = list(extracted.keys())
    if "author" in keys and "authors" in keys:
        return False
    return True


for article in crawler.crawl(max_articles=2, only_complete=custom_filter):
    print(article)

[dobbersc]

Cool feature and very flexible. I like how you have implemented this :).

[dobbersc]

Also, since you have given some nice examples in the PR description, wouldn't it be good to add them as tests as well?

[MaxDall]

Also, since you have given some nice examples in the PR description, wouldn't it be good to add them as tests as well?

Hmm. Adding them as they are would bring back external dependencies to the unit tests and I want to avoid this. Adding them without the external dependency will be really complicated at least.

[dobbersc]

Also, since you have given some nice examples in the PR description, wouldn't it be good to add them as tests as well?

Hmm. Adding them as they are would bring back external dependencies to the unit tests and I want to avoid this. Adding them without the external dependency will be really complicated at least.

Yes, that is a problem. One way would be to mock a custom Source for the Scraper that yields articles from a resource folder.