input_processor called only once

[dolohow]

Hi guys,

I am using input_processor on custom ItemLoader along one inside Field declaration that belongs to Item which is being used by ItemLoader. The problem is that only ItemLoader fires up the input_processor. Is it a desired behavior?

Related code:

class ProductsItem(scrapy.Item):
    # input_loader is not called
    currency = scrapy.Field(input_processor=MapCompose(utils.get_unified_currency_name))

class CustomProductLoader(ItemLoader):
    default_output_processor = TakeFirst()
    # input_loader works fine here
    currency_in = MapCompose(lambda x: x[0])

class MySpider(scrapy.Spider):
    # ....
    def parse(self, response):
        for product in response.css('bla bla'):
            loader = CustomProductLoader(ProductsItem(), product, response=response)
            loader.add_css('currency', 'bla bla')

[redapple]

Hi @dolohow , I'm having trouble understanding the behavior you are describing.
Could you show the behavior in a scrapy or Python shell session, with item loader calls and comments on what you expected vs. what you see as outputs?

[dolohow]

@redapple I work around this by creating a custom pipeline for validation.

get_unified_currency_name is not called on a scrapy.Field currency. Is like one input_processor overrides the other one. Maybe this is a desired behaviour, but I was thinking that it could execute both of them:

# first one that is called
currency_in = MapCompose(lambda x: x[0])

# second one that is not called
currency = scrapy.Field(input_processor=MapCompose(utils.get_unified_currency_name))

but if you remove the currency_in = MapCompose(lambda x: x[0]), utils.get_unified_currency_name) will be called.

[Gallaecio]

I think it is indeed intended behavior, and I’m personally not sure it would be wise to implement a way to change that behavior.