Add PlantNet classifier to improve automatic species identification from photos

[kazlauskis]

This is a follow-up from #238. @JimBacon once you have the time it would be good to add the PlantNet to the new proxy module you've built recently.

• @JimBacon Add PlantNet option.
• @JimBacon Allow initial classification from NIA automatically repeat the classification with species-specific classifiers (e.g PlantNet)
• @Vilius-Stankaitis Add PlantNet credits to the app.

[kazlauskis]

@JimBacon any chance you could find the time to add the PlantNet proxy to the API this month or in early January? We'll need to have the St Helena app ready by mid-January and David asked to include the plant identification functionality in it.

[JimBacon]

I've only one more working day this month but might be able to get on it in the first week of January unless something more urgent comes up. Thanks for the advance notice.

[JimBacon]

I've implemented the first stage of this and deployed it to a clone of iRecord, https://indicia-ai-brc-irecord.pantheonsite.io. The site is connected to the live warehouse so use training mode when testing.

You can call the PlantNet API in exactly the same way as the existing Nature Identification API but just set a parameter in the query string, _api_proxy_uri=plantnet

See https://github.com/Indicia-Team/drupal-8-modules-indicia-ai/tree/master/indicia_ai for documentation.

The next stage is to allow an initial classification to be done by NIA and then automatically repeat the classification with PlantNet if a plant has been suggested by NIA.

[kazlauskis]

@JimBacon thanks! I had a quick look, and the response is good - classifier id and version changes which is great. One odd thing that I've noticed is empty suggestions.

Identifying this daisy returns some suggestions with probability scores but nothing else:

curl --location --request POST 'https://indicia-ai-brc-irecord.pantheonsite.io/api-proxy/indicia?_api_proxy_uri=plantnet' \
--header 'Authorization: Bearer XXX' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'list=15' \
--data-urlencode 'image=https://imagesvc.meredithcorp.io/v3/mm/image?url=https%3A%2F%2Fstatic.onecms.io%2Fwp-content%2Fuploads%2Fsites%2F34%2F2020%2F10%2F21%2Fmontauk-daisy-getty-1020-2000.jpg&q=60'

The response:

{
    "classifier_id": "20099",
    "classifier_version": "2022-10-24 (6.0)",
    "suggestions": [
        {
            "probability": 0.51789,
            "taxon": "Leucanthemum vulgare",
            "taxa_taxon_list_id": "65197",
            "taxon_group_id": "89"
        },
        {
            "probability": 0.16598
        },
        {
            "probability": 0.04436,
            "taxon": "Leucanthemum lacustre x maximum = L. x superbum",
            "taxa_taxon_list_id": "179728",
            "taxon_group_id": "89"
        },
        {
            "probability": 0.02858
        },
        {
            "probability": 0.02456
        }
    ]
}

[JimBacon]

The empty suggestions is the expected result when the name proposed by the classifier does not match any name in the Indicia list.

When I try that photo, the second choice is Nipponanthemum nipponicum, a Japanese species which I am not surprised is missing from the UK Master List.

If your app is going to be recording against the UK Master List then it makes sense for us to set PlantNet to limit itself to the subset "Plants of Western Europe". We can do that by changing the PlantNet setting for Service path: Project from all to weurope in https://indicia-ai-brc-irecord.pantheonsite.io/admin/config/services/api-proxy/settings

[kazlauskis]

Thanks. To me, it feels like these empty results are better filtered out as they give nothing useful to the client but I can live with that.

If your app is going to be recording against the UK Master List then it makes sense for us to set PlantNet to limit itself to the subset "Plants of Western Europe".

Good idea.

Could you also deploy this code to the St Helena site? Unlike with the iRecord site, the species list should be open for this one I think.

[kazlauskis]

@JimBacon any chance this could be deployed to the St Helena website this week?

[JimBacon]

@kazlauskis I've deployed the modules in the current state of development to the St Helena website.

I haven't filtered out the empty results because I think it needs some thought to decide how we want to handle mismatches between species suggested by the classifier and species in our list.

[kazlauskis]

Thanks! I have tried the API with the St Helena species list, and it hasn't identified any of their local flora that I've tried. However, probably this is not an issue with our implementation; the PlantNet API isn't great at recognising their species.

[JimBacon]

Try https://upload.wikimedia.org/wikipedia/commons/5/51/Carpobrotus_edulis_flower.jpg. I get an identification of Carpobrotus edulis with probability 0.06 (fourth choice but first match in the St Helena list)

I encountered a 500 server error when I tried this initially and had to deploy a fix for the url to be identified as being for an image. I didn't experience any problems yesterday but some Drupal updates deployed this morning perhaps changed something.

[kazlauskis]

Yeah, this one works, thanks. Low probability scores render this feature for St Helena useless though. Anyways, we have done now so let's leave it at that.

The next stage is to allow an initial classification to be done by NIA and then automatically repeat the classification with PlantNet if a plant has been suggested by NIA.

I will keep this ticket open until the above is done too. No rush though.