Lots: Error when maximumLotsBidPerSupplier is set to infinity (1e9999)

[jpmckinney]

The OCDS for EU profile maps maximumLotsBidPerSupplier to 1e9999 if there is no limit to the number of lots for which a bid can be submitted.

This mapping was preferred to a new boolean field, because it reused an existing field, while preserving the logical behavior of the field (e.g. comparing this value to the number of lots to which a given bid relates). It was also preferred to setting the field to a large 32-bit integer; while, in reality, no request for tender would have that many lots, and no bid would be related to that many lots, 2,147,483,647 looks like an error, whereas 1e9999 looks deliberate (it's also a common and consistent way to overflow floating-point parsing into infinity).

However, the jsonschema library will error. It considers infinity to be a number, but not an integer:

import json
from jsonschema.validators import Draft4Validator

Draft4Validator({"type":"number"}).validate(json.loads('1e9999'))
# []
Draft4Validator({"type":"integer"}).validate(json.loads('1e9999'))
# ValidationError: '1e9999' is not of type 'integer'

The same is true in Python:

import math
import decimal

isinstance(float('inf'), int)
# False
isinstance(math.inf, int)
# False
isinstance(decimal.Decimal('Infinity'), int)
# False

We can:

1. Weaken the type of the field to allow numbers, though numbers only make sense for infinity (e.g. you can't bid on 0.3 of a lot).
2. Use a more complex oneOf rule, in which one branch is specific for infinity. In that case, we'll need to check that the Data Review Tool (and possibly others) can handle it.
3. Patch the jsonschema library (in the Data Review Tool and elsewhere) to treat infinity as a valid value for an integer field.

I prefer the third option, if this problem is specific to Python. If Java, etc. validators have the same issue, then we should opt for another option.

[jpmckinney]

Update: Same issue in plain Ruby at least:

require 'bigdecimal'

Integer === Float::INFINITY
# false
Float === Float::INFINITY
# true

Integer === BigDecimal('Infinity')
# false
Float === BigDecimal('Infinity')
# false
BigDecimal === BigDecimal('Infinity')
# true

[duncandewhurst]

I'll look into option 2.

[jpmckinney]

Sounds good

[duncandewhurst]

I added a oneOf with an enum for infinity:

"maximumLotsBidPerSupplier": {
  "title": "Maximum lots per supplier",
  "description": "The maximum number of lots that one supplier can bid on as part of this contracting process.",
  "oneOf": [
    {
      "type": [
        "integer",
        "null"
      ]
    },
    {
      "type": [
        "number"
      ],
      "enum": [
        1e9999
      ]
    }
  ]
},

Invalid data results in an error message that isn't particularly helpful. Users would need to look at the schema to understand how to resolve the error:

I think we'll need to update the custom oneOf validator in lib-cove-ocds to handle this specific case or to report the underlying errors for any use of oneOf. Alternatively, in the CoVE instance for the Risk Data Library, we implemented a field-specific custom error message for that schema's use of oneOf: https://github.com/GFDRR/rdls-cove/blob/9b69d274eea60afb612e2612a1a7cffb2815d3ee/cove_rdls/templates/cove_rdls/validation_table.html#L89-L90.

Data Review Tool results for different values:

• 4.0 (invalid)
• "Four" (invalid)
• 4 (valid)
• 1e9999 (valid)

I also tried using anyOf instead of oneOf, which resulted in the same error message, as expected. The enum could be replaced with maximum and minimum, but that won't make any difference either.

[duncandewhurst]

I think we'll need to update the custom oneOf validator in lib-cove-ocds to handle this specific case or to report the underlying errors for any use of oneOf.

@jpmckinney is this worth doing or do you want to incorporate it into open-contracting/cove-ocds#223?

[jpmckinney]

We can incorporate it into that work.