Updates to organisation handling in OCDS

[timgdavies]

A substantial update to organisation handling is proposed for version 1.1 of OCDS.

The current situation

We currently have an organisation building block which is used for buyer, procuringEntity, tenderers and suppliers.

The block consists of:

• A name
• An address block
• An identifier block
• A contact point block

We use the identifier block for identifying the providerOrganization and receiverOrganization of a transactional payment. In this design choice there is an implicit assumption that the providerOrganization is the buyer (providing the funds), and the receieverOrganization is the supplier, so additional address information on them is not required.

An example organisation block is shown below.

{
    "name": "London Borough of Barnet",
    "address": {
        "countryName": "United Kingdom",
        "locality": "London",
        "postalCode": "N11 1NP",
        "region": "London",
        "streetAddress": "4, North London Business Park, Oakleigh Rd S"
    },
    "contactPoint": {
        "email": "[email protected]",
        "faxNumber": "01234 345 345",
        "name": "Procurement Team",
        "telephone": "01234 345 346",
        "url": "http://example.com/contact/"
    },
    "identifier": {
        "id": "E09000003",
        "legalName": "London Borough of Barnet",
        "scheme": "GB-LAC",
        "uri": "http://www.barnet.gov.uk/"
    }
}

The issue(s)

(1) There are other organizations involve in a contracting process

Some we have encountered include:

• Contract administration entity Contract administration entity #150
• Executing Entity Extension/update: different kinds of organisations involved #264
• Social Witness
• Review body
• Mediation body
• Finance provider (e.g. in Public Private Partnership)

(2) The 'buyer' and 'supplier' terminology does not work for all situations

Issue #156 suggests renaming 'buyer' to better capture the organisation who is party to the contract.

In frameworks we may have cases where:

• A contract is signed by a central body;
• The buyer of the goods in a call off from the framework is a different local agency or department;
• There are multiple buyers for one contracting process;

In non-procurement cases, we may have:

• Government as the 'seller' disposing of assets, or as the licensing agent for a concession;
• Private sector as buyers (of an asset being sold off) or concession operators / licensee for licenses

(3) Publishers want to classify organisations

Issue #181 calls for classification of organisations by different dimensions (e.g. size: micro, small, medium etc; ownership: VCS, foreign owned, minority-owned).

Issue #269 considers the EU requirement to classify the main activity of a buyer, using codes derived from COFOG.

(4) Publishers want to provide multiple contact points

Issue #266 #291 and #275 consider the demand for additional addresses or contact points to be associated with an organisation. This may be for a number of reasons:

• To provide addresses for different purposes (e.g. registered address of an organisation, address to write to in order to participate in a tender, address to write to for further information)
• To provide addresses for different language correspondence;

(5) Users want location information on organisations

At present, the proposed location extension only applies to items. Some use-cases want to understand where an organisation is registered.

(6) Our organisation building block lacks a root ID

All the other building blocks in OCDS have an id property.

In the organisation block, this is only provided at the identifier.id level. At the identifier.id level there is no guarantee of a unique, unambigious identifier - as identifier.id is a two-part identifier, consisting of identifier.scheme and identifier.id. (i.e. it is theoretically possible to have something like:

{
"tenderers":[
 {
    "identifier": {
            "id": "123456789",
            "scheme": "GB-COH",
        }
 },
 {
    "identifier": {
            "id": "123456789",
            "scheme": "US-EIN",
        }
 }]
}

where we have to identifier.id values of '123456789' but in different schemes.

Issue #361 notes the problems this complex id, nested one level down, creates for our approach to flattening data. It also has implications for data merging, and currently requires special case handling.

(7) The same organisation may appear many times in a single release

In complex framework contracts, for example, a buyer or supplier could appear many times.

In an auction situation, the same bidder may exist against many bids.

If the full organisation information is repeated each time, this could lead to a lot of redundant data.

However, there may be cases when organisation blocks would refer to the same legal entity, but would contain subtly different information. For example, a single legal entity might be both the procuringEntity and the buyer, but with a different department and contact point for each role it plays.

(8) Not all the entities we are dealing with are strictly organizations

Individuals are able to be party to a contract. And in some cases, an organization is only brought into being after a contract is signed (e.g. in Public Private Partnerships which create new Joint Purpose Vehicles).

The language or organization may not be correct in these cases.

(9) Flattening organizations involves multiple tables

A full organization block includes an array of additionalIdentifiers. This means that in our current flattening approach, a single organization array such as award.suppliers requires two tables (awa_suppliers and awa_sup_additionalIdentifiers)

If we have more kinds of organizations emerging this increases the number of tables substantially.

The proposal

We are seeking comments on a major refactor of organization handling to rationalise and future-proof the standard.

We propose:

• Adding a top-level section called entities which will consist of an array of organization objects. All organizations who are part of a contracting process in any form should be included here.
• Adding an id field to the top level of the organization object. id should be used to cross-reference to the organization whenever it occurs.
• Adding role as a field of organization with a codelist of organization roles (buyer, supplier, procuringEntity, etc.)
• Deprecating the use of full organization blocks in buyer, supplier, procuringEntity and other locations, and replacing with a block containing: id and name.
• Developing extensions to further classify organizations

In detail

The entities section will sit alongside planning, tender, award and contract. It will be an array of organization building blocks.

An id field will be added to the organization building block used within the parties section. We will recommend that this should be composed of {identifier.scheme}-{identifier.id}-{department-identifier}, although publishers will not be required to follow a specific pattern.

organization.role will be use an open codelist, containing initially:

Code	Name	Description
buyer	Buyer	The buyer is the entity whose budget will be used to purchase the goods.
procuringEntity	Procuring Entity	The entity managing the procurement, which may be different from the buyer who is paying / using the items being procured.
supplier	Supplier	The entity awarded or contracted to provide supplies, works or services.
tenderer	Tenderer	All entities who submit a tender

All organizations that occur in a release must be included in the entities section.
Wherever organizations occur elsewhere in the file, they must referenced by at least id and name.

Implications

Consuming applications will need to cross-reference to identifiers in the organization block. Duplication of redundant organization information will be reduced.

Flattened data could have a single entities table listing all the organizations and their role in a contracting process.

More advanced validation rules will be needed (e.g. if an organization id appears under suppliers, the entities record for the organization should include a role of ‘supplier’)

Worked example

The following example shows an abbreviated award release, with information on the buyer, tenderers and supplier awarded the contract. It also includes an additional contact point for complaints in the entities block, showing how this approach allows additional organisations or contact points to be included more easily.

{

"releases":[{
        "language": "en",
        "ocid": "ocds-213czf-1.1-upgrade",
        "id": "ocds-213czf-1.1-upgrade-06-award",
        "date": "2011-01-10T09:30:00Z",
        "tag": ["award"],
        "initiationType": "tender",
        "entities": [
            {
                "id":"GB-LAC-E09000003-557",
                "roles": ["buyer","procuringEntity"],
                "identifier": {
                    "scheme": "GB-LAC",
                    "id": "E09000003",
                    "legalName": "London Borough of Barnet",
                    "uri": "http://www.barnet.gov.uk/"
                },
                "name": "London Borough of Barnet",
                "address": {
                    "streetAddress": "4, North London Business Park, Oakleigh Rd S",
                    "locality": "London",
                    "region": "London",
                    "postalCode": "N11 1NP",
                    "countryName": "United Kingdom"
                },
                "contactPoint": {
                    "name": "Procurement Team",
                    "email": "[email protected]",
                    "telephone": "01234 345 346",
                    "faxNumber": "01234 345 345",
                    "url": "http://example.com/contact/"
                }
          },
          {
               "id":"GB-COH-1234567844",
               "roles": ["tenderer","supplier"],
               "identifier": {
                   "scheme": "GB-COH",
                   "id": "1234567844",
                   "legalName": "AnyCorp Ltd",
                   "uri": "http://www.anycorp.example"
               },
               "additionalIdentifiers": [
                   {
                       "scheme": "GB-VAT",
                       "id": "987654321"
                   }
               ],
               "name": "AnyCorp Cycle Provision",
               "address": {
                   "streetAddress": "100 Anytown Lane",
                   "locality": "Anytown",
                   "region": "AnyCounty",
                   "postalCode": "AN1 1TN",
                   "countryName": "United Kingdom"
               },
               "contactPoint": {
                   "name": "Contracts Team",
                   "email": "[email protected]",
                   "telephone": "12345 456 343",
                   "faxNumber": "12345 456 343"
               }
           },
           {
               "id":"GB-CHC-987654321",
               "roles": ["tenderer"],
               "identifier": {
                   "scheme": "GB-CHC",
                   "id": "987654321",
                   "legalName": "AnyCharity",
                   "uri": "http://www.anyCharity.example"
               },
               "name": "AnyCharity Cycle Scheme",
               "address": {
                   "streetAddress": "101 Anytown Lane",
                   "locality": "Anytown",
                   "region": "AnyCounty",
                   "postalCode": "AN1 1TN",
                   "countryName": "United Kingdom"
               },
               "contactPoint": {
                   "name": "Director",
                   "email": "[email protected]"                   
               }
           },
           {
                "id":"GB-LAC-E09000003-complaints",
                "roles": ["complaintsBody"],
                "identifier": {
                    "scheme": "GB-LAC",
                    "id": "E09000003",
                    "legalName": "London Borough of Barnet",
                    "uri": "http://www.barnet.gov.uk/"
                },
                "name": "London Borough of Barnet",
                "address": {
                    "streetAddress": "Complaints Team, 4, North London Business Park, Oakleigh Rd S",
                    "locality": "London",
                    "region": "London",
                    "postalCode": "N11 1NP",
                    "countryName": "United Kingdom"
                },
                "contactPoint": {
                    "name": "Procurement Compaints Team",
                    "email": "[email protected]"
                }
          },
         ],
         "buyer": {
            "name":"London Borough of Barnet",
            "id":"GB-LAC-E09000003-557"
         },
         "tender": {
            "id": "ocds-213czf-000-00001-01-tender",
            "title": "Planned cycle lane improvements",
            "description": "Tenders solicited for work to build new cycle lanes in the centre of town.",
            "status": "active",
            "procuringEntity": {
                "name":"London Borough of Barnet",
                "id":"GB-LAC-E09000003-557"
            },
            "tenderers":[
              {
                "name":"AnyCharity Cycle Scheme",
                "id":"GB-CHC-987654321"
              },
               {
                    "name":"AnyCorp Cycle Provision",
                    "id":"GB-COH-1234567844"
               }
            ],
            "...":"..."
        },
         "awards": [
                   {
                       "id": "ocds-213czf-000-00001-award-01",
                       "title": "Award of contract to build new cycle lanes in the centre of town.",
                       "description": "AnyCorp Ltd has been awarded the contract to build new cycle lanes in the centre of town.",
                       "status": "active",
                       "date": "2010-05-10T10:30:00Z",
                       "value": {
                           "amount": 11000000,
                           "currency": "GBP"
                       },
                       "suppliers": [
                           {
                               "name":"AnyCorp Cycle Provision",
                               "id":"GB-COH-1234567844"
                           }
                       ],
                       "...":"..."
                    }
               ],

   }]
}

Engagement

Please indicate support or opposition for this proposal using the +1 / -1 buttons or a comment. If opposing the proposal, please give clear justifications, and where possible, make an alternative proposals.

Clarifying questions are also encouraged.

[mireille-raad]

How is the new id for an entity generated?
Does it have to be unique for each entity through all the records?

[timgdavies]

We suggest that the ID could be generated from {identifier.scheme}-{identifier.id}-{department-identifier} but the standard would be agnostic as long as the cross-reference was stable within the scope of a single release.

[mireille-raad]

thanks a lot for reply!

Will Open Data Service make a migration script available for this change?

I imagine that this task is relatively easy to automate and that a script would save clients lots of headache for the OCDS upgrade. Updates to user interface and built-in reports/stats will probably stay a challenge.

[timgdavies]

I believe we could make a script available for this - yes.

[JachymHercher]

On slide 17 of the V1.1 slide deck you say that referencing to "organisation" block should be done via ID and name. Why not just id? To prevent mistakes...?

[timgdavies]

The main reason I think is that it makes simple uses of the data easier.

There are many use-cases (such as just displaying the declared name of the contractor) which don't need to the full organisation identification info.

For these cases, asking the publisher to provide the name in the data (which should be programatically accessible alongside the ID, so no heavy-lift) makes the users life a lot easier, as they don't need to perform the cross-reference lookup for these sorts of simple uses.

This also impacts on flattened spreadsheet representations of the data, where it would let users skim over a spreadsheet and see names in the main sheet where a tender or contract award is, rather than having to lookup the ID in a separate sheet to get a name.

[Bjwebb]

We had some feedback on this proposal at IODC.

One source of confusion is that "entity" is a much broader term than organisation, and is used in other context to include all physical objects e.g. pencils, which in OCDS are not entities, but items.

We also had some different feedback on how difficult the new structure would be to implement. Some publishers have a centralised system that already stores organisations independently of releases/records, so it is relatively straightforward to publish to the new format.

Other publishers have multiple systems that are responsible for publishing releases about data at different stages of the contracting process. Currently each stage of the contracting process corresponds roughly to a separate block in an OCDS release, so it is easier to merge data from these multiple systems. Outputting a common entities block at the top level is more technically difficult to implement, particularly with the need for each particular block to reference back to it.

Which OCDS merge strategy applies to the entities list? The best fit looks to be arrayMergeById, but it's worth noting this is different to how lists of organisations are currently merged.

Additionally it's not clear to me if some data about (relationships with) organisations might be different for different stages of the contracting process (e.g. the contact point may differ?), which would be overridden when merged. Even if these fields should not be different,*

There may still be differences between systems (e.g. a different writing of the name), and compiled record might alternate between these. (Which was always a risk with merging, but was mitigated somewhat by having data from different processes in different blocks).

Edit: *this should be addressed by the use of different identifiers for each department.

[timgdavies]

The naming question is an interesting one.

One alternative I've been looking at is: 'parties' c.f. parties in law.

This would need some good documentation to explain we are talking about 'parties to the contracting process' not parties to a specific contract, unless they are named as parties at that point.

Other views on nomenclature would be very welcome

[timgdavies]

I've started an extension repository to allow us to work up these changes to the schema in more detail. There are a number of open issues on that repository.

[timgdavies]

We discussed terminology in the upgrade session in Bogota yesterday.

There was a view that 'parties' as a term does not translate easily into Spanish.

The alternatives suggested were 'actors' and 'stakeholders'.

Actors was put forward as the preferred term.

[timgdavies]

Another possible term is participants

[timgdavies]

The commit here applies updates to the documentation and schema based on the terminology of 'parties' ready for review.

This terminology was carried forward from the work on the PPP extension to OCDS and will be checked with reviewers.

[myroslav]

I cannot get an idea of how proposal is covering case of multiple contact points of procuringEntity that speak in different languages. Can you elaborate?

[timgdavies]

The multiple contact points would be handled by additionalContactPoints extension. I've added this at https://github.com/open-contracting/ocds_additionalContactPoints_extension

[timgdavies]

Comment from Juan Pane during peer review:

Perhaps it could be useful to describe which attribute of the Organization object will be used for cross-referencing in the parties attribute, so that publisher note the importante of that attribute (is it going to be ID which is the Organization ID which spans across multiple releases, or is there going to be another local id, or the URI?) This might be important as the standar uses one o another approach in different attriburtes.
There seem to be a issue in the description of the parties object, it references the "Organization" object, but in the award class, the suppliers object references the "OrganizationReference" object that seems to be the Paties class, which contains different attributes from Organization class, so it is confusing as to which one is used for listing the parites that later are going to be used for reference.
Please revise the descriptions as to make them consistent for the classes used for listing and referencing

[JachymHercher]

My comment from the 1.1 peer review:

1. I have no opinion on what seems most important to me (Updates to organisation handling in OCDS #368 [7]), which is whether the benefits of reducing the repetition in the data outweigh the costs of changing the systems as well as mildly reducing the human readability of the data.

2. Based on the information available (OCDS organisations have a complex id nested a level down #361) I'm not sure the proposed repetition of the ID in a section (Updates to organisation handling in OCDS #368 [6]) is the best solution - compared to changing the code used for flattening the data.

Minor comments:

3. I think you could expand the "organization.role" to include the review body, as that is a fairly standard organisation important in the procurement process.

[timgdavies]

Thanks for catching the omission of 'reviewBody'. I've added this to the staged codelist. We had drafted it up, but failed to include it:

• reviewBody - Review Body - A party responsible for the review of this procurement process. This party often has a role in any challenges made to the contract award.

[timgdavies]

This is now merged into OCDS 1.1