Old release in old format

You are viewing an release from the early development stages of the Open Contracting Data Standard. These pages have been frozen as they use an earlier documentation model which is not compatible with the current standard website.

The following features are no longer in operation on this page:

  • links to the the latest versions from the top panel - please go to the latest version and browse other versions of the standard from here.
  • annotation, and any features which require you to be logged in

Unless you are interested in this page for historical reasons, please go to the latest version.

Open Contracting Data Standard - 0.3.2

Released on: Wed, 03 Sep 2014 08:40:19 GMT

Share your comments

The online version of this document is open for comment. You can share your comments in two ways:

  • Inline comments on the document - Login, by clicking on , and then highlight portions of text to add comment. To "reply" to an existing comment, highlight the same portion of text, and then add your comment. See for more help on commenting.
  • Mailing list - If you have more general comments that don't fit well as inline comments, then please do join our mailing list and start a discussion with your thoughts.
Throughout this document there are specific questions for consultation included, but you are not limited to responding to these questions, we welcome all input you can offer.

Not the latest release.

You are viewing v0.3.2. The latest release is v0.3.3


Open for consultation

This version is open for consultation until September 30th. Visit http://standard.open-contracting.org for more background information.

Purpose

This is the beta release of the Open Contracting Data Standard. This release provides:

  • A description of the overall Open Contracting Data Standard Model
  • A JSON Schema for open contracting releases and records
  • Details of suggested implementation.

This beta offers early adopters the chance to test the overall data model for the standard, and to provide feedback on the coverage and definition of fields provided by the standard.

Both structural and field changes should be anticipated between the beta and initial release version.

Background

Countries signing up to the Open Contracting Global Principles commit to share "information related to the formation, performance, and completion of public contracts" including data on all stages of contracting, including subcontracting arrangements.

The Open Contracting Data Standard is being developed to allow as much of this information as possible to be shared as structured data in addition to documents. The objective of the OCDS is to enable the widest possible range of stakeholders to use contracting data and documents to achieve value for money for the government, a competitive playing field for the private sector, and high quality goods, works, and services for citizens.

The development of the Data Model Draft has been based on:

  • Assessing data currently supplied through contract portals - in order to understand the data that governments currently hold and publish, and how it is structured. To carry out this analysis we created the Contracting Data Comparison tool, and Contracting Data Map which explores the fields of data available from a wide range of different government contracting data portals around the world.

  • Exploring demand for data on contracting - to identify the ways in which different users think about contracting data, and how they want to use it. This work is ongoing, and initial use cases are factored into this conceptual model and will continue to shape future iterations.

  • A Review of existing schemas: There are already well-known schemas for Organizations, People and Addresses, and related purposes. This project will build upon existing standards efforts where appropriate.

The Open Contracting Data Standard is a core product of the Open Contracting Partnership. Version 1.0 of the standard is being developed for the OCP by the World Wide Web Foundation and World Bank, through a project supported by The Omidyar Network and the World Bank.

More detailed information about the development process is available below.

Conceptual Model

The contracting process

The contracting process is the sequence of stages related to the lifespan of a contract, beginning with planning and culminating in contract closure.

Contracting Process

* Formation, when dealing with basic procurement processes, will often be referred to as Tendering & Award.

For descriptions of the phases see the "Definitions" tab above.

Each Open Contracting Process should be given a unique Open Contracting ID (ocid). We define an Open Contracting Process as having a single unique tendering / competitive phase, but with the possibility of this resulting in multiple awards and contracts. See defining a unique contracting process below for more details.

Contracting releases and record

Information about an Open Contracting Process may accumulate over time. As a result, the Open Contracting Data Standard provides for two kinds of data:

  • Contracting releases - each release provides information pertaining to a particular stage in the contracting process -- such as tender notices, award notices, or details of a finalized contract. These releases may equate to formal legal notices as part of a procurement process, or may be new releases of information. They can have attached documents. Release may also include amendments. Once published a release should not be changed, and new information should be shared through new releases with relevant amendments.

  • Contracting record. - A contracting record provides a snapshot of all the key elements of a unique contracting process, including its planning, formation, performance and completion. It is updated as new information becomes available through contracting releases to accurately reflect the current state of the contract processes. The contracting record should provide an at-a-glance view of key information and can then be used to access more detailed information from the releases. A full contracting record can contain detailed revision information for fields. Contracting records may be published in full, or may contain a pointer to releases, allowing third-parties to assemble and verify a record of the whole contracting process.

Both contracting releases and contracting records should be provided within data packages, containing meta-data about the publisher, publication data and licensing information.

Publishers can package a number of releases and records together in the same file, or can publish one release / record per-file.

Data model and serialisations

For the beta of the Open Contracting Data Standard we have developed a JSON Schema rendering of a data model, whilst at the same time paying attention to ensure this model can be easily serialised in other formats. We have proposed an approach to map this data model into flat file CSV and DataPackage formats.

We have carried out initial work to identify JSON-LD mapping but this is not currently a development priority for version 1.0.

Release data

A full schema for releases can be accessed from the 'Release Schema' tab above. Click each section to expand that component of the schema.

A release should always contain:

  • ocid - the Open Contracting Identifier for this Contracting Process. This should be globally unique.
  • releaseID - this should uniquely identify this release within the current 'package'. Generally these should be unique to a publisher, but are not required to be globally unique.
  • releaseTag - A tag that helps to identify the type of release being made. These may be used for advanced validation (i.e. certain kinds of release may in future require certain fields to be provided). The current list of tags is: planning, tenderNotice, awardNotice, contractSignature, contractAmendment, report, spending, terminationNotice
  • language - An ISO 2-Digit Country code for the default language used in the document
  • buyer - identifier and address details of the buying organisation (see below for the specification of an organisation)

A release can then contain elements within one or more of the following blocks:

  • planning
  • formation
  • awards
  • contracts
  • performance

The blocks are described in more detail below

Planning

Tender

Awards

Contracts

Performance

Record data

There should be one record for each open contracting process. The record MUST contain:

  • An ocid - An globally unique Open Contracting ID for this contracting process.
  • An array of releases - either by providing URLs to each of the releases (so that a user can compile their own record), or embedding copies of each of the releases relating to this contracting process.

It SHOULD contain:

  • compiledRelease - the latest version of all open contracting process fields, represented using the release schema. For example, if a contractSignature release has been issued with with a contractValue of $100, and then a contractAmendment release has been issued with a contractValue updated to $200, the compiledRelease would have contract/contractValue of $200.

and

  • versionedRelease - containing the history of the data in the compiledRecord, with all known past values of any field and the release that information came from.

Common data elements

The following data elements are used in a number of places throughout the standard.

Values

Periods

Attachments

Organization

Used for representing buyers, suppliers and bidders. We follow the IATI Organisation ID Standard for organisational identifiers.

Items

Used to itemsToBeProcured, itemsAwarded and itemsContracted

Notices

Milestones

Amendment information

A note on amendments

When amending arrays (e.g. a list of itemsToBeProcured) then the amendment should contain all the items in the updated array, not just new items that are being added.

Consultation questions

  • Does the proposed conceptual model make sense for the ways you want to publish or use open contracting data?
  • Which do you need most: a summary contracting record, or detailed releases of information at each stage of the contracting process?
  • Do these components cover all the relevant categories of information you want to publish or access about contracting?
  • Are we missing key elements in our description of what each component will contain?
  • How does the proposed approach of having the contracting record represent the latest stage of contracting (i.e. overwriting earlier data) affect how you might publish or use open contracting data?

Further information

Defining a unique contracting process

For the data standard, defining a unique contracting process is critical for getting useful, comparable, clean data. But, there are cases where what the unique process is not obvious. For example, a framework contract has only one tender and award but many contracts associated with that award.

We define a unique contracting process as that with a unique tendering / competitive phase.

Example 1

Multiple processes

This Tender notice has a single ID and six line items. However, to secure each item suppliers must enter a separate bid and the competition is handled separately for each item leading to 6 contracts. Because the competitive process is unique for each 6 items there are 6 contracting processes here, in spite of the single Tender Notice ID.

Example 2

Single process

This snippet from a tender notice offers an unspecified number of contracts for the successful supplier(s). The details note that there is a limit of $25k per contract, anything higher than that must be rebid competitively. All of the $25k or less contracts that are awarded under the award that will result from this tender are part of a single contracting process, because of the single bidding process.

Add-ons

In addition to the core components, there will be cases where publishers, or users, need to augment the core data with their own information. The standard will provide a mechanism for Add-On information. This will include additional fields in core components as well as Add On components (e.g. new kinds of contracting release).

The publishing and re-use of add-ons will be encouraged to try and reduce duplication and facilitate reuse of tools. The use of Add-Ons will be subject to the following restrictions 1

  • It must not use terms from outside this specification's terms where this specification's terms would suffice

  • It may use terms from outside this specification's terms where this specification's terms are insufficient.

We have identified that a location add-on will be a priority for future development, but as no publishers are currently providing location data, we were not able to develop this during the iterations towards the beta standard.

A note on framework contracts

Many public procurements take place under framework agreements. These help facilitate routine purchasing. Suppliers are pre-approved to provide a list of goods or services. Under a framework agreement, there are typically multiple contracts that are all authorized by a single award. In the data standard, an award notice release would define the framework and this information would be stored in the contracting record under Award details. Then there will be multiple contract signature releases and each one would create a new Contract section in the contracting record. This provides a way to aggregate all the information on the contracts given under a single framework agreement.

Alternative names for framework agreements: Dynamic Purchasing System (EU), Standing Offers and Supply Arrangements (buyandsell.gc.ca)

Validator

A prototype validator is available at http://ocds.open-contracting.org/validator/ which can be used to check draft documents against the schema.

Development process

The development of the Data Model has been based on:

1) Supply-side research

The supply-side research is focused on comparing contract data (37 datasets with over 175

downloadable assets from 27 publishers, from across the world). 26 publishers are from 15 countries that were selected as priority countries due to their current activity in open government and the Open Government Partnership (OGP) Action Plan. The effort sought to understand:

  • which elements of the contracting process are captured in currently published data;

  • which fields are commonly found across different datasets;

  • how do different datasets represent and model the contracting process; and

  • how far are there common identifiers that can be used to link datasets.

To answer the above questions a Contracting Data Comparison tool was created with a team of volunteer developers to provide a platform for curating meta-data about public contracting datasets from the priority countries.

The effort focused on capturing the metadata of contract data available. As it develops, the intention is for the Contracting Data Comparison tool to allow for wider public participation and thus create an ever more detailed picture of the landscape of contracting data availability and focus.

2) Demand-side use-cases

This draft is based primarily on research of existing published datasets. At the same time, the project has begun to explore priority use cases for open contracting data through workshops, webinars, a mailing list, and bilateral discussions with more than 200 stakeholders. The purpose of demand-side research is to capture the real needs and circumstances of the publishers and users of public contracting data. We wanted to understand who wants to use contracting information, what information do they need, how do they want to use the information, and why?

A questionnaire was created and shared with our online community, and with specific stakeholders representing different user profiles (including government publishers, policymakers, procuring entities, auditors, donors, journalists, auditors, data aggregators, contractors, contract monitors, service delivery monitors, anti-corruption monitors, researchers, and transparency activists).

In our consultations four primary use cases emerged. These users are interested to use open contracting data in order to:

  • Achieve value for money for the procuring entity;
  • Enable the private sector to compete for public contracts;
  • Monitor service delivery for effectiveness; and
  • Detect corruption and fraud in public contracting.

Each of the use cases we created includes a user profile, their objectives, and their contracting information needs. This document was posted here for public comment.

Drawing on these summaries we then identified 141 specific user requirements and 145 related technical requirements for the open contracting data standard. You can find a list of all the requirements we identified here.

For the most recent release of the standard we have worked through over 50 of the requirements identified, and have found that 30 are already handled by fields in the draft standard. For the remaining requirements, a number have resulted in issues for future cycles of development logged on the standards GitHub pages, and others highlight some of the tools that may need to be created alongside the standard, such as aggregators and analysis tools.

These use cases not only demonstrate what can and could be done with open contracting data, but will also shape the development of the Open Contracting Data Standard as we move towards defining the field-level specification.

Consultation questions

  • In your opinion, does the proposed conceptual model work for these use cases?
  • Are there other important use cases we should be considering?
  • Are there use cases you can envisage where the proposed conceptual model does not work well?

There are existing standardization efforts for budget and spending data under development through the Open Spending Project. In line with the principles of Joined Up Data this project will explore ways to connect data in the Open Contracting Data Standard with data in the proposed Open Spending and Open Budget Data Standards, among others.

In addition, the approach taken is informed by the development of other data standards. Research related to the Data Standard Architecture and Data Standard Governance will be released separately from the Data Model Draft.

Next steps

Over the coming months (September - November 2014) we are working towards a version 1.0 standard which can be implemented, with future changes taking place through an open consultative process.

To get there:

  • We are inviting general feedback on the data model and fields in this beta release through the public mailing list and/or by commenting on the document directly (see instructions at the top of this page);

  • We are establishing working groups around the private sector, civil society, and publishers to provide additional input;

  • We will be exploring further user demands and building these into the standard by:

    • Completing an assessment of user requirements and how these can be met either directly in the standard, or in accompanying tools and services;
    • Building up a list of issues to be addressed in the next iteration of the standard as we move towards version 1.0 later in the year - and prioritising these in consultation with key potential users of open contracting data; and
    • Using the GitHub issue tracker to identify which requirements can be met over the short-term, and which will need further development beyond the version 1.0 release.
  • We will conduct field missions to the first two pilot countries in October 2014 to:

    • Explore how the standard may be adapted to other contracting use cases outside public procurement including extractives and land;
    • Meet and/or host workshops with civil society and government to better understand demand use-cases; and
    • Compare the draft data standard with country-specific procurement datasets including the possibility of piloting the data standard.

Acknowledgements

The Open Contracting Data Standard is a core product of the Open Contracting Partnership (OCP). Version 1.0 of the standard is being developed for the OCP by the World Wide Web Foundation, through a project supported by The Omidyar Network and the World Bank".

This document contains significant contributions from Sarah Bird (Aptivate), Ana Brandusescu and Tim Davies (World Wide Web Foundation). Other contributors include: Jose M. Alonso (World Wide Web Foundation), Steven Davenport (World Bank), Lindsey Marchessault, Michael Roberts (World Wide Web Foundation), and Marcela Rozo (World Bank).


  1. The use of add-on conditions were adapted from the The Popolo Project - http://popoloproject.com/specs/#conformance 

Vocabulary

The world of procurement and contracting has many specialist terms. However, the precise way that these terms are used can vary from sector to sector, and country to country. Below we define how we are using a number of key terms.

  • Contract - a legally enforceable agreement between two (or more) parties. Often times, contracts are thought of as a single document, the document that is signed by the parties. However, in many cases, a contract can be made up of multiple documents incorporated by reference containing additional terms, specifications, provisions, standard forms or other information (known as addendums, annexes, appendices, schedules, riders, etc.) A contract should also be understood to include any later agreements of the parties that change the terms of the contract (amendments). As a legal document, the law (statutes, regulations, case law) may also impose additional terms or conditions or otherwise affect a contract.

  • Open Contracting - norms and practices for increased disclosure and participation in public contracting including tendering, performance and completion. It includes the variety of contract types, from more basic contracts for the procurement of goods, to complex contracts, joint venture agreements, licenses and production sharing agreements. Open contracting encompasses all public contracting, including contracts funded by combinations of public, private and donor sources.

  • Public Contracting Process - the planning, formation, award, execution, performance, and completion of a contract to which a government entity is a party.

    • Planning - the process of deciding what to contract for, when and how. A procurement plan is a product of the planning process that indicates an procuring entities planned procurement activities over a period of time.

    • Formation - the activities undertaken in order to enter into a contract. Good public contracting practice favours open competitive bidding or tendering as a method to form contracts (the process of publicly inviting prospective contractors to submit bids for evaluation and selecting a winner or winners). However, contracts can be awarded in other ways, such as through non-competitive direct negotiation, prequalification of bidders, and other methodologies. The contract is then "awarded" to a selected contractor. Once the contractor has been selected and the terms of the contract have been agreed, the contract is signed (or executed) by the parties, and becomes active on an effective date.

    • Performance - the implementation of the contract in accordance with the obligations laid out therein. Often, contractors will enter into additional contracts with sub-contractors in order to implement the public contract.

    • Completion - the confirmation that all obligations (deliverables and payments) of the contract have been completed, or that the contract has been terminated for some other reason.

Written in JSON Schema v4
* required field


This contains the merge strategies to build a versionedRelease for the record. Click the {} button to see the schema detail.



Merging

To turn releases into a record, they must be merged into a single record.

The basic format of a record is simple. There are three component:

  • A list of all the releases in the record. This can be with linked data to identify the release, or embedding the release themselves.
  • A compiled release, this is the same format as a release but contains the most up-to-date information compiled from all the releases.
  • A versioned release, this contains the information about the compiled field. For each field in the release, the versioned release contains a list of objects describing where the data came from and previous values of a field.

The basic principle of of merging is very simple, the latest data overwrites previous data. For example:

release_1 = {
    "releaseDate": "2014-01-01",
    "releaseID": "1",
    "releaseTag": "tenderNotice",
    "ocid": "A",
    "formation": {
        "tenderId": "A",
        "method": "Selective"
    } 
}
release_2 = {
    "releaseDate": "2014-01-02",
    "releaseID": "2",
    "releaseTag": "tenderNotice",
    "ocid": "A",
    "formation": {
        "tenderId": "A",
        "method": "Open"
    } 
}
record = {
    // Other required fields omitted for clarity
    "compiledRelease": {
        "ocid": "A",
        "releaseID": "2",   // Has no meaning, but kept for validation
        "releaseDate": "2014-01-02",    // Has no meaning but kept for validation
        "releaseTag": "compiled",
        "formation": {
            "tenderId": "A",
            "method": "Open"
        } 
    },
    "versionedRelease": {
        "ocid": "A",
        "formation": {
            "tenderId": "A",
            "method": [
                {
                    "value": "Selective",
                    "releaseDate": "2014-01-01",
                    "releaseTag": "tenderNotice",
                    "releaseID": "1"
                },
                {
                    "value": "Open",
                    "releaseDate": "2014-01-02",
                    "releaseTag": "tenderNotice",
                    "releaseID": "2"
                }
            ]
        }
    }
}

Merging gets more complicated when trying to merge arrays / lists of data.

We adopt two approaches to merging arrays: - overwrite (the same treatment as the fields above) - overwriteByKey (overwrite looks for a uniqueID when making the merge)

For most array fields, overwrite is used.

OverwriteByKey is used on only two arrays:

  • Awards
  • Contracts

First, we will discuss, basic overwrite. In this strategy a list of objects must always be given in the same order in each release. Whilst this may be a problem for data quality, the alternative, as used in awards and contracts is that every object requires a unique value, which is also practically challenging.

Basic, overwrite works as follows, where the value of Item 2 changes:

release_1_snippet = {
    "formation": {
        "itemsToBeProcured": [
            {
                "description": "Item 1",
                "quantity": 1,
                "valuePerUnit": {
                    "amount": 100,
                    "currency": "GBP"
                }
            },
            {
                "description": "Item 2",
                "quantity": 1,
                "valuePerUnit": {
                    "amount": 200,
                    "currency": "GBP"
                }
            }
        ]
    }
}

release_2_snippet = {
    "formation": {
        "itemsToBeProcured": [
            {
                "description": "Item 1",
                "quantity": 1,
                "valuePerUnit": {
                    "amount": 100,
                    "currency": "GBP"
                }
            },
            {
                "description": "Item 2",
                "quantity": 1,
                "valuePerUnit": {
                    "amount": 150, // Value changed
                    "currency": "GBP"
                }
            }
        ]
    }
}

record_snipped = {
    "compiledRelease": {
        "ocid": "A",
        "releaseTag": "compiled",
        "formation": {
            "itemsToBeProcured": [
                {
                    "description": "Item 1",
                    "quantity": 1,
                    "valuePerUnit": {
                        "amount": 100,
                        "currency": "GBP"
                    }
                },
                {
                    "description": "Item 2",
                    "quantity": 1,
                    "valuePerUnit": {
                        "amount": 150, // Value changed
                        "currency": "GBP"
                    }
                }
            ]
        }
}

If only the changed item was provided in the second release, when the entire array is updated, only the second item would be available in the merged record, as follows.

release_1_snippet = {
    "formation": {
        "itemsToBeProcured": [
            {
                "description": "Item 1",
                "quantity": 1,
                "valuePerUnit": {
                    "amount": 100,
                    "currency": "GBP"
                }
            },
            {
                "description": "Item 2",
                "quantity": 1,
                "valuePerUnit": {
                    "amount": 200,
                    "currency": "GBP"
                }
            }
        ]
    }
}

release_2_snippet = {
    "formation": {
        "itemsToBeProcured": [
            {
                "description": "Item 2",
                "quantity": 1,
                "valuePerUnit": {
                    "amount": 150, // Value changed
                    "currency": "GBP"
                }
            }
        ]
    }
}

record_snippet = {
    "compiledRelease": {
        "ocid": "A",
        "releaseTag": "compiled",
        "formation": {
            "itemsToBeProcured": [
                {
                    "description": "Item 2",
                    "quantity": 1,
                    "valuePerUnit": {
                        "amount": 150, // Value changed
                        "currency": "GBP"
                    }
                }
            ]
        }
}

We have made two exceptions to this type of merging with the overwriteByKey on awards and contract. Over the course of a large or framework contracting process many awards and contracts are made, it is useful to be able to just release information about one award or contract at a time rather than having to include all information about all awards and contracts previously made each time. The items in the array are first matched on the unique identifier, awardID and contractID respectively, and then the standard overwrite takes place. This means that awardID and contractID cannot change.

A simplified example is given below (with incomplete data) to illustrate the pattern:

release_1 = {
    "awards": [
        {
            "awardID": "ABC",
            "awardValue": 100
        }
    ]
}
release_2 = {
    "awards": [
        {
            "awardID": "DEF",
            "awardValue": 100
        }
    ]
}
release_3 = {
    "awards": [
        {
            "awardID": "DEF",  
            "awardValue": 150 // Updating information from release_2
        }
    ]
}

record_snippet = {
    "compiledRelease": {
        "awards": [
            {
                "awardID": "ABC",
                "awardValue": 100
            },
            {
                "awardID": "DEF",
                "awardValue": 150
            }
        ]
    }
}

Implementing merging

To merge releases into records, we use the jsonmerge library with its add-ons to json schema that specify a mergeStrategy on each field.

A customized version of the jsonmerge library is available at https://github.com/open-contracting/jsonmerge using the ocds branch (set as default).

An example of merging releases into a record can be seen in this ipython notebook: http://nbviewer.ipython.org/github/open-contracting/sample-data/blob/master/buyandsell/processing/Demonstrate%20merging%20a%20release.ipynb

The compiledRelease is built using the mergeStrategies that are in the normal Release Schema.

To build a versionedRelease, we use the mergeStrategies specified in the Versioned Release Schema.

Data in the compiledRelease, should validate against the regular Release Schema. Data in a versionedRelease needs to be validated against a seperate schema, which will be published soon (and can be generated using the jsonmerge library).