Organisation Data Service : ODS Archived or Deleted Records

Created by Laura Reeves, last modified on Aug 08, 2024

Archiving of Organisation and Site Records

Closed records are appended to an archive file periodically, in order to maintain the efficiency of the live data. Archiving a record (based on its closure before a certain date) will override any of the other criteria for publication in live data.

The archive file follows exactly the same schema as the live data so there is only one universal schema. Any future changes made to the schema will apply to both the live data and the archive file.

Each time an archiving event is triggered, a cut-off date is defined. Everything with an operational close date that falls before the most recently defined cut-off is published in the archive product, instead of live data.

Open records and records closed after the defined cut-off date will remain within the full file, which will continue to accumulate closed records in between archive events. Decisions to archive will be taken by the issuing authority (ODS) periodically, based on an assessment of the size of the release products and detrimental effects judged to be having on consumers’ ability to consume them. The rate at which closed records accumulate within the live release products will vary depending on external influences driving change to the health and social care organisational landscape. As such, the frequency of archive events (i.e. the movement of closed records from the live release products to the archive file) and the volume of records that are archived each time will vary. Any archiving event will be signposted to users via advance communications (however will not be notified in an information standard).

Obtaining ODS Archive

The archive file can be accessed by registering for TRUD (Technology Reference Update Distribution) and subscribing to the ODS XML Organisation Data pack:  https://isd.digital.nhs.uk/trud/user/guest/group/0/home

RefOnly Records

The existence of relationships in the data allows an active record to reference a record that has been archived. In order to preserve a degree of referential integrity, a ‘skeleton’ version of archived records (i.e. with relationships and other content omitted) are published in the data. These are known as RefOnly records. 

Current Archive Cut Off / Baseline Date

At the time of writing, all records with an Operational close date on or before 31 March 2017 will be published in the archive product and, accordingly, omitted from the live release products other than as part of a RefOnly record. All other records regardless of their status are included in the full file.

Replicating Legacy .csv File Content

It is not possible to replicate the exact content (records) found within some legacy .csv files as they had a different archive cut off date for closed records than current ODS products. Any records legally and operationally closed before 31 March 2017 will NOT be available in current ODS products.


Archiving of Practitioner Records

tbc


Baseline for ODS Data Search & Export (including Predefined Reports)

It must be noted that the baseline for ODS Data Search & Export (including the Predefined Reports) differs from the baseline used to create the legacy .csv files. Therefore, users may notice additional records in search results.

Legacy csv Baseline

  • the scope of csv files were generally based on only the legal context (i.e. organisations with no legal end date), or organisations that had legally closed since the previous financial year.

ODS Data Search and Export Baseline

  • ODS DSE and ODS API data uses the operational context (operational start/end dates) to determine a record's status (active or inactive)
  • Any records legally and operationally closed before 31 March 2017 will NOT be available via ODS Data Search & Export (as they are considered archived). 'RefOnly' / skeleton versions of archived records are returned by the ODS API (see above).
  • Records which are operationally open, but have a legal close date, will be returned/included in results. Both operational and legal start/end dates are available to users via the ODS Data Search & Export tool and ODS APIs to enable the data to be further filtered if required.


Record Deletions

There are rare occasions where ODS are forced to delete whole or partial records. This is due to incorrect information being provided to us, or human error has caused part of a record to be created incorrectly. We try to avoid deletion wherever possible and apply end dates instead. However, sometimes this is not possible and our only option is to delete the record.

Partial Record Deletion

As an example, there are business rules within our data repository that restrict which dates can be updated/added, particularly for relationships where some only permit 1 relationship of a given type at a time. This can mean an incorrect relationship must be deleted in order for us to accommodate the correct information.

Another example would be, where a commissioner provides the incorrect PCN details for a GP Practice 'partner' relationship and subsequently asks for the practice to be moved to a different PCN for the same period of time. Due to PCN relationships being key for payment calculations, we have to delete the incorrect relationship and add in the new/correct relationship.

For users, this will mean that before the deletion occurs, ODS data will contain the unique relationship ID for the incorrect relationship (with all associated relationship details), but post deletion, that relationship record, including the unique relationship ID, will no longer be found in ODS data.

Whole Record Deletion

It is highly unlikely that an entire record will be deleted/removed from publication, but it is possible if ODS are instructed to withdraw code details from publication due to incorrect/sensitive information etc or due to human error when creating codes in our internal database.

The internal ODS actions that trigger a deletion are:

  • 'XML' flag - this internal ODS database attribute governs what records are sent to the ODS APIs (STU3/ORD/R4).  The flag can be switched on/off at organisation type level (i.e. applies to all records of a given type/primary role), or at record level (we can choose to exclude an individual record of a given type from the API feed if required). If the flag is changed from 'include' to 'exclude' at org type or record level then this triggers a delete notification to the API Notifications Endpoint.
  • Publication Flag - applied at individual record level. Allows ODS to 'reserve' a code in our database prior to publication. If a previously published record is unpublished, then this will trigger a delete notification to the API Notifications endpoint.
  • Operational end date prior to archive cut off - in the highly unlikely circumstance, that ODS are forced to close an active record using an operational end date that is prior to the archive cut off date, this will trigger a delete notification to the API Notifications endpoint.

Where a whole record has been deleted, the API will return a 404 error - Record not found. Confirmation of the deletion is available via the Notifications Endpoint - see below.

Note that a record can be unpublished / deleted but then re-published at a later date.  In this scenario, the deletion record will be removed from the Notifications Endpoint and a valid API response will be presented when the record is queried. The LastChangeDate will be updated when the record is republished.

ORD Notifications Endpoint

The ORD Notifications Endpoint can be queried to return details of all ODS organisation or sites records deleted in the previous 185 days. This is limited to records that have been removed in their entirety from the API. It does not currently report on elements of an ODS record that have been deleted, such as relationships. 

Further information regarding the ORD Notifications endpoint is available on the ODS website and within the ORD API specification.

FHIR R4 Notifications Endpoint

The ODS FHIR R4 API Notification Endpoint caters for both scenarios of partial or full record deletion.  Partial deletion includes the deletion of non primary roles, relationships or succession within an organisation or site record, but the code remains active.

Types of deletion notification:

        • 'OrgDeletion' – A deleted Organisation record
        • 'RelDeletion' – A deleted Relationship record
        • 'RoleDeletion' – A deleted Role record
        • 'SuccDeletion' – A deleted Succession record

For further information, please refer to the OAS specification.