Baseline for ODS Data Search & Export (including Predefined Reports)
It must be noted that the baseline for ODS Data Search & Export (including the Predefined Reports) differs from the baseline used to create the legacy .csv files. Therefore, users may notice additional records in search results.
Legacy csv Baseline
- the scope of csv files were generally based on only the legal context (i.e. organisations with no legal end date), or organisations that had legally closed since the previous financial year.
ODS Data Search and Export Baseline
- ODS DSE and ODS API data uses the operational context (operational start/end dates) to determine a record's status (active or inactive)
- Any records legally and operationally closed before 31 March 2017 will NOT be available via ODS Data Search & Export (as they are considered archived). 'RefOnly' / skeleton versions of archived records are returned by the ODS API (see above).
- Records which are operationally open, but have a legal close date, will be returned/included in results. Both operational and legal start/end dates are available to users via the ODS Data Search & Export tool and ODS APIs to enable the data to be further filtered if required.
Archiving of Organisation and Site Records (API and Data Search and Export)
Closed records are appended to an archive file periodically, in order to maintain the efficiency of the live data. Archiving a record (based on its closure before a certain date) will override any of the other criteria for publication in live data. The archive file follows exactly the same data model as the live data so there is only one universal schema.
Each time an archiving event is triggered, a cut-off date is defined. Everything with an operational close date that falls before the most recently defined cut-off is published in the archive product, instead of live data. Open records and records closed after the defined cut-off date will remain within the live/current data, which will continue to accumulate closed records in between archive events.
Decisions to archive will be taken by the issuing authority (ODS) periodically, based on an assessment of the size of the release products and detrimental effects judged to be having on consumers’ ability to consume them. The rate at which closed records accumulate within the live release products will vary depending on external influences driving change to the health and social care organisational landscape. As such, the frequency of archive events (i.e. the movement of closed records from the live release products to the archive file) and the volume of records that are archived each time will vary. Any archiving event will be signposted to users via advance communications (however will not be notified in an information standard).
Obtaining ODS XML Archive
The XML archive file can be accessed by registering for TRUD (Technology Reference Update Distribution) and subscribing to the ODS XML Organisation Data pack: https://isd.digital.nhs.uk/trud/user/guest/group/0/home
RefOnly Records
The existence of relationships in the data allows an active record to reference a record that has been archived. In order to preserve a degree of referential integrity, a ‘skeleton’ version of archived records (i.e. with relationships and other content omitted) are published in the data. These are known as RefOnly records.
Current Archive Cut Off / Baseline Date
At the time of writing, all records with an Operational close date on or before 31 March 2017 will be published in the archive product and, accordingly, omitted from the live release products other than as part of a RefOnly record. All other records regardless of their status are included in the full file.
Archiving of Practitioner Records (API and Data Search and Export)
Nurse Prescribers and Consultants (Enurse, econcur and edconcur): these predefined reports are snapshot files (raw data supplied to ODS as snapshots), therefore no archiving applies.
Private Controlled Drug Prescribers (Epcdp): no archiving applied to closed records, all closed records available via the epdp predefined report in Data Search and Export
General Practitioners (egparc): as above, this is a non dynamic/static file that has not been updated since 2014. ODS will continue to make this available to users via TRUD and the ODS website.
Legacy Archive .csv File Content
It is not possible to replicate the exact content (records) found within some legacy .csv files as they had a different archive cut off date for closed records than current ODS products. Any records legally and operationally closed before 31 March 2017 will NOT be available in current ODS products (DSE and API). The below table indicates how archived data/some legacy archived data products can still be obtained:-
| Legacy .csv archive file name | Legacy .csv Archive File Description | What we have now - Data Search and Export (DSE) |
|---|---|---|
| earchive.csv | Based upon an incremental archive date which aligned with the legacy ODS production schedule. Data for organisations which have closed since 1991, excluding organisations closed within the current or previous financial year (when an organisation is closed it remains in its relevant current file for one financial year). Note that many of the organisations in the file belong to a type that is now obsolete entirely (e.g. Primary Care Groups) and so there is no corresponding ‘current’ file for them any longer. The file includes everything apart from Egparc.csv and Epracarc.csv. | DSE Predefined Reports are dynamic and update every night. Because of this, ODS no longer have a production schedule (monthly/quarterly releases on a set date), users can self serve data at any point they require. We will publish the final earchive.csv file via TRUD and via the ODS website as part of the final production of the legacy .csv files - currently planned for September 2025. This final .csv archive file will continue to be available to users to download, but it will not be updated. |
| epracarc.csv | Name and address information and identifying codes for GP Practices in England and Wales that have been closed prior to the beginning of the previous financial year. Non dynamic / static file not updated since 2014 | ODS will continue to publish these static files for reference only via TRUD and ODS website. |
| niarchive.csv | Name and address information and identifying codes for organisations in Northern Ireland that have been closed prior to the beginning of the previous financial year. | |
| egparc.csv | Name and address information and GNC codes for General Practitioners working in England and Wales, that have been closed prior to the beginning of the previous financial year. Non dynamic / static file not updated since 2014 | |
| succarc.csv | Records successor relationships for archived organisations (closed before the start of the previous financial year) | DSE Predefined report 'Succ' (successor organisations) now returns all succession data (an archive cut off is no longer applied) |
Record Deletions
There are rare occasions where ODS are forced to delete whole or partial records. This is due to incorrect information being provided to us, or human error has caused part of a record to be created incorrectly. We try to avoid deletion wherever possible and apply end dates instead. However, sometimes this is not possible and our only option is to delete the record.
Partial Record Deletion
As an example, there are business rules within our data repository that restrict which dates can be updated/added, particularly for relationships where some only permit 1 relationship of a given type at a time. This can mean an incorrect relationship must be deleted in order for us to accommodate the correct information.
Another example would be, where a commissioner provides the incorrect PCN details for a GP Practice 'partner' relationship and subsequently asks for the practice to be moved to a different PCN for the same period of time. Due to PCN relationships being key for payment calculations, we have to delete the incorrect relationship and add in the new/correct relationship.
For users, this will mean that before the deletion occurs, ODS data will contain the unique relationship ID for the incorrect relationship (with all associated relationship details), but post deletion, that relationship record, including the unique relationship ID, will no longer be found in ODS data.
Whole Record Deletion
It is highly unlikely that an entire record will be deleted/removed from publication, but it is possible if ODS are instructed to withdraw code details from publication due to incorrect/sensitive information etc or due to human error when creating codes in our internal database.
The internal ODS actions that trigger a deletion are:
- 'XML' flag - this internal ODS database attribute governs what records are sent to the ODS APIs (STU3/ORD/R4). The flag can be switched on/off at organisation type level (i.e. applies to all records of a given type/primary role), or at record level (we can choose to exclude an individual record of a given type from the API feed if required). If the flag is changed from 'include' to 'exclude' at org type or record level then this triggers a delete notification to the API Notifications Endpoint.
- Publication Flag - applied at individual record level. Allows ODS to 'reserve' a code in our database prior to publication. If a previously published record is unpublished, then this will trigger a delete notification to the API Notifications endpoint.
- Operational end date prior to archive cut off - in the highly unlikely circumstance, that ODS are forced to close an active record using an operational end date that is prior to the archive cut off date, this will trigger a delete notification to the API Notifications endpoint.
Where a whole record has been deleted, the API will return a 404 error - Record not found. Confirmation of the deletion is available via the Notifications Endpoint - see below.
Note that a record can be unpublished / deleted but then re-published at a later date. In this scenario, the deletion record will be removed from the Notifications Endpoint and a valid API response will be presented when the record is queried. The LastChangeDate will be updated when the record is republished.
ORD Notifications Endpoint
The ORD Notifications Endpoint can be queried to return details of all ODS organisation or sites records deleted in the previous 185 days. This is limited to records that have been removed in their entirety from the API. It does not currently report on elements of an ODS record that have been deleted, such as relationships.
Further information regarding the ORD Notifications endpoint is available on the ODS website and within the ORD API specification.
FHIR R4 Notifications Endpoint
The ODS FHIR R4 API Notification Endpoint caters for both scenarios of partial or full record deletion. Partial deletion includes the deletion of non primary roles, relationships or succession within an organisation or site record, but the code remains active.
Types of deletion notification:
- 'OrgDeletion' – A deleted Organisation record
- 'RelDeletion' – A deleted Relationship record
- 'RoleDeletion' – A deleted Role record
- 'SuccDeletion' – A deleted Succession record
For further information, please refer to the OAS specification.