The National Infrastructure Database (NID) is the foundation of DAFNI. The service provides the means for infrastructure researchers to access and publish those datasets which are at the heart of their research.
The core of the NID is a data-store with the capacity to store hundreds of terabytes of data. This is supported by databases which store and manage the rich metadata records for each dataset, providing comprehensive Data Search and Data Versioning capabilities around the data-store itself.
The NID is a core service of the DAFNI platform, managing the provision of data to models, workflows and visualisations. Outputs from model workflows can be published back to the NID, allowing the research community immediate access to the latest model outputs.
The DAFNI data store is a MinIO object storage instance with a capacity of up to 900 terabytes. The adoption of object storage allows DAFNI to be completely flexible and store any data in any format required by the infrastructure research community. MinIO provides a Cloud Native solution which integrates seamlessly into DAFNI’s underlying kubernetes ecosystem.
DAFNI researchers interact with this data store through a set of pages on the DAFNI platform’s front end. With these resources, researchers can upload data to the NID and manage the permission on that data, allowing others on the platform to access it either globally, individually or through groups. In addition, researchers can update their datasets and create new versions, and all visitors to the DAFNI platform can access and download the open access datasets through the front end pages.
Metadata
DAFNI has adopted a rich metadata schema, based primarily on DCAT V2 The aim of this approach is to support excellent Data Search capabilities on the DAFNI platform as well as to position the platform for interoperability with other data stores in the future.
The information captured in the metadata combines top-level contextual and licencing information with more detailed dataset attributes, which drill down to the file level. This information is combined with a description of the dataset’s ownership and publication history in order to provide traceability and link each dataset on DAFNI to its established position in the infrastructure research community today.
Data Search
Fuelled by the rich metadata schema, the newest service to be constructed for the DAFNI platform is the Data Search service. This service is built on Elasticsearch – a powerful full-text search and analytics engine.
Through the NID’s front end pages, DAFNI researchers can find datasets of interest to their research via a text search or by spatiotemporal filtering. Filters by data source, theme and file format are also supported.
Interested in collaborating with DAFNI?
If you would be interested in using DAFNI, would like to access Data on DAFNI or involve us in a new project, we would like to hear from you. Please complete your details on the contact form using the link and we will be in contact with you by email.