CERN DIGITAL MEMORY

Problem definition:
We need to archive the CERN Digital Memory which consists of the digital production of the institution for the 21st century (including new types like web sites, social media, emails, etc) as well as the analogue documents produced by the institution in the 20th century, composed of digitised papers (physical archive) and various multimedia: audio (e.g. recordings of meetings), still images and moving images.

The digitized institutional content is loaded and maintained into CERN Live information systems. These information systems use various underlying storage solutions (e.g. systems like DFS, EOS etc.) but none of them is OAIS compliant. The goal is to connect the active services with a dark archive where Archival Information Packages will keep comprehensive information for each 'document'. The goal is to have at our disposal at the end a standard trustworthy ISO16363-compliant digital archive where live systems can deposit content selected for long term preservation.
By ensuring that data are kept in the same archiving solution, we can introduce specific standards and formats for long term preservation and, as a consequence, minimize fragility which is one of the highest risks for long term preservation in the organization. We want to ensure that we keep globally accepted preservation processes and standard formats for the same type of content (e.g. video).
Aligning the CERN digital archive to best practices (OAIS) for the sake of long term digital preservation is one of the most important benefits for this use case. In addition to this, the existence of successful disaster recovery solution institution-wide could impact all individuals, as everyone produces the same types of content.

Lifecycle - Workflow Characteristics:
● The authentication needs for the basic use case are minimal, as only the Service Manager will need to access the Archiving Service and not the end-user.

● One important aspect of the use case is the ability to have partial access to the data, i.e. to recall just one file or even a chunk of a file out thousands of files in a specific dataset.
The system manager should be able not only to submit and download data from the service but also to update and re-ingest specific parts.
● We want the files to be accessed reasonably quickly but not live and we expect the files to be accessed relatively rarely. Consequently, the data ingestion needs for this scenario are not particularly high.
● The existence of personal data is also important for this scenario. For example, we need to be able to deal with cases of users asking their photos to be accessed or removed from the archiving service, in compliance with CERN Internal Regulation (Operation Circular No11) or GDPR.

Authentication and Management Functions:
● The authentication needs for the basic use case are minimal, as only the Service Manager will need to access the Archiving Service and not the end user.
● The user access control requirements for this scenario are particularly high as the ability to store files that are readable only by specific users is essential. For example, the Service Manager should not be able to access to the audio of a CERN Council meeting, even if he will still be the one submitting the files to the service.. As a consequence, files need to be encrypted and this drastically increase the R&D challenges in terms of long term preservation.

Data and Metadata Characteristics:
The overall size of the CERN Digital Memory is estimated to be ~700TB of digitized material and approximately ~700TB of digitally born files. For the vast majority of the data we have well defined metadata. However, as there are all different types of content, including documents, images, videos, sound recording etc. the existing metadata also vary drastically in terms of schema, existing fields and volume. There is no universal standards and formats for the contents of the CERN Digital Memory. After the backlog of data is ingested, the daily ingestion rate will require some more evaluation as it will depend on which information systems will be plugged into the archiving service. Ingestion of Invenio-based services like videos.cern.ch or zenodo.org should be sized to be able to deal with an average speed of 1.5MB/s.

Interface Characteristics:
There are many types of interfaces needed. These are not exhaustive.
● API from active system to the dark archive to enable:
○ Automation of the transfer of a SIP (or Baggit) from the information systems into an archiving pipeline to create the AIP
○ Automation of the inclusion of the metadata part into the indexes of the Archive system

○ Automation of the definition of the processing steps (configuration options) to overwrite the default configuration for a given SIP
○ Automation of the fetching of ‘external’ files at the time of archiving
○ Automation of the re-ingestion of a SIP with modified metadata or modified files
○ Automation of the creation of an AIC (Compound object composed of multiple AIPs)

● API from dark archive to the active information systems to enable:
○ Return the status of an archiving process for a given SIP (id)
○ Provide detailed information in case of failures, individually or by batch (e.g.: all
the failures on Week 23)
○ Provide direct access to the created AIPs
○ Provide direct access to converted files (depending on the formats), and
corresponding checksum files
○ Provide the log of the operations run for the preservation of a given SIP (id)
○ Provide a risk assessment for a given AIP (id)
● Web Interface for active system managers
○ A Dashboard interface with Browsing and Searching capabilities to search/discover the preservation processes. Each Info System manager should only access the dashboards relative to the SIPs/Baggit coming from his system.
○ An Audit log interface where the details of all actions can be analysed.
● Web Interface for the Archive Manager
○ Administrative and Access interface to manage the dashboards and the storage features of the Archive system,

Reliability Requirements:
The possibility to deploy new pipelines to cover the preservation of new content types (e.g. emails, websites and others) and scale up the archive whenever needed is key to the reliability of the service. In addition, the possibility to run the process directly on the CERN Data Cloud would be a plus - so that the Data will not be hosted on external non-controlled storage. This is particularly true for confidential data.
The possibility to support encryption of Data would provide more reliability for the handling of confidential content. Encryption could be at the info system level (bit preservation, provenance, authenticity only) or at the AIP level, after processing the submitted SIP in clear.
To provide more trust in the Archive, some preservation information like the fixity and the provenance could also be published to a public Blockchain (distributed and read-only). The key would be auditable, unchangeable, and open and it would serve as a single source of truth which all users could trust. At any point this information could then be used to audit and validate the information stored in the Archiving system.

Compliance and Verification:
Ideally, the system would be able to support a data management plan. For examples:

- Interval to run fixity checks on the assets should be configurable and managed in the Archiving system admin interface
- Verification on the end of embargoed files and changes of access rights accordingly (closed -> public or closed -> restricted etc) should be supported and reported.
- Batch migration of formats depending on planned obsolescence, or new emerging standards.
- Additional Documentation Information required by OAIS (e.g. Designated Community, etc) could be integrated into the Archiving System to help its maintenance and to support regular reviews.

Cost Requirements:
Maintenance costs should not jeopardize the preservation funding. Some costs can be well defined, and others are more difficult to estimate. For this deployment scenario, we are particularly interested to receive cost estimations through the Curation Costs Exchange Interface.