Storage and Data Deletion

uni.lu

This section is specific to the main Adept instance adept.lcsb.uni.lu.

Storage

The data is stored in 3 locations: PostgreSQL, Elastic Search, MinIO.

1. The metadata (e.g. dataset names) is stored in PostgreSQL.
2. The uploaded data goes into MinIO. Only the last file is kept.
3. Then, an import process runs and stores the uploaded file in Elastic Search.

See more in the architecture page.

The uploaded file isn't deleted after the import process completes. This helps with debugging if any issues occur during import.

Data deletion

If a dataset is deleted from Adept

• The uploaded dataset file is deleted from MinIO
• The corresponding data is deleted from Elastic Search
• The exported data from the dataset is deleted

Basically, the data is completely removed from Adept, except for backups. We do have some precautions with restoring backups like mentioned below.

If there is a request to delete specific data

If a request is made to delete data (e.g. if a patient no longer wishes to participate in the study), follow the process outlined below:

1. An Adept manager (role) reimports the dataset without the relevant data.
2. The Adept team is informed on what to do if a backup needs to be restored (e.g. remove dataset, ask dataset manager to redo deletion operation)

Step 2. is needed because we cannot delete data from backups.

Note: It's only possible to delete specific data if the data is not anonymised or is pseudonymised. We cannot delete anonymised data because we can't identify the data.