Data Retrieval API

SDA-DOA

Configuration

Outbox functionality

Outbox in DOA is RabbitMQ-based listener that can be triggered by incoming "export request". Template of such message:

{
    "jwtToken": "...",         // mandatory: Elixir AAI token (see below)
    "datasetId": "...",        // optional: either datasetId, or fileId should be specified
    "fileId": "...",           // optional: either datasetId, or fileId should be specified
    "publicKey": "...",        // mandatory: Crypt4GH public key of the requester
    "startCoordinate": "...",  // optional
    "endCoordinate": "...",    // optional
}

Upon receival of such message, DOA acts exactly the same way as if this information arrived via REST endpoint. The difference is that data is not "returned" to the requester in a response, but is being dumped to the outbox location (re-encrypted for the requester).

The reason for having this functionality is so-called "offline" use-case, where DOA is running in the isolated environment (like TSD) and can't expose REST API (but still can receive RabbitMQ messages).

Handling Permissions

Data Retrieval API can be run with connection to an AAI or without. If connection to an AAI provider is not possible, the PASSPORT_PUBLIC_KEY_PATH and CRYPT4GH_PRIVATE_KEY_PATH need to be set.

If connected to an AAI provider the current implementation is based on GA4GH Passports

The AAI JWT payload should contain a GA4GH Passport claim in the scope:

{
    "scope": "openid ga4gh_passport_v1",
    ...
}

The token is then intended to be delivered to the /userinfo endpoint at AAI, which will respond with a list of assorted JWTs gathered from providers that need to be parsed in order to find the relevant information.

{
    "ga4gh_passport_v1": [
        "JWT",
        "JWT",
        "JWT",
        ...
    ]
}

Each third party token (JWT, RFC 7519) consists of three parts separated by dots, in the following manner: header.payload.signature. This module processes the assorted tokens to extract the information they carry and to validate that data. The process is carried out as such:

Dataset permissions are read from GA4GH RI claims of the type "ControlledAccessGrants"

{
    "ga4gh_visa_v1": {
        "type": "ControlledAccessGrants",
        "value": "https://www.ebi.ac.uk/ega/EGAD000000000001",
        "source": "https://ega-archive.org/dacs/EGAC00000000001",
        "by": "dac",
        "asserted": 1546300800,
        "expires": 1577836800
    }
}

SDA-download

Recommended provisioning method for production is on a kubernetes cluster using the helm chart sda-svc which contains the sda-download.

sda-download focuses on enabling deployment of a stand-alone version of SDA, with features such as: - trusted JKU and ISS pairs; - custom dataset names including DOI URLs; - etc.

REST API Endpoints

API endpoints listed as OpenAPI specification is available:

openapi: 3.0.0
info:
  description: SDA Data Retrieval API Documentation derived from EGA Data API
  version: "1.0"
  title: SDA Data Retrieval API Documentation 
  license:
    name: Apache 2.0
    url: http://www.apache.org/licenses/LICENSE-2.0
tags:
  - name: file
    description: File Streaming
  - name: metadata
    description: Metadata Controller
paths:
  "/files/{fileId}":
    get:
      tags:
        - file
      summary: getFile
      operationId: getFileUsingGET
      parameters:
        - name: fileId
          in: path
          description: fileId
          required: true
          schema:
            type: string
        - name: destinationFormat
          in: query
          description: destinationFormat
          required: false
          schema:
            type: string
            default: plain
        - name: startCoordinate
          in: query
          description: startCoordinate
          required: false
          schema:
            type: integer
            format: int64
            default: 0
        - name: endCoordinate
          in: query
          description: endCoordinate
          required: false
          schema:
            type: integer
            format: int64
            default: 0
        - name: Public-Key
          in: header
          description: Public Encryption key
          required: false
          schema:
            type: string
      responses:
        "200":
          description: OK
        "401":
          description: Unauthorized
        "403":
          description: Forbidden
        "404":
          description: Not Found
  "/metadata/datasets":
    get:
      tags:
        - metadata
      summary: list
      operationId: listUsingGET
      responses:
        "200":
          description: OK
        "401":
          description: Unauthorized
        "403":
          description: Forbidden
        "404":
          description: Not Found
  "/metadata/datasets/{datasetId}/files":
    get:
      tags:
        - metadata
      summary: getDatasetFiles
      operationId: getDatasetFilesUsingGET
      parameters:
        - name: datasetId
          in: path
          description: datasetId
          required: true
          schema:
            type: string
      responses:
        "200":
          description: OK
        "401":
          description: Unauthorized
        "403":
          description: Forbidden
        "404":
          description: Not Found