Documentation /  User Manual
NAV Navbar

Introduction

This document explains web user interfaces and application programming interfaces that were developed by Neuroinformatics Platform (NIP) during the Ramp-Up phase of the Human Brain Project (HBP) that took place between October 2013 and March 2016. The applications are directly available for users and their first versions were launched at the end of the Ramp-Up phase.

This user manual is composed of 3 subsections:

  1. An overview of all web applications available in the platform:
    a. Search
    b. Multi-resolution Atlas Viewer
    c. Ontology Viewer
    d. Registration App
    e. KnowledgeSpace
  2. An overview of the programmatic interfaces (APIs) available in the platform:
    a. Ksearch
    b. Scigraph
    c. Voxel Brain
    d. Image Service
    e. HBP-PROV Validator
    f. REST APIs Demo
  3. Some general information about the project

One of the main aspects of the platform is to bring together all available HBP neuroscientific data and make them accessible to the users. For accessing available data, we developed a Search application which helps neuroscientists find and download data registered in the NIP through free text and semantic tags queries.

For accessing available brain imaging datasets that are currently registered in the NIP, we developed the Multi-resolution Atlas Viewer application to explore the image datasets in several brain atlas spaces of different species. This interactive viewer allows users to specifically target a brain region in one of the three orientations (sagittal, coronal and axial) by showing brain regions as overlay on top of the images. Moreover, by clicking on the brain region, we could get information about other available datasets from KnowledgeSpace such as physiology, expression, morphology etc.

During data integration process and making data accessible through the user interfaces, we use many different ontology terms. To make it easier for users to view available ontologies in the NIP, we have developed the Ontology Viewer. This application shows ontologies and the ontology terms (concepts) they define as well as how those terms are related to each other through relations. Descriptions of ontologies and ontology terms can be retrieved.

Neuroscientific data can be registered into the platform in different ways. One of the readily available application is called Registration app and it is a web application allowing an authenticated user to submit experimental datasets and their corresponding metadata to the NIP. The main metadata to provide for any data submission are defined in Minimal Information for Neuroscience Datasets (MINDS).

The KnowledgeSpace is a community encyclopedia linking brain research concepts to data, models, and literature. It provides ontologies as defined by the Neuroscience Information Framework (NIF). The Neuroinformatics Platform relies on some of these ontologies to describe the scientific concepts describing the data we integrate in the platform.

To support above mentioned web applications or user interfaces, we have developed backend programmatic interfaces. The second part of the documentation is intended mainly at developers wishing to connect with the NIP through a programmatic interface.

Ksearch is the search component of the NIP. It is a REST API allowing search for curated datasets using different filters that are mainly taken from MINDS. Its main features are full text search, boolean search, filtering to narrow down, query expansion through term synonyms and specialisation, and documents aggregation such as search results could be grouped by data modality, brain regions or by other metadata available in MINDS.

Scigraph REST API is the NIP ontology service and it is graph based. As such, it provides read access to neuroscience related ontologies (brain regions, cell types, species, measurement methods etc.) used in the NIP. As a REST API, Scigraph is best suited for programmatic usage and can be involved in most data integration pipelines whenever ontology data access is needed. Ontology terms, properties or attributes of the ontology term which describe further the term, and relations which links ontology terms could be retrieved from Scigraph.

Voxel Brain REST service provides access to volumetric brain atlas and their annotation layers in the form of voxels. Several use cases were described in detail with real life examples. User could query available voxel brains and their unique identification numbers, and later on could ask to retrieve voxels that are tagged as any brain region within that voxel brain release.

Image Service provides access to image tiles through REST API operation request. Image tiles are created at different zoom levels for sagittal, coronal and/or axial orientations, in order to display two dimensional large images (captured from brain slices) of brain imaging dataset (volumetric) through browser. All tiles of each orientations are written into one single file as stacks in BBIC format. Image Service also provides meta information about the brain image stack file structure and content through operation request.

HBP-PROV Validator REST service provides the ability to check the correctness of an HBP-PROV file instance syntactically and semantically. Syntactic check verifies that the format is correct against the corresponding Json schema, this encompasses the naming and cardinality of entities as well as their attributes. Semantic validation uses custom validation rules that can perform neuroscience oriented verifications as well as other aspects that a JSON schema cannot enforce.

For users who would like access our APIs using iPython Notebooks, several demos have been prepared for Ontology API, KnowledgeGraph API and Voxel Brain API.

At the end of the User Manual, you will find general information on Neuroinformatics Platform answering many questions such as how to submit data, how to contact us, what licenses available and other generic questions that could be raised.

Intended Audience

This manual is intended at end users of either the Neuroinformatics Platform website or APIs. If you are looking for neuroscientific data, use Search client application and explore available data by their metadata.

Acknowledgement

Finance information

This Platform was financed through the Human Brain Project, an European FET Flagship [ICT-2013.9.9], Grant agreement no: 604102.

The Neuroinformatics Platform would like to acknowledge the additional institutions and contributors:

  • INCF
    • Linda Lanyon
    • Mathew Abrams
  • Neuroscience Information Framework
    • Maryann Martone
    • Jeff Grethe
    • Tom Gillespie
    • Willy Wong

Web Applications

Search Client

Purpose

The Search client is an information retrieval system design to help neuroscientists, students, and users to find and download data registered in the NeuroInformatics Platform (NIP) through free text and semantic tags queries.

The Search client is a front end application to the KSearch REST API and directly uses features it provides, albeit in a simplified and non-exhaustive way.

Features

Quick filters

The landing page of the Search client offers 2 levels of quick filters to narrow down your search to the relevant subset of data: by species / order and then brain region. If you select a species with few datasets registered, you will not see the 2nd level of quick filters (brain region).

Landing page view

And selecting ‘Rattus Norvegicus’ we get the following screen:

Rattus Norvegicus

If you want to bypass the quick filters, you can click on the “More search options” button to show all datasets. The button is shown below.

More search options

You can also enter a free form text query to directly land to the search results for the query.

Free form text query

At any time, you can type a free form text query in the input field at the top of the page, and press the Enter key or click on the looking glass icon to apply the query. Please note that all existing filters will be kept.

Free form text query

Paginated list of results

The main screen consists in a list of paginated search results (10 results per page) with a list of filters on the left.

Paginated list of results

Adding / removing filters

Filters are ordered by category. You can add a filter by clicking on it in the left sidebar. It will refresh the list of results and list of available filters. The active filters and categories are moved to the top of the sidebar. By clicking on the cross icon next to one of them, you can disable the filter.

Adding / removing filters

Detailed info

To see detailed info about a given result, click on it. It will display more metadata along with a download link if relevant.

Detailed info

You can put the cursor over a contributor’s name to see the contributor’s affiliations.

Affiliations

Download dataset

Dataset download is restricted to HBP members. If you’re not logged in yet, you will be prompted to do so. You can login at any time through the link at the top left of the page.

Download dataset

Atlas Viewer

Purpose

The Multiresolution Atlas Viewer allows you to easily visualize brain imaging datasets coming from several species, and mapped to several brain atlases spaces. This interactive context allows you to specifically target a region of the brain in the 3 axes (sagittal, coronal and axial) rapidly.

The Multiresolution Atlas Viewer sits upon the KSearch and Image Service REST APIs to discover and display the brain imaging datasets currently registered in the NIP.

Features

Visualizing brain slice

Most of the screen is taken up by the currently displayed brain slice. By default the Waxholm Space Rat brain atlas dataset will be selected, displaying the middle slice in the sagittal orientation.

Visualizing brain slice

Dataset information

The left pane displays information about the currently selected dataset. By default it displays only a summary of the metadata. By clicking on the “More information” button, you will be able to see detailed information.

Dataset information

Previews

The right pane displays your current position in the dataset in all 3 orientations.

Previews

Clicking on the arrows next to the current orientation will browse to the previous / next slice in the preview, and change the main slice if you did it for the current orientation.

Browse to the next / previous slice

Previews have crosshairs on top of them that are showing your current position in the atlas space. By clicking anywhere on these preview images, it will cause the 2 other orientations to quickly jump to the slice matching the crosshairs position.

Quickly jump to the slice

By clicking on the orientation name in the right pane, you will switch the current dataset orientation between sagittal, coronal and axial.

Switch the current dataset orientation

Pan and zoom

You can navigate through the current slice like in Google Maps and similar services: click and hold while moving the cursor to pan, scroll or click on the “+” and “-” buttons to zoom in / zoom out.

Zoom in / zoom out

Overlayed annotations

On most datasets, a checkbox in the left pane allows you to enable / disable the overlay of brain regions on top of the current slice.

Enable / disable the overlay

Placing the cursor hover a region on the overlay will display the region name at the top of the screen.

Display the region name

On selected datasets such as Allen Mouse Brain Atlas v3, clicking on a region will furthermore display a popup with links to related data fetched from KnowledgeSpace and our KnowledgeGraph.

Display related data

Clicking on the title of the popup will lead you to the corresponding page on KnowledgeSpace with full information about the brain region.

Display full information

Clicking on one of the links will lead you to a page with the list of resources.

Display list of resources

Switch dataset

The two select lists in the left pane allow you to switch between atlases and datasets. An atlas may have one or more datasets mapped to it.

Switch between atlases and datasets

Access to some datasets is restricted to HBP members, and you will be prompted to log in in order to view them.

Dataset restricted access

You can share a direct link to a given dataset by copy-pasting the URL and sending it to someone else. The URL will be something like: https://nip.humanbrainproject.eu/atlas/#/?dataset=5954630a-ec21-11e4-a5f8-6003088da632

Ontology Viewer

Purpose

The ontology viewer allows to visualize ontologies available in the NIP Platform, the ontology terms (concepts) they define as well as how those terms are related each other through relations. Descriptions of ontologies and ontology terms can be retrieved.

Features

The application consists on the search bar and the tree viewer and provides the following features:

The viewer support both ontologies where the main relationship is subClassOf and those where the main relationship is part of (partonomy). An example of a partonomy is the Allen mouse brain atlas v3 ontology.

Allen mouse brain atlas v3 ontology

Using the viewer, it’s possible to pick different ontologies:

Pick different ontologies

Search for an ontology term

It’s possible to search for a term using the search bar. Suggestions of terms gradually appear right below the search bar as the user is typing. The suggestions can be filtered by the picked ontology, as shown in the screenshot below:

Search for an ontology term

Spelling correction

In case of wrong spelling of an ontology term, corrections are suggested in a ‘Did you mean section’ right below the search bar:

Spelling correction

Identify the place of a retrieved term in the ontological structure

From the search results it’s possible to click on any term and see its place in the ontological structure: its parents and immediate children are shown:

Term in the ontological structure

In order to facilitate the communication, it’s possible to share a link to any given ontology or ontological term.

Use case

The main use case of the ontology viewer is during the communications of the data curator and data provider:

  • Data provider has the data related to the brain region X.
  • He enters the name of the brain region to the search and finds the region.
  • He looks at the parents of the region using the partonomic tree view, looks at the description of the term and ensures that it is indeed the region he means.
  • He then copies the identifier (URI) of the brain region and gives it to the Data curator.
  • The data curator proceeds with formalizing the metadata knowing that the ambiguity of the brain regions is resolved.

Registration App

Purpose

The registration app is a web application allowing an authenticated user to submit experimental data (datasets and their corresponding metadata) to the Neuroinformatics Platform (NIP). The main metadata to provide for any data submission are defined in Minimal Information for Neuroscience Datasets (MINDS).

General dataset metadata

  • Dataset title: information describing the dataset such as type of data, brain region, cell type, version. A good title is important in order to search and find a dataset using Ksearch. Some examples of dataset titles: ‘P40 mouse hippocampus oligodendrocyte single-cell transcriptomics’, 'Subcellular signalling kinetic model of the striatal medium spiny neuron (MSNs),
  • Dataset description: more detailed information about the data submitted including type of data, brain region, cell type, if the data was annotated with a brain template or any other important information. This description can be used to find the dataset as well,
  • publications: PubMed and/or DOI identifiers of the publication that describes the generation of the dataset,
  • Production date: the date by which the production of the dataset has been finished.
  • License: the license of the dataset,
  • Data categories: the broad categories (modalities) of the data, like electrophysiological data or simulation data. It’s possible to pick more than one category.

Metadata about the sample

It’s possible to fill the following metadata regarding the sample: sex and age of the specimen used for the experiment, list of brain regions, comments.

Contributors

Information about agents that contribute to data generation can be provided along with their roles.

Methods and protocol used to generate the data

The app allows to describe dataset generation protocols and to list the measurement and analytical methods used.

User manual

In order to register the data within the NIP, the following steps should be taken:

Go to the main page of the Registration App

Dataset registration

Fig. Main page of the Registration App.

Agreement with the Terms of Service

Read carefully our Terms of Service and accept them

Agreement terms of service

Fig. Terms and Condition page.

Register the Metadata related to the dataset

For a given dataset the metadata would typically record the following

  • Specimen: age, sex, species, strain
  • Classification properties, such as cell type
  • Brain location using the ontological term
  • Contributors and their affiliations
  • Methods and protocols used to generate the data
  • Access to the data and its format.

Metadata page of the app

Fig. The metadata page of the app

“Data categories” and “Methods” fields have “Browse” buttons that will allow to choose the terms from the respective ontologies:

Ontology visualization for the
Categories

Fig. Ontology visualisation for the Categories ontology

Fill in the contributors information

This information will define acknowledgements as well as contact information for a specific dataset. It consists of the following:

  • Organisations involved
  • Contributors informations specifically the email of the institution they are affiliated to.

Add contributor

Fig. “Add contributor” modal window.

Save the metadata

Click save button to save the metadata. It will be possible to return to the editing of the metadata later.

Publish Dataset

After the file you uploaded is processed by neuroinformatics system, the Publish button will become active. Clicking it will result of the dataset to be published and the metadata will be fixed.

To be noticed:

  • The submitted data is published as-is (ie. not curated).
  • The register dataset is final. Once published it’s not possible to change it,
  • The metadata associated to the dataset can’t be modified.

Do not close your browser window until the dataset is completely uploaded.

Follow the NIP registration process

Shortly after submitting your data, on the MyUploads page you can check the status of the submitted dataset.

Status of the dataset

Fig. Status of the dataset.

For any information, question or help, we suggest to contact the team.

Security

All people that have a Collaboratory account can access the app. To get an access the user should first request a HBP Collaboratory account.

Licenses

During the process of the registration a choice between 2 licenses will be required depending on the commercial potential use of your Dataset.

We strongly recommend to carefully read the information related to the following licenses in our ToS:

  • Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)
  • Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)

KnowledgeSpace

Purpose

The KnowledgeSpace is a community encyclopedia linking brain research concepts to data, models, and literature. To enable this semantic linking, a particular effort is put on defining, reusing and managing many neuroscience ontologies through Neuroscience Information Framework (NIF).

Integrated resources

Numerous data sources are integrated in KnowledgeSpace, such as:

Integration with NIP platform

Where appropriate, the Neuroinformatics Platform provides deeper integration with KnowledgeSpace in order to get more insight (different related experimental data, curated litterature,…) about a particular neuroscience concept like a brain region for instance. The use case subsection below describes this example.

Design, definition and share of common ontologies and vocabularies

The first step for integrating two ontology based platforms like NIP and KnowledgeSpace is to define, reuse and share the same set of ontologies and vocabularies. The two platforms need to speak the same language in some extend when it comes to annotate resources and search them. More specifically:

  • All neuroscience related ontologies used in NIP are imported (before being extended if necessary) from KnowledgeSpace ontology repository. The extensions made by NIP platform (mainly from modeling activity use cases) are submitted to KnowledgeSpace so that they can be reused.
  • both NIP and knowledgeSpace use the same ontology service SciGraph (developed by KnowledgeSpace) and thus the same set of ontology inferences it enables.
  • an ontology for registering brain parcellation schemes were co-designed by the two platforms. This ontology should enable a better integration of existing atlas ontologies within the two platforms.

Christian, as a neuroscience researcher, is interested in finding information related to the Thalamus in the mouse brain.

He browses to the Multiresolution Atlas Viewer and switch dataset to the Allen Reference Atlas v3 for the mouse by using the select list.

Use case - switch dataset

He clicks on the “Draw brain regions” checkbox to enable the overlaid annotations.

Use case - enable overlayed annotations

He uses the mouse scroll wheel to zoom in a bit to focus on the relevant part of the brain slice, then he puts the mouse cursor over the overlay regions to find the Thalamus, as he can see the brain region name at the top of the screen.

Use case - show brain region name

He clicks on the Thalamus region, which opens a popup with a list of thematic resources about the brain region.

Use case - open thematic resources

He can see there are 81 morphology results in NeuroMorpho, and clicks on the NeuroMorpho link. He lands on a KnowledgeSpace page with the links to all individual NeuroMorpho results for the Thalamus.

Use case - open NeuroMorpho link

Since Christian is mostly interested in the Allen v3 dataset, he bookmarks a direct link to it in his browser.

Use case - bookmark direct link

Programmatic Interfaces

Intended Audience: this part of the documentation is intended mainly at developers wishing to interface with the Neuroinformatics platform through a programmatic interface.

Ksearch

Purpose

Ksearch is the search component of the Neuroinformatics Platform (NIP). It is a REST API allowing searching curated datasets using different filters that are mainly taken from MINDS.

Features

Ksearch allows searching datasets shared in NIP. There are five main features:

  • Full text search:
    • free text search on all dataset fields: all datasets fields will be searched,
    • field free text search: specific dataset fields can be targeted when search for a term,
  • Boolean search:
    • Boolean operator (AND, OR,…) can be specified between the keywords of a text
  • Filtering: a set of filters are available to narrow down search results

  • Query expansion through term synonyms and specialisation: for example a search for “Rodentia” will be able to retrieve datasets annotated with “Rattus Norvegicus” because “Rattus Norvegicus” is a rodent

  • Documents aggregation: Ksearch results can be grouped by data modality, brain regions by other metadata available in MINDS. This feature enables document exploration and faceted navigation.

Details of how the previous Ksearch features are implemented in term of endpoints are described as follow.

Ksearch endpoints

The following endpoints are exposed by Ksearch. As shown in the figure below, only read access (GET http verb) to data is allowed. An interactive documentation of Ksearch API is available here.

ksearch search endpoints

Verticals

Verticals are the types of documents exposed for search in Ksearch. This means a call to one of the /search endpoints will always return documents whose type is one of the verticals. The list of available verticals can be obtained by calling:

curl -X GET https://nip.humanbrainproject.eu/api/ksearch/verticals

A sample response is:

[
  {
    "name": "dataset",
    "schema_path": "https://schema.humanbrainproject.eu/neuroinformatics/search/search-dataset-schema-v1.0.json",
    "filters": [
      {
        "name": "species",
        "field": "specimen.species.term.raw_term",
        "description": "Species",
        "type": "String"
      },
      {
        "name": "methods",
        "field": "methods.term.raw_term",
        "description": "Methods",
        "type": "String"
      },
      {
        "name": "protocols",
        "field": "protocols.title.raw_term",
        "description": "Protocols",
        "type": "String"
      },
      {
        "name": "classifications",
        "field": "classifications.term.raw_term",
        "description": "Cell types",
        "type": "String"
      },
      {
        "name": "data_modalities",
        "field": "data_modalities.term.raw_term",
        "description": "Data Modalities",
        "type": "String"
      },
      {
        "name": "full_name",
        "field": "",
        "description": "Contributing organisations",
        "type": "String"
      },
      {
        "name": "affiliations",
        "field": "contributors.affiliations.raw_affiliations",
        "description": "Contributor's affiliations",
        "type": "String"
      },
      {
        "name": "brain_regions",
        "field": "brain_regions.term.raw_term",
        "description": "Brain Regions",
        "type": "String"
      }
    ]
  }
]

For each vertical, the following fields are retrieved:

Field Description
name A vertical name which is a document type: dataset for example
schema_path The URI of the schema the vertical instances (the documents) are consistent with
filters Available filters that can be used to narrow down results having the current vertical as type

The term ‘document’ will be used below to name Ksearch results.

Response format

A call to a Ksearch endpoint will return search results in json format. All Ksearch results are wrapped in the following json document (which is consistent with this json schema):

{
  "took" : 26,
  "total" : 2126,
  "max_score" : 1.0,
  "hits" : [ ],
  "aggregations":[]
}
Field Description Data type Default
took The query execution duration in milliseconds long 0.0
total the total number of results integer 0
max_score The highest result score float 1.0
hits The array of the results array empty
aggregations a list of buckets/facets that group results array empty

An element of the “hits” array is a result wrapping a document (a dataset for example) that matches the user query:

  {
    "id" : "05b4691e-ee47-11e4-8074-6003088da632",
    "score" : 1.0,
    "source" : { },
    "schema" : ""
 }

The corresponding json schema of the previous result is available here.

Field Description Data type Default
id The identifier of the result String NA
score A number measuring how much a document matches the user query float 1.0
source The actual document object NA
schema The schema with which the result (in source field) is consistent String NA

Results Pagination

Large number of datasets can be potentially retrieved from Ksearch when making a call. In that case, it will be useful to have results paginated. The Ksearch endpoints supporting results pagination are:

ksearch search endpoints

When using the above search endpoints, the size of a page and the start point (offset) of the page can be controlled respectively
with “size” and “from” parameters:

Parameter Description Data type Default
from the index (within results set) of the first result you want to get integer 0
size the total number of results you want to get integer 20

To go through the results page by page, set the size value to a value S and start from 0:

  • get the S first results
  • the nth page will start from (n-1) x S (n starting by 2) while the size stays the same.
  • If from+size >= total result set size, then all result pages have been visited.

Filtering syntax

Filtering allows retrieving documents that match exact values for specified fields. It is the base of faceted/aggregated search. Two things are then needed to build a filter query: fields and their corresponding exact values. Note that only /search endpoints allow documents filtering. The syntax of a filter parameter in Ksearch is as follows:

  filters={"field_1":["value1_1","value1_2",...],"field_2":["value2_1"]}

As an example, let retrieve datasets that fall into “Single cell morphology reconstruction” data modality:

  curl -g -X GET "https://nip.humanbrainproject.eu/api/ksearch/search?q=*
&filters={%22data_modalities.term.raw_term%22:[%22Single%20cell%20morphology%20reconstruction%22]}"

Only an AND filtering strategy is currently implemented. With the above syntax, it means that only documents with value1_1 and value1_2 as field_1 values and with value2_1 as field_2 value, will be retrieved.

Sorting syntax

Results can be sorted in an ascendant or descending way and by fields using the following syntax:

  sort={"field_1":”asc”,"field_2":"desc"}

For example, the following query will ordered documents by title:

  curl -g -X GET "https://nip.humanbrainproject.eu/api/ksearch/search?q=*&sort={%22title%22:%22desc%22}"

Note that by default Ksearch results are sorted by score.

Field selection

A user can specify which document fields to retrieve when using Ksearch or the fields he want to exclude.

The syntax of field selection is:

  retrieve={"field_1":["subfield"],"field_2":["subfield"]}

For example, instead of retrieving all dataset fields defined in MINDS, a user can choose to only retrieve dataset id, title and the affiliations of its contributors:

  curl -g -X GET "https://nip.humanbrainproject.eu/api/ksearch/search?q=*
&retrieve={%22id%22:[%22%22],%22title%22:[%22%22],%22contributors%22:[%22affiliations%22]}"

Let exclude the field “specimen” of datasets:

  curl -g -X GET "https://nip.humanbrainproject.eu/api/ksearch/search?q=*&exclude={%22specimen%22:[%22%22]}"

The “retrieve” and “exclude” parameters can be mixed. In that case, the “exclude” parameter takes precedence over the “retrieve” one: an excluded field will never be retrieved.

With the following query, only documents having the keywords rat AND morphologies as values of at least one field will be retrieved:

  curl -X GET "https://nip.humanbrainproject.eu/api/ksearch/search?q=rat%20morphologies"

To consider “rat morphologies” as a phrase and not as two tokens, double quotes can be used:

  curl -X GET "https://nip.humanbrainproject.eu/api/ksearch/search?q=%22rat%20morphologies%22"

So by default, the full text search is a conjunctive boolean query between user keywords. However, the keywords can be searched in specific fields. Let search for documents with “rat” or “morphologies” in their “title”:

  curl -X GET "https://nip.humanbrainproject.eu/api/ksearch/search?q=title:rat%20OR%20title:morphologies"

The possible boolean operators are: AND, OR and NOT.

Aggregation

By default, a call to a Ksearch search endpoint will retrieve a results list (hits) and groups of results which are retrieved under the “aggregations” array. To disable the aggregations, the parameter “aggreg” can be set to “false”:

  curl -X GET "https://nip.humanbrainproject.eu/api/ksearch/search?q=*&aggreg=false"

To only retrieve aggregations ( not the hit list), the “size” parameter can be set to 0:

  curl -X GET "https://nip.humanbrainproject.eu/api/ksearch/search?q=*&size=0"

Let now see how to use aggregations to narrow down Ksearch results. An aggregation is a json object with at least the following fields (the complete schema is available here):

{
  "aggregations": [
    {
      "buckets": [
        {
          "key": "Untufted pyramidal cell",
          "doc_count": 49
        },
        {
          "key": "Tufted pyramidal cell towards layer IV",
          "doc_count": 15
        },
        {
          "key": "Tufted pyramidal cell towards layer I",
          "doc_count": 17
        }
      ],
      "name": "Cell Types",
      "field": "classifications.term.raw_term"
    }
  ]
}
field Description
name The name of the aggregation
field The field to use for document filtering (see filter syntax)
buckets The list of document groups: groups of a number of “aggregations[].buckets[].doc_count” documents with “aggregations[].buckets[].key” value for the field “aggregations[*].field”

The value of aggregations[].field can be used as filter field and the value of aggregations[].buckets[*].key can be used as filter value.

How to download a dataset

Dataset metadata can be retrieved by full text search or by id. The metadata of the dataset with id “b9cab473-9f32-4260-9a73-cc7d714ec791” can be obtained using the following call:

  curl -X GET "https://nip-dev.humanbrainproject.eu/api/ksearch/datasets/b9cab473-9f32-4260-9a73-cc7d714ec791"

A sample response is:

  {
  "title" : "Electrophysiology [Action potential dependent sIPSCs from a 3-4 months old C57BL/6J X SJL female mice hippocampus CA1 pyramidal cell (B1)]",
  "uuid" : "b9cab473-9f32-4260-9a73-cc7d714ec791",
  "data_modalities" : [ {
    "id" : "HBP_DAMO:0000008",
    "term" : "Electrophysiology"
  } ],
  "specimen" : [ {
    "uuid" : "e0b9fa0f-e217-4cf4-84cc-af82f58e273e",
    "species" : {
      "id" : "obo:NCBITaxon_10090",
      "term" : "Mus musculus"
    },
    "strain" : {
      "id" : "HBP_TAX:0000015",
      "term" : "C57BL/6J X SJL"
    },
    "sex" : "female",
    "age" : "Post-natal 90.0"
  } ],
  "contributors" : [ ... ],
  "representations" : [ {
    "access" : "https://zenodo.org/record/XXX/files/expYYYY.txt",
    "datatype" : {
      "id" : "HBP_DATT:0000077",
      "term" : "application/vnd.ebri.traces.txt;version=0"
    },
    "storagetype" : {
      "id" : "HBP_STO:0000007",
      "term" : "Zenodo",
      "term_suggest" : {
        "output" : "Zenodo [null]",
        "input" : [ "Zenodo" ]
      }
    },
    "uuid" : "0a588da8-8cde-44af-abed-eb88790f7fc6"
  } ],
  "protocols" : [...],
  "classifications" : [ {
    "id" : "HBP_CELL:0000145",
    "term" : "Hippocampus CA1 pyramidal cell"
  } ],
  "brain_regions" : [ {
    "id" : "MBA:382",
    "term" : "Field CA1"
  } ]
}

The “representations” field of the above dataset holds the necessary download information:

  curl -g -X GET "https://nip.humanbrainproject.eu/api/ksearch/datasets/b9cab473-9f32-4260-9a73-cc7d714ec791
?retrieve={%22representations%22:[%22%22]}}"

A sample response is:

  {
  "representations" : [ {
    "access" : "https://zenodo.org/record/XXX/files/expYYYY.txt",
    "datatype" : {
      "id" : "HBP_DATT:0000077",
      "term" : "application/vnd.ebri.traces.txt;version=0"
    },
    "storagetype" : {
      "id" : "HBP_STO:0000007",
      "term" : "Zenodo"
    },
    "uuid" : "0a588da8-8cde-44af-abed-eb88790f7fc6"
  } ]
}

For each dataset, the ‘representations’ array holds the following fields:

field Description
storagetype The storage where the raw registered data are stored.
access An id or a URL that can be used (with the storage type) to download the data:If storagetype.term=”Zenodo” then “access” is the url of the raw data (access=”https://zenodo.org/record/XXX/files/expYYYY.txt” in the example above)If storagetype.term=”HBP Document Service” then the value of “access” is a UUID to be used with HBP internal storage (not available from outside)
datatype The type of data: “Neurolucida ASC” for example

The list of available storage types can be retrieved from Ksearch using the following call:

  curl -X GET "https://nip.humanbrainproject.eu/api/ksearch/datasets/storage/types"

A sample response is:

  [ {
  "description" : "Zenodo builds and operates a service that enables researchers to share and preserve research results.",
  "curie" : "HBP_STO:0000007",
  "name" : "Zenodo"
}, {
  "description" : "Repository maintained by Human Brain Project.",
  "curie" : "HBP_STO:0000008",
  "name" : "HBP Document service"
} ]

Scigraph can be used to see more details about the above storage type ontology terms.

Demo

A demo of the Ksearch API is available here.

Scigraph: a graph-based ontology service

Purpose

Scigraph is the Neuroinformatics platform (NIP) ontology service and is implemented as a REST API. As such, it provides access to neuroscience related ontologies (brain regions, cell types, species, measurement methods,…) . As a REST API, Scigraph is best suited for programmatic usage and can be involved in most data integration pipelines whenever ontology data access is needed. There is a web-based client available for a more user friendly view or browse of ontologies available in the NIP platform.

It is not possible to use scigraph to import/load any ontologies into the NIP platform. To achieve that, please see the ontology page in the curation section or contact data.nip@humanbrainproject.eu.

Features

Scigraph provides read access to the ontologies used in the NIP platform. To simplify, the following entity types can be retrieved from scigraph:

  • ontology terms:in Allen human brain atlas v2 ontology, “gray matter” with http://api.brain-map.org/api/v2/data/Structure/4006 as identifier is an ontology term,
  • properties/attributes: Boolean operator (AND, OR,…) can be specified between the keywords of a text,
  • relations: ontology terms are linked by the mean of relations.

All the previous ontology elements will be referred to as ontology entities below. Using Scigraph, a client can:

  • Get the list of available ontologies
  • Get the list of ontology entities that are defined in a specific ontology
  • Get a specific ontology entity along with it’s properties or attributes:
    • Labels
    • definitions
    • synonyms
    • acronyms
    • abbreviations
  • Traverse the ontology graph: retrieve terms that in relation with a specific ontology term.
  • Search for ontology entities by its properties
  • Get ontology entity suggestions from a text query
  • Autocomplete a text query (useful to implement a did you mean functionality)

Ontology entities are referred to with their identifiers in Scigraph. Because those identifiers can take different forms and thus be confusing, the next section will introduce them.

Ontology entity identifiers in Scigraph

An ontology as well as an ontology entity has a URI as identifier. For example, the URI of the Allen human brain atlas ontology (as integrated in NIP) is http://api.brain-map.org/api/v2/data/Structure, while the URI of the specific term “gray matter” is http://api.brain-map.org/api/v2/data/Structure/4006. The previous two URIs can have a short form which is called prefix (a stable string) for the ontology and CURIE for the ontology entities. Let take again the previous example. The prefix name of (the short form of) “http://api.brain-map.org/api/v2/data/Structure” can be ‘HBA’ while the curie of the term “grey matter” is 'HBA:4006’. Given the curie 'HBA:4006’, 'HBA’ is the prefix name and '4006’ is the fragment.

Scigraph REST API endpoints

An interactive documentation of the Scigraph API is available here as well as a wiki page with more details. The next points describe how Scigraph endpoints can be used to benefit from the previous features in the context of NIP.

Scigraph REST API data formats

All Scigraph responses are in json format as described here.

Vocabulary endpoint

An overview of the vocabulary endpoints:

Scrapgh vocabulary endpoints

  • list available ontology prefix names useful as filter to narrow down ontology entities results:
curl -X GET "https://nip.humanbrainproject.eu/api/scigraph/vocabulary/prefixes"

A sample response is:

{
  "list": [
    "MESH",
    "HBP_REC",
    "obo",
    "efo",
    "nifstd",
    "MBA",
    "HBP_ACT",
    "HBP_CELL",
    "HBP_DAMO",
    "HBP_DATT",
    "HBP_HBA",
    "HBA",
    "HBP_ROLE",
    "HBP_SCL",
    "HBP_SEX",
    "HBP_STO",
    "HBP_LICENSE",
    "cc",
    "HBP_TAX",
    "HBP_MEM",
    "HBP_DTAT",
    "JHBA",
    "HBP_WHS_SD",
    "HBP_BATT"
  ]
}

Only ontology prefix names are obtained from the API by the previous GET call.

  • Get all entities defined in a given ontology identified by it’s prefix name

Let retrieve all entities defined in HBA (Allen human brain atlas ontology v2):

curl -X GET "https://nip.humanbrainproject.eu/api/scigraph/vocabulary/search/*?prefix=HBA"

A sample response is:

{
  "concepts": [
    {
      "uri": "http://api.brain-map.org/api/v2/data/4006",
      "labels": [
        "gray matter"
      ],
      "fragment": "4006",
      "curie": "HBA:4006",
      "synonyms": [
        "GM"
      ],
      "acronyms": [],
      "abbreviations": [],
      "deprecated": false,
      "definitions": []
    },
    ...
  ]
}

For each ontology entity retrieved from the vocabulary endpoints, the following fields are available:

Field Description
URI The identifier of the ontology entity
labels An array of the ontology entity names
fragment The last part of the URI after the last separator character (mainly after / or #)
curie The short form of the ontology entity URI
synonyms An array of the ontology entity synonyms
acronyms An array of the ontology entity acronyms
abbreviations An array of the ontology entity abbreviations
deprecated A flag to indicate whether the ontology entity is deprecated (true) or not (false)
definitions An array of the ontology entity definitions
  • Search for a specific ontology entity:
    • by id

The entity identifier can be either a curie or a fragment. Let search for the entity with the curie HBA:4006

curl -X GET "https://nip.humanbrainproject.eu/api/scigraph/vocabulary/id/HBA:4006"

The response is:

{
  "concepts": [
    {
      "uri": "http://api.brain-map.org/api/v2/data/4006",
      "labels": [
        "gray matter"
      ],
      "fragment": "4006",
      "curie": "HBA:4006",
      "synonyms": [
        "GM"
      ],
      "acronyms": [],
      "abbreviations": [],
      "deprecated": false,
      "definitions": []
    }
  ]
}
  • by labels

Let search for “gray matter”:

curl -X GET "https://nip.humanbrainproject.eu/api/scigraph/vocabulary/term/gray matter"

When searching for a multi-token term (“gray matter” for example), ontology entities that match an individual token (“gray” or “matter”) will be retrieved. There is thus no guarantee to only retrieve one result with the previous query. Also a targeted ontology prefix name can be specified when searching by label for any ontology entity. In such case, only entities defined in the specified ontology will be retrieved.

curl -X GET "https://nip.humanbrainproject.eu/api/scigraph/vocabulary/term/gray matter?prefix=HBA"
  • by synonyms

Search by synonyms can be enabled by setting the parameter “searchSynonyms” to true:

curl -X GET "https://nip.humanbrainproject.eu/api/scigraph/vocabulary/term/gray matter?searchSynonyms=true"
  • suggest ontology entity by label (useful to implement did you mean feature and spell checking features):
curl -X GET "https://nip.humanbrainproject.eu/api/scigraph/vocabulary/suggestions/grey mtter?limit=1"

A sample response is:

{
  "list": [
    "gray matter"
  ]
}
  • autocomplete query:
curl -X GET "https://nip.humanbrainproject.eu/api/scigraph/vocabulary/autocomplete/inferior occipital?limit=1

A sample response is:

{
  "list": [
    {
      "completion": "gray matter",
      "type": "label",
      "concept": {
        "uri": "http://api.brain-map.org/api/v2/data/4006",
        "labels": [
          "gray matter"
        ],
        "fragment": "4006",
        "curie": "HBA:4006",
        "categories": [],
        "synonyms": [
          "GM"
        ],
        "acronyms": [],
        "abbreviations": [],
        "deprecated": false
      }
    }
  ]
}

The vocabulary endpoints doesn’t allow to navigate between ontology entities. For example, it’s not possible to know which brain parcels “gray matter” is sub-region of. The graph endpoints described in the following paragraph can answer to that question.

Graph endpoint

An overview of the graph endpoints:

Scigraph graph endpoints

It allows to:

  • get a specific ontology entity by curie or fragment:
curl -X GET "https://nip.humanbrainproject.eu/api/scigraph/graph/HBA:4006"

A sample response is:

{
  "nodes": [
    {
      "id": "HBA:4006",
      "lbl": "gray matter",
      "meta": {
        "fragment": [
          "4006"
        ],
        "synonym": [
          "GM"
        ],
        "types": [
          "Class"
        ],
        "http://www.w3.org/2000/01/rdf-schema#label": [
          "gray matter"
        ],
        "http://www.FIXME.org/nsupper#synonym": [
          "GM"
        ]
      }
    }
  ],
  "edges": []
}

Note the vocabulary switch between vocabulary endpoints and graph endpoints responses. In vocabulary endpoints, “concepts” is used as container of retrieved entities while “nodes” and “edges” are used in graph endpoints. The navigation between ontology entities is done by following edges.

  • get the available relationship types between ontology entities:
curl -X GET "https://nip.humanbrainproject.eu/api/scigraph/graph/relationship_types"

A sample response is:

[
  "equivalentClass",
  "filler",
  "has_domain_aspect",
  "has_rank",
  "isDefinedBy",
  "is_smaller_than",
  "legalcode",
  "part_of",
  "property",
  "seeAlso",
  "subClassOf"
]
  • get the ontology terms that are in relation with (in the neighbor of) a specific one identified by an id (curie or fragment) and through a specific relation (part_of is used in the example below) taken from the above relationship types:
curl -X GET "https://nip.humanbrainproject.eu/api/scigraph/graph/neighbors/HBA:4006?depth=1&relationshipType=part_of&direction=OUTGOING"

The direction of the relation can be specified through the “direction” parameter.The possible values are “OUTGOING”, “INCOMING” AND “BOTH”. A sample response is:

{
  "nodes": [
    {
      "id": "HBA:4005",
      "lbl": "brain",
       ...
    },
    {
      "id": "HBA:4006",
      "lbl": "gray matter",
      ...
    }
  ],
  "edges": [
    {
      "sub": "HBA:4006",
      "obj": "HBA:4005",
      "pred": "part_of",
      "meta": {}
    },
    ...
  ]
}

The ‘edges’ array in the graph endpoints responses has the following fields:

Field Description
sub (for subject) is the curie or URI of entity under interest. HBA:4006 in the example above.
obj (for object) is the curie or URI of a neighbor of the term referenced in sub
pred (for predicate) a relation (curie, uri or fragment) linking sub and obj

Navigation through ontology terms is possible by getting the neighbors of the entity having the value of “obj” as id.

Demo

An iPython notebook demonstration is available here.

Voxel Brain

Purpose

This REST service provides access to volumetric brain atlas and their annotation layers in the form of voxels.

Endpoints

HTTP Method Endpoint Usage
GET https://nip.humanbrainproject.eu/api/analytics/atlas/metadata List metadata available for a given atlas. It provides a description for each metadata item.
GET https://nip.humanbrainproject.eu/api/analytics/atlas/filters List all parameters users can leverage to filter when looking for atlases.
GET https://nip.humanbrainproject.eu/api/analytics/atlas/releases List all available atlases, filters can be specified to restrict results. Note that the ‘id’ field for each atlas is subsequently used to refer to a given atlas. We show it as [ID] in subsequent API calls.
GET https://nip.humanbrainproject.eu/api/analytics/atlas/releases/[ID] Lookup an atlas by its identifier.
Parameters
[ID] denotes the atlas we search in.
GET https://nip.humanbrainproject.eu/api/analytics/atlas/releases/[ID]/data Lists all annotation layers available for that atlas. Each layer is an independent data file in NRRD format.
Parameters
[ID] denotes the atlas we search in.
GET https://nip.humanbrainproject.eu/api/analytics/atlas/releases/[ID]/filters/brain_region/search/[TERM] Search brain regions by name in a given atlas parcellation scheme. The given query can match either the name or acronym of ontological terms.
Parameters
[ID] denotes the atlas we search in.
[TERM] denotes the ontology term name or acronym (not case sensitive, substring matching).
GET http://nip.humanbrainproject.eu/api/analytics/atlas/download?uri=[LAYER URI] Typically, this API call is not built by the developer but given by the output of the API call: /api/analytics/atlas/releases/[ID]/data

An interactive version of this endpoint documentation is available here.

Use Case Illustration: Build a virtual brain

Description

To build a virtual brain for the purpose of running a software simulation, a user needs to get access to data positioned in a brain template in the form of voxels. The data required for this exercise need to be aligned in a brain template, e.g. the Allen mouse CCFv2 with voxel size of 25 micron.

The following information need to be made available for each voxels in the accessed data:

  • brain region id
  • Boolean value or probabilistic distribution indicating if cells are located in a given voxel
  • Intensity values of given image modalities

The template is annotated using brain regions. These brain regions are represented in a hierarchical structure, i.e. a region can have sub-regions.This brain region hierarchy is specific to the given brain template. The user needs the ability to filter target voxels by a given brain region. If the given brain region has sub-regions, all voxels annotated with either the parent regions or sub-regions should be selected. The user should have the ability to search for a brain region by its name or acronym (non case sensitive search). For a given brain region id, the user should be able to retrieve the hierarchy of its children.

How to use the API to execute this use case

Step 1 - The user looks up for all existing releases available

API call:

curl -X GET https://nip.humanbrainproject.eu/api/analytics/atlas/releases

returns

[
  {
    "id": "6C4F0E89-6D2A-46D4-B4D5-B2BE4840F87D",
    "name": "Mouse 25 micron CCFv2",
    "species": "mouse",
    "resolution": "25",
    "template_source": "Allen mouse brain CCFv2",
    "hbp_release": "2016-01-12",
    "data_types": [
      "nissl",
      "gray_levels",
      "brain_regions"
    ]
  },
  {
    "id": "1192CEAC-17C5-4A74-B88C-48065C048678",
    "name": "Mouse 25 micron CCFv3",
    "species": "mouse",
    "resolution": "25",
    "template_source": "Allen mouse brain CCFv3",
    "hbp_release": "2016-01-12",
    "data_types": [
      "nissl",
      "gray_levels",
      "brain_regions"
    ]
  }
]

The user selects the above release and keep notes of its id: 6C4F0E89-6D2A-46D4-B4D5-B2BE4840F87D.

Step 2 - The user lookup filters available in the system

API call:

curl -X GET https://nip.humanbrainproject.eu/api/analytics/atlas/filters

returns

[
  {
    "filter": "data_type",
    "description": "The data modality",
    "type": "string",
    "values": [
      "gray_levels",
      "brain_regions"
    ]
  },
  {
    "filter": "brain_region",
    "description": "The region of the brain a voxel is located into",
    "type": "string",
    "values": [
      "cortex",
      "hypocampus"
    ]
  }
]

Here we can see that there are two filters available: - data_type - brain_region

Step 3 - The user searches in this release for brain regions by name: ‘hippocamp’

API call:

curl -X GET https://nip.humanbrainproject.eu/api/analytics/atlas/releases/6C4F0E89-6D2A-46D4-B4D5-B2BE4840F87D/filters/brain_region/search/hippocamp

returns:

[
  [
    "1089",
    "Hippocampal formation",
    "HPF"
  ],
  [
    "1080",
    "Hippocampal region",
    "HIP"
  ],
  [
    "822",
    "Retrohippocampal region",
    "RHP"
  ],
  [
    "333",
    "Septohippocampal nucleus",
    "SH"
  ],
  [
    "618",
    "hippocampal commissures",
    "hc"
  ],
  [
    "443",
    "dorsal hippocampal commissure",
    "dhc"
  ],
  [
    "449",
    "ventral hippocampal commissure",
    "vhc"
  ],
  [
    "1063",
    "hippocampal fissure",
    "hf"
  ]
]

The user inspects the returned list above, and select the term 'hippocampal region’ for which the ID is 1080.

Step 4 - The user now searches for all children terms of this term by ID

API call:

curl -X GET https://nip.humanbrainproject.eu/api/analytics/atlas/releases/6C4F0E89-6D2A-46D4-B4D5-B2BE4840F87D/filters/brain_region/1080

returns

{
  "id": 1080,
  "acronym": "HIP",
  "name": "Hippocampal region",
  "children": [
    {
      "id": 375,
      "acronym": "CA",
      "name": "Ammon's horn",
      "children": [
        {
          "id": 382,
          "acronym": "CA1",
          "name": "Field CA1",
          "children": [
            {
              "id": 391,
              "acronym": "CA1slm",
              "name": "Field CA1, stratum lacunosum-moleculare"
            },
            {
              "id": 399,
              "acronym": "CA1so",
              "name": "Field CA1, stratum oriens"
            },
            {
              "id": 407,
              "acronym": "CA1sp",
              "name": "Field CA1, pyramidal layer"
            },
            {
              "id": 415,
              "acronym": "CA1sr",
              "name": "Field CA1, stratum radiatum"
            }
          ]
        },
        {
          "id": 423,
          "acronym": "CA2",
          "name": "Field CA2",
          "children": [
            {
              "id": 431,
              "acronym": "CA2slm",
              "name": "Field CA2, stratum lacunosum-moleculare"
            },
            {
              "id": 438,
              "acronym": "CA2so",
              "name": "Field CA2, stratum oriens"
            },
            {
              "id": 446,
              "acronym": "CA2sp",
              "name": "Field CA2, pyramidal layer"
            },
            {
              "id": 454,
              "acronym": "CA2sr",
              "name": "Field CA2, stratum radiatum"
            }
          ]
        },
        {
          "id": 463,
          "acronym": "CA3",
          "name": "Field CA3",
          "children": [
            {
              "id": 471,
              "acronym": "CA3slm",
              "name": "Field CA3, stratum lacunosum-moleculare"
            },
            {
              "id": 479,
              "acronym": "CA3slu",
              "name": "Field CA3, stratum lucidum"
            },
            {
              "id": 486,
              "acronym": "CA3so",
              "name": "Field CA3, stratum oriens"
            },
            {
              "id": 495,
              "acronym": "CA3sp",
              "name": "Field CA3, pyramidal layer"
            },
            {
              "id": 504,
              "acronym": "CA3sr",
              "name": "Field CA3, stratum radiatum"
            }
          ]
        }
      ]
    },
    {
      "id": 726,
      "acronym": "DG",
      "name": "Dentate gyrus",
      "children": [
        {
          "id": 10703,
          "acronym": "DG-mo",
          "name": "Dentate gyrus, molecular layer"
        },
        {
          "id": 10704,
          "acronym": "DG-po",
          "name": "Dentate gyrus, polymorph layer"
        },
        {
          "id": 632,
          "acronym": "DG-sg",
          "name": "Dentate gyrus, granule cell layer"
        },
        {
          "id": 10702,
          "acronym": "DG-sgz",
          "name": "Dentate gyrus, subgranular zone"
        },
        {
          "id": 734,
          "acronym": "DGcr",
          "name": "Dentate gyrus crest",
          "children": [
            {
              "id": 742,
              "acronym": "DGcr-mo",
              "name": "Dentate gyrus crest, molecular layer"
            },
            {
              "id": 751,
              "acronym": "DGcr-po",
              "name": "Dentate gyrus crest, polymorph layer"
            },
            {
              "id": 758,
              "acronym": "DGcr-sg",
              "name": "Dentate gyrus crest, granule cell layer"
            }
          ]
        },
        {
          "id": 766,
          "acronym": "DGlb",
          "name": "Dentate gyrus lateral blade",
          "children": [
            {
              "id": 775,
              "acronym": "DGlb-mo",
              "name": "Dentate gyrus lateral blade, molecular layer"
            },
            {
              "id": 782,
              "acronym": "DGlb-po",
              "name": "Dentate gyrus lateral blade, polymorph layer"
            },
            {
              "id": 790,
              "acronym": "DGlb-sg",
              "name": "Dentate gyrus lateral blade, granule cell layer"
            }
          ]
        },
        {
          "id": 799,
          "acronym": "DGmb",
          "name": "Dentate gyrus medial blade",
          "children": [
            {
              "id": 807,
              "acronym": "DGmb-mo",
              "name": "Dentate gyrus medial blade, molecular layer"
            },
            {
              "id": 815,
              "acronym": "DGmb-po",
              "name": "Dentate gyrus medial blade, polymorph layer"
            },
            {
              "id": 823,
              "acronym": "DGmb-sg",
              "name": "Dentate gyrus medial blade, granule cell layer"
            }
          ]
        }
      ]
    },
    {
      "id": 982,
      "acronym": "FC",
      "name": "Fasciola cinerea"
    },
    {
      "id": 19,
      "acronym": "IG",
      "name": "Induseum griseum"
    }
  ]
}

The user now decides to go ahead and to collect data for this region.

Step 5 - The user introspect the release to find out which data_types are available

API call:

curl -X GET https://nip.humanbrainproject.eu/api/analytics/atlas/releases/6C4F0E89-6D2A-46D4-B4D5-B2BE4840F87D/data

returns

[
  {
    "url": "http://nip-dev.humanbrainproject.eu:80/api/analytics/atlas/download?uri=6C4F0E89-6D2A-46D4-B4D5-B2BE4840F87D/brain_regions/brain_regions.nrrd",
    "atlas_id": "6C4F0E89-6D2A-46D4-B4D5-B2BE4840F87D",
    "data_type": "brain_regions"
  },
  {
    "url": "http://nip-dev.humanbrainproject.eu:80/api/analytics/atlas/download?uri=6C4F0E89-6D2A-46D4-B4D5-B2BE4840F87D/gray_levels/gray_levels.nrrd",
    "atlas_id": "6C4F0E89-6D2A-46D4-B4D5-B2BE4840F87D",
    "data_type": "gray_levels"
  },
  {
    "url": "http://nip-dev.humanbrainproject.eu:80/api/analytics/atlas/download?uri=6C4F0E89-6D2A-46D4-B4D5-B2BE4840F87D/nissl/nissl.nrrd",
    "atlas_id": "6C4F0E89-6D2A-46D4-B4D5-B2BE4840F87D",
    "data_type": "nissl"
  }
]

Here we can see that for the selected release, there are 3 data_types available:

  • brain_regions: integer values that match the identifiers of the corresponding ontology terms
  • gray_levels: integer value of the image data of the averaged mouse template
  • nissl: numerical values of the nissl staining

The user notes that at this stage, all data_types are distributed using the NRRD data format. This is likely to change in the future in favour of an Avro based format.

Step 6 - Now the user requests the same data_types but this time filtered by the chosen region 1080 (i.e. hippocampal region)

API call:

curl -X GET https://nip.humanbrainproject.eu/api/analytics/atlas/releases/6C4F0E89-6D2A-46D4-B4D5-B2BE4840F87D/data?brain_region=1080

returns

[
  {
    "url": "http://nip-dev.humanbrainproject.eu:80/api/analytics/atlas/download?uri=6C4F0E89-6D2A-46D4-B4D5-B2BE4840F87D/brain_regions/brain_region/1080.nrrd",
    "atlas_id": "1192CEAC-17C5-4A74-B88C-48065C048678",
    "data_type": "brain_regions"
  },
  {
    "url": "http://nip-dev.humanbrainproject.eu:80/api/analytics/atlas/download?uri=6C4F0E89-6D2A-46D4-B4D5-B2BE4840F87D/gray_levels/brain_region/1080.nrrd",
    "atlas_id": "1192CEAC-17C5-4A74-B88C-48065C048678",
    "data_type": "gray_levels"
  },
  {
    "url": "http://nip-dev.humanbrainproject.eu:80/api/analytics/atlas/download?uri=6C4F0E89-6D2A-46D4-B4D5-B2BE4840F87D/nissl/brain_region/1080.nrrd",
    "atlas_id": "1192CEAC-17C5-4A74-B88C-48065C048678",
    "data_type": "nissl"
  }
]

The user notes that the same data_types are returned. However the content of these NRRD files are limited to the voxels that are annotated with the requested brain region (i.e. 1080) or any of its children. Now the user only needs to use the URL provided in each data_type to download the corresponding NRRD file.

Image Service

Purpose

Image Service provides access to image tiles through REST API operation request. Image tiles are created at different zoom levels for sagittal, coronal and/or axial orientations, in order to display two dimensional large images (captured from brain slices) of brain imaging dataset (volumetric) through browser. These tiled images become part of image pyramid. All tiles of each orientations are written into one single file as stacks in BBIC format. Image Service also provides meta information about the brain image stack file structure and content through operation request. The client application that consumes Image Service is currently Multiresolution Atlas Viewer.

Features

We could get a single tile at certain zoom level, meta information about the image stack, and a tile after running minimum or maximum intensity projection of several slices like 64. These query requests are explained below:

Meta information query:

<image-service>/bbic?mode=meta&fname=<fname>

Tile query:

<image-service>/bbic?mode=ims&fname=<fname>&prog=’TILE 0’stack=<n>&level=k(&slice=<m>|&slice_orig_id=<id>)&x=<x>&y=<y>&proj=<proj>&proj_d=<d>&extend_tiles=<e>

This query returns a tile image specified by BBIC filename , stack number , slice number , detail level , “x” position , “y” position and optional parameters: specifying projection type, specifying projection depth and specifying handling of last row/column/diagonal tiles, which are usually smaller than declared .

In the following figure, we could see image tiles at different zoom levels.

Sagittal slice of BigBrain dataset were divided into image tiles in different zoom levels

Endpoints

HTTP Method Endpoint Usage Returns
GET https://nip.humanbrainproject.eu/api/image-service/image/v0/api/bbic?fname=/gpfs/bbp.cscs.ch/project/proj39/template/human/bigbrain/release2015/bbic/sections/bigbrain.h5&mode=ims&prog=TILE 0 2 2 3285 4 3 none 10 1 Get a tile of BigBrain without any image manipulation Image tile (original)
GET https://nip.humanbrainproject.eu/api/image-service/image/v0/api/bbic?fname=/gpfs/bbp.cscs.ch/project/proj39/template/rat/waxholm/v2/bbic/sections/whs.h5&mode=ims&prog=TILE 0 0 1 511 0 0 max 8 1 Get a tile of Waxholm with maximum intensity projection of 8 slices Tile with image manipulation
GET https://nip.humanbrainproject.eu/api/image-service/image/v0/api/bbic?fname=/gpfs/bbp.cscs.ch/project/proj39/data/golgi/Golgi_rat_sections_can/golgi_rat_sections_proj2.h5&mode=ims&prog=TILE 0 0 0 7078 60 3 min 64 1 Get a tile of Golgi stained rat with minimum intensity projection of 64 slices x
GET https://nip.humanbrainproject.eu/api/image-service/image/v0/api/bbic?mode=meta&fname=/gpfs/bbp.cscs.ch/project/proj39/template/human/bigbrain/release2015/bbic/sections/bigbrain.h5 Get meta data about BigBrain image stack Meta data

HBP-PROV Validator

Purpose

This REST service provides the ability to check the correctness of an HBP-PROV file instance.

This verification is two fold:

  • Syntactic: it verifies that the format is correct against the corresponding Json schema, this encompasses the naming and cardinality of entities as well as their attributes.
  • Semantic: we have the ability of implemented custom validation rules that can perform neuroscience oriented verifications as well as other aspects that a JSON schema cannot enforce.

Please note that currently 3 data formats are supported in the Neuroinformatics Platform:

  • HBP-PROV,
  • Excel
  • SAM (Single Activity Metadata).

However, only HBP-PROV and Excel are meant to be used externally, the SAM format is only used internally by the Neuroinformatics Platform for the purpose of the Registration App.

Features

HBP-PROV Semantic rules available

Entity UUID references

This rule verifies that when an entity refers to an other one by its UUID, then such entity should exist in the HBP-PROV file.

Examples:

  • if a Sample was generated by an Activity with uuid 025e58ea-8f40-4136-97b7-63a0e13f74ee, we will check that an Activity exists in the current file with that specific UUID,
  • if an Activity refers to a Contributor by UUID, we check that a contributor exists with that specific UUID.
Ontology usage

This rule verifies that when using an Ontology term to annotate a dataset, the ontology from which the selected term comes from is appropriate.

Examples:

  • When defining the species of a Specimen, the ontology term should originate from the NCBI taxonomy.
  • When defining a brain parcel for a dataset originating from a mouse Specimen, the ontology term should originate from the Allen Mouse Parcellation Scheme.

This section will be updated when new rules are being added.

Validation output data format

{
  "success": true,
  "messages": [
    {
      "type": "SYNTAX|SEMANTIC",
      "severity": "INFO|WARNING|ERROR|FATAL",
      "message": "string",
      "location": "string"
    }
  ]
}

Sample validation output:

Successful validation

{
  "success": true,
  "messages": []
}

Failed syntactic validation (2 errors)

{
  "success": false,
  "messages": [
    {
      "type": "SYNTAX",
      "severity": "ERROR",
      "message": "object has missing required properties ([\"id\"])",
      "location": "{\"pointer\":\"/activities/0/protocol\"}"
    },
    {
      "type": "SYNTAX",
      "severity": "ERROR",
      "message": "object has missing required properties ([\"unit\"])",
      "location": "{\"pointer\":\"/specimen/0/age\"}"
    }
  ]
}

Failed semantic validation (1 error)

{
  "success": true,
  "messages": [
    {
      "type": "SEMANTIC",
      "severity": "ERROR",
      "message": "Ontology term of type 'ACTIVITY_TYPE' are expected to be defined using 'http://www.hbp.FIXME.org/hbp_activity_ontology', instead found term ('null', 'MBA:1063') of type 'http://www.hbp.FIXME.org/hbp_abam_ontology'",
      "location": "Activity: [id: 0e9e371e-ee47-11e4-8c8b-6003088da632]"
    }
  ]
}

Endpoints

HTTP Method Endpoint Usage Returns
POST https://nip.humanbrainproject.eu/api/validator/hbp-prov/v3 The HBP-PROV to validate is given in the body of the request. The given data instance is validated both syntactically and semantically and report summarises the validation status and potential errors. Validation output defined above.
POST https://nip.humanbrainproject.eu/api/validator/single-activity-metadata The SAM to validate is given in the body of the request. The given data instance is validated both syntactically and semantically and report summarises the validation status and potential errors. Validation output defined above.

An interactive version of this endpoint documentation is available here.

How to validate an HBP-PROV file on the command line:

curl -X POST https://nip.humanbrainproject.eu/api/validator/hbp-prov/v3 --header 'Content-Type: application/json' --header 'Accept: application/json' -d '<insert your json content here>'

REST APIs Demos

Several demos have been compiled as iPython Notebooks to illustrate the use of our APIs:

Ontology API

In this demo you will learn to query a SciGraph server and perform the following operations:

  • Search for ontology terms by name
  • Compare ontology terms available in two instances of SciGraph
  • List existing relationships between ontology terms
  • Build and render a hierarchy of ontology terms matching a given name

Search API (KnowledgeGraph)

In this demo you will learn how to run searches in the KnowledgeGraph and interpret the results, it covers the following operations:

  • Search for a dataset by brain region in the Search web application and the same search in the Search REST API (via Swagger using curl),
  • Find the list of available filters through the Search REST API,
  • Find the list of available storage types through the Search REST API,
  • Use the JSON output of a Search to print a user friendly summary of a dataset,
  • Learn how to enable/disable search result aggregations,
  • Learn how to customise a search output to make it return the details that you care about,
  • Download a morphology dataset that was stored in the HBP Document Service
  • Render a 3D view of the downloaded morphology using NeuroM

Voxel Brain API

In this demo you will learn a little about the Multiresolution Atlas Viewer web application as well as how to use the Voxel Brain API. It will take you through the following steps:

  • Browse brain atlas in your browser using the Multiresolution Atlas Viewer web application
  • Select a brain region in the Brain Atlas and browse brain region related datasets in the KnowledgeSpace
  • Programmatically list all atlases available in the Voxel Brain API
  • List all the filters one can use to restrict the atlases returned
  • Search for brain parcels by name in a selected atlas
  • Get the annotation layers available for a selected atlas and brain region
  • Download an annotation layer (nissl staining) and render a 2D visualisation

Glossary

Voxel Brain annotation layer

In the context of the Voxel Brain, an annotation layer is the dataset we attach to the voxels of a data volume.

Here are a few examples:

  1. A brain region annotation layer would store an ontology term describing a given brain region or brain parcel per voxel in the atlas space.
  2. A cell density Annotation layer would store a float value to every voxel of a given atlas space.
  3. A gene expression annotation layer would store in each voxel of the atlas space a list of gene names with associated numerical expression level.

Atlas space

Atlas space refers to three-dimensional coordinate system of a given brain atlas.

Brain atlas

A brain atlas is a spatial representation of the brain that comprises ontology, parcellation, coordinate system, and multiple features/layers of the cerebral characteristics.

Brain parcel

A brain parcel is a specific brain region that is defined deterministically or probabilistically in the context of a given brain atlas.

Contributor

Individual or institution that produced the Data Set. The data will be accessible to the whole community. Therefore, experimental data are not permitted. Moreover, no personal information related to patients or de-anonymized data are accepted.

Data Registration

A unique set of information related to one specific dataset. It includes all piece of information required by the API or service registration pages as well as all piece of information about the Contributor registration, the contributors of the dataset and data user conditions.

Dataset

Digital data, either raw or derived, and its metadata provided through the data access portal interface, or through other media, including data and metadata derived from HBP monitoring protocols, field observations, collections, laboratory analysis, camera trap images, all written, recorded, graphic, audio, visual, and other materials in any media, whether or not subject to copyright protection, or the post-processing of existing data and identified by a unique identifier issued by the HBP.

All datasets provided by contributors should have been produced following EC ethical regulation.

Dataset Contact

Party designated in the accompanying metadata of the dataset as the primary contact for the dataset.

Data User

Individual to whom access to this dataset may be granted, subject to acceptance of these Terms and Conditions by such individual and his or her immediate collaboration sphere, defined here as the institutions, partners, students and staff with whom such individual collaborates and to whom access must be granted in order to fulfill the such individual’s intended use of the dataset.

Human Brain Project

Human Brain Project (HBP) is a European Commission Future and Emerging Technologies Flagship that aims to achieve a multi-level, integrated understanding of brain structure and function through the development and use of information and communication technologies (ICT).

HBP-PROV

HBP-PROV is an exchange data format intended to represent how data was created, which organism was used, how it was processed - including which contributor participated, which software was used - and what dataset/files resulted from it.

KnowledgeGraph

KnowledgeGraph is a metadatabase built on a provenance data model HBP-PROV. In other words, it is a provenance graph to which data are registered for discovery, search, access and tracking. Currently, the KnowledgeGraph consists of Search API which is public and Indexer API which is private.

KnowledgeSpace

KnowledgeSpace (KS) is a community encyclopaedia that links brain research concepts with data, models and literature from around the world. It is an open project and welcomes participation and contributions from members of the global research community.

Ksearch

Ksearch is the search component of the Neuroinformatics Platform. It is a REST API allowing searching curated datasets using different filters that are mainly taken from MINDS.

MeSH

MeSH is the National Library of Medicine’s controlled vocabulary thesaurus used for indexing articles for PubMed. It consists of sets of terms naming descriptors in a hierarchical structure that permits searching at various levels of specificity.

MINDS

MINDS stands for Minimum Information for Neuroscience DataSets. Data shared on the Neuroinformatic Platform are enriched with minimal metadata to provide essential information in order to ensure traceability of any data artefact.

Neuroinformatics Platform

Neuroinformatics Platform (NIP) is Sub-Project 5 of the Human Brain Project. The Neuroinformatics Platform and the Brain Atlases allow neuroscientists to collaboratively curate, analyse, share, and publish heterogeneous neuroscience data.

Parcellation

Brain parcellations define spatial or volumetric boundaries of brain region/structure. They could be represented in 2D and 3D depending on the use case.

Registration

For 2D or 3D volumetric datasets, we use different registration methods to convert one dataset from its own space to another space. Following registration methods are used:

  • Linear registration is used to capture the global transformation between the subject image and the atlas image.
  • Landmark registration module is used in the 3D Slicer to pick corresponding landmark in both subject image and atlas image.
  • Deformable registration is a process consisting of establishing functional or spatial correspondences between two images.

Note: There are “Data registration” and “Registration App”, which are separate terms.

Voxel

The etymology of the work comes from a blend of ‘volumetric’ and ‘pixel’. A voxel is the three-dimensional analogue of a pixel in a two-dimensional space.

Voxel Brain

Voxel brain is a REST service provides access to volumetric brain atlas and their annotation layers in the form of voxels.