Strategies for sharing NWB:N extensions (NDX)

• Version: 0.3.1 (DRAFT)
• Authors:
  • Oliver Ruebel
  • Andrew Tritt
  • Benjamin Dichter
  • Ryan Ly
• Last update: October 18, 2019

1. Overview

The purpose of this document is to define standard practices and strategies for sharing format extensions for NWB, so called Neurodata Extensions (NDX).

1.1 Definitions

• The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119.
• “Neurodata Extensions” (NDX) refers to extensions to the NWB data standard. An NDX MUST be described by a formal format specification using the NWB specification language.
• “Internal NDX” refers to extensions used within a particular lab, organization, or project that are not intended for use outside of that group or context.
• “Public NDX” refers to extensions intended for use by the public (or specific larger community).
• “NDX Catalog” refers to the public catalog for NDX. The nwb-extensions catalog is implemented via a dedicated public GitHub organization at https://github.com/nwb-extensions and website https://nwb-extensions.github.io/.
• “ndx-template” refers to the repository containing a template for creating an extension from scratch.
• “ndx-custom” refers to a specific user extension repository that has usually been created from the ndx-template and is used to manage the sources of the extension. This repository lives in the user’s own Git space and is NOT part of the nwb-extensions organization.
• “staged-extensions” repository refers to the GitHub repository that holds extension records awaiting review for registration of the extension with the catalog. It also contains a template for creating an extension record.
• “ndx-custom-record” refers to a specific record repository for deploying a particular extension as part of the NDX Catalog. This repo is part of the “nwb-extensions” GitHub organization. This repo is automatically generated by the CI of the staged-extensions repo and includes basic metadata about the NDX and information needed for install and deployment.
• “user/ndx-custom-record” refers to the user’s fork of the ndx-custom-record repo. This repo lives in the users Git space (NOT in the nwb-extensions GitHub organization).

1.2 Summary

NWB defines a set of Versioning Guidelines for versioning extension namespaces and Sharing Guidelines that public NDX MUST follow and internal NDXs are highly RECOMMENDED to follow. In order to help ensure compliance with these rules, NWB provides a number of tools, templates, and guidelines.

2. Methods and tools for sharing extensions

2.1 nwb-extensions: Extensions GitHub Organization

NDXs are shared via a dedicated nwb-extensions GitHub organization for the NDX Catalog. The NDX Catalog provides the ndx-template and staged-extensions extension template repositories (see below) as well as all public ndx-custom-record repositories. In this way, all public extensions are managed in one place and MUST follow the same basic internal organization. The public ndx-custom-record repositories contain only metadata about the extensions, such as installation instructions and URLs to the extension sources. This strategy allows for labs, universities, and independent groups to maintain the source code for their extensions in their own Git space. The ability to do so can be important in particular when the developers of the extension are funded by their own grants and/or are applying for funding.

2.1.1 ndx-template: Extension template

The ndx-template provides a standard Git template repository with a standard setup for creating and sharing extensions and instructions for how to use the template. Using Cookiecutter, users can easily modify the template to create the basic setup for their ndx-template repository. See the ndx-template repository for details on the organization of the template.

2.1.2 staged-extensions: Creating custom ndx records

The staged-extensions repository is a place to submit new NDX records for registration with the NDX Catalog, i.e., before a ndx-custom-record repo exists for the extension. The repository holds an example record which should be copied and modified for a new extension to be registered. See the staged-extensions repository for details on how to use the repo for registering extensions with the NDX Catalog.

2.2.3 ndx-custom-record: Custom extension catalog record

The NDX Catalog contains a collection of ndx-custom-record repositories created by the creators of the extensions. See below for further details.

2.3 Custom Extension Code Repositories

2.3.1 ndx-custom-record: Custom extension catalog record

The ndx-custom-record is created from the staged-extensions repository as part of the NDX catalog and MUST have the same name as the main namespace of the NDX. This strategy helps ensure that there are no name conflicts between public NDX namespace names. The ndx-custom-record repository MUST contain the following:

• / : root directory of the repo
  • /ndx-meta.yaml: REQUIRED YAML file with the metadata describing the extensions. The YAML file MUST contain a dict with the following keys:
    • name: name of the namespace
    • version: Semantic version number of the extension. See the Versioning Guidelines for details.
    • **src: **: URL to the public repository with the sources of the extension
    • **pip: **: URL for installing the extensions from PyPI
    • license: License type
    • maintainers: List of maintainers (GitHub usernames) for the extension

  • **/README.(md

    rst)** : REQUIRED readme with basic description of the extension, badges for CI, install instructions, etc.

  • /.gitignore: Containing standard ignore options to ignore sphinx build files, autogenerated docs files from the nwb-docutils and sphinx, python and Matlab temporary files, etc.

In addition the ndx-custom-record may contain the following OPTIONAL documents related to the review of the extensions:

• /review: OPTIONAL folder with GitHub Markdown (or RST documents) related to the review of the extension for acceptance as part of the NWB core schema.
  • If present the /review folder MUST contain:
    • /review/README.md : Readme file with instructions on how to build and use the review docs and for linking all the documents together.
    • /review/RELEASE_NOTES.md : Markdown file with the release notes for the review docs. This file MUST be linked to from README.md.
    • /review/REVIEW.md : Main document with the review discussion for the NDX. This file MUST be linked to from README.md.
  • In addition the folder MAY contain the following SUGGESTED documentation files:
    • /review/CREDITS.md : Markdown file with the credits for the extension review. If present, the file MUST be linked to from the README.md.
    • /review/LEGAL.md : Markdown file with the legal details about the extension review (e.g., license and copyright). If present, the file MUST be linked to from the README.md.

2.3.2 ndx-custom: Custom NWB Extensions

The ndx-custom repository contains the sources of the user extension that has usually been created from the ndx-template. This repository lives in the user’s own Git space and is NOT part of the nwb-extensions organization. This provides users the ability to maintain ownership of the extension.

2.4 Additional Tools and Methods

2.4.1 Documenting extensions

The nwb-docutils provide tools for generating Sphinx RST documentation directly from the YAML sources. The “generate_format_docs.py” command-line tools MUST be used to auto-generate documentation from the YAML sources.

As part of the ndx-template Git repository, NWB provides a standard setup for generating Sphinx documentation for an extension based on the nwb-docutils. Extensions MAY customize the documentation but MUST follow the standard for building and generating the extensions.

2.4.2 Creating extension specifications

Extensions SHOULD be created using the PyNWB specification API to improve maintainability of the extensions and to insulate the creators of the extensions from low-level details of the NWB specification language (see the PyNWB docs and tutorial for details). The sources for creating the extensions SHOULD be shared along with the extension YAML specification files to facilitate maintenance and compliance with the NWB specification language. Extensions MAY also be created by just sharing YAML files (e.g., written by hand) that are compliant with the NWB specification language, however, this is NOT RECOMMENDED. The ndx-template includes dedicated locations for the YAML specification and sources, and includes a template Python file for implementing specifications using PyNWB.

3. Extension process (create, deploy, update)

The figure below illustrates the process for (1) creating a new extensions, (2) creating a record to register an extension with the extension catalog, and (3) to update an extension/record. The figure also illustrates the automated CI processes that are managed in the nwb-extensions catalog. The catalog process is modeled after the conda-forge model, enabling us to largely automate the catalog process using free, public services and avoid the need to host our own services.

3.1 Continuous Integration (CI) for Extensions

This can be separated into CI for three main repositories:

1. the staged-extensions repository as part of the NDX Catalog,
2. the ndx-custom-record repo as part of the NDX Catalog, and
3. the user’s ndx-custom repository.

Parts 1 and 2 are part of the NDX Catalog and are maintained by the NWB development team. Part 3 will need to be setup and maintained by the extension developers. The ndx-template can provide templates for defining CI for ndx-custom repositories, but ultimately this will be up to the extension developers to do.

3.1.1 staged-extensions CI

For the staged-extensions repositories, we have common tests to validate basic compliance and working order of the extensions before creating the ndx-custom-record repo. Here we can test, e.g., the extensions metadata, i.e., are all required fields for the catalog specified and valid.

3.1.2 ndx-custom-record CI

Similar to the staged-extensions, we can provide CI for ndx-custom-record repositories to automatically test compliance and working order of the extensions before merging pull requests. Here we can test, e.g., the extensions metadata, i.e., are all required fields for the catalog specified and valid.

3.1.3 ndx-custom CI

Here we provide example configuration files and instructions (e.g., as part of the ndx-template repository), but ultimately this is up to the user to setup and maintain. The ndx-template repo itself will likely not include CI itself. However, it may include templates for setting up CI for ndx-custom user extension repositories. See the ndx-template repo for details. We generally recommend to use CI to:

• Run unit tests (on all platforms) for API classes for the extension (and measure test coverage)
• Flake8 to ensure compliance with coding guidelines
• Test that YAML specifications are compliant with the specification language
• Test that sphinx documents (and galleries) can be built