Exports and imports

The internal format for storing metadata in NRP is JSON with a structure defined in YAML models. However, users often need to export or import metadata in various standard formats for interoperability with other systems or data exchange.

This guide explains how to add custom export and import formats for metadata schemas. We assume that you have already created a custom metadata schema as described in the model customization guide.

DataCite export

The DataCite export has been pre-generated for your model in the equipment/serializers.py file. If your model is not based on the CCMM or RDM full template, you need to customize the DataCite export to include all required fields. For inspiration, check the invenio-rdm-records package.

Adding custom export formats

Export formats are registered within the equipment/model.py file inside the “customization” section:


my_model = model(
    # ...
    customizations=[
        AddMetadataExport(
            code="datacite",
            name=_("DataCite export"),
            mimetype="application/vnd.datacite.datacite+json",
            serializer=DataCiteJSONSerializer(),
        ),
    ],
)

To add a new export format for your metadata schema, you need to create a serializer class that converts the internal JSON representation to the desired export format. Then register the class as shown in the example above.

AddMetadataExport accepts the following arguments:

Argument	Description
`code`	Code of the export format, used to identify the format (e.g., in URLs)
`name`	Human-readable name of the format, shown in the UI
`mimetype`	MIME type of the format. The REST API returns this format when a client sends it in the `Accept` header
`serializer`	Serializer instance that converts the record to the format
`display`	Whether the format is offered in the UI (default `True`). Set to `False` for formats that should only be available through the API
`oai_metadata_prefix`, `oai_schema`, `oai_namespace`	OAI-PMH metadata prefix, schema URL and namespace. The format can be served over OAI-PMH only if all three are set
`about_serializer`	Optional serializer for the `about` section of OAI-PMH records
`description`	Optional human-readable description of the format
`extension`	File extension used when the export is downloaded (e.g., `.json`). Guessed from `mimetype` if not set

For example, to make the export available over OAI-PMH:


AddMetadataExport(
    code="oai_dc",
    name=_("Dublin Core"),
    mimetype="application/x-dc+xml",
    serializer=MyDublinCoreSerializer(),  # your serializer producing oai_dc XML
    oai_metadata_prefix="oai_dc",
    oai_schema="http://www.openarchives.org/OAI/2.0/oai_dc.xsd",
    oai_namespace="http://www.openarchives.org/OAI/2.0/oai_dc/",
)

To download an export from the REST API, request the record with the export’s MIME type:


curl -H "Accept: application/vnd.datacite.datacite+json" https://<repository>/api/<model>/<record id>

Creating a serializer class

A serializers.py file has already been created in your model directory. You can add your custom serializer classes there, or create separate files for each serializer if you prefer.

Serializing to JSON

If the export format is JSON, you can use the existing flask_resources.MarshmallowSerializer as the base class. Example:


from flask_resources import BaseListSchema, MarshmallowSerializer
from flask_resources.serializers import BaseSerializerSchema, JSONSerializer
from marshmallow import fields
 
 
class MyJSONSerializer(MarshmallowSerializer):
    """Marshmallow-based serializer for records."""
 
    def __init__(self, **options):
        """Constructor."""
        super().__init__(
            format_serializer_cls=JSONSerializer,  # the resulting format is JSON
            object_schema_cls=MySchema,  # schema for single object
            list_schema_cls=BaseListSchema,  # schema for list of objects
            schema_kwargs={},
            **options,
        )
 
class MySchema(BaseSerializerSchema):
    """Schema for serializing records to custom JSON format."""
 
    # Define the fields to include in the export
    title = fields.String(attribute="metadata.title")
    serial_number = fields.String(attribute="metadata.serial_number")
    manufacturer = fields.String(attribute="metadata.manufacturer")

The MarshmallowSerializer is responsible for calling MySchema on each serialized record. The MySchema class is where you define the actual fields and their serialization logic. Refer to the Marshmallow documentation for more details on defining schemas and fields.

Serializing to other formats

To serialize to other formats, you have two options:

Inherit from MarshmallowSerializer and provide a different format serializer class (for example, XMLSerializer) - see flask-resources for more details.
Inherit directly from BaseSerializer and implement the serialize_object and serialize_object_list methods.

Here we’ll use the second option and create a CSV serializer from scratch. Note that this is a synthetic example to illustrate how to create a custom serializer. For CSV export in real scenarios, you’d want to use a combination of MarshmallowSerializer with a format serializer that handles CSV.


import csv
from flask_resources.serializers.base import BaseSerializer
 
class CSVSerializer(BaseSerializer):
    """Custom serializer for exporting records to CSV format."""
    
    header = ['name', 'serial_number', 'manufacturer']
 
    def serialize_object(self, obj: dict):
        """Serialize a single object to CSV format."""
        return self._create_csv(
            [self.header, self._serialize_object(obj)]
        )
 
    def serialize_object_list(self, obj_list: list):
        """Serialize a list of objects to CSV format."""
        return self._create_csv(
            [self.header] +
            [self._serialize_object(obj) for obj in obj_list]
        )
 
    def _create_csv(self, rows):
        """Create CSV string from rows."""
        from io import StringIO
        output = StringIO()
        writer = csv.writer(output)
        writer.writerows(rows)
        return output.getvalue()
 
    def _serialize_object(self, obj):
        """Serialize a single object to a CSV row."""
        metadata = obj.get('metadata', {})
        return [
            metadata.get('name', ''),
            metadata.get('serial_number', ''),
            metadata.get('manufacturer', ''),
        ]

Registering the custom serializer

Once you’ve created your custom serializer, register it in your model’s customization section:


my_model = model(
    # ...
    customizations=[
        AddMetadataExport(
            code="csv",
            name=_("CSV export"),
            mimetype="text/csv",
            serializer=CSVSerializer(),
        ),
    ],
)

Adding custom import formats

Import formats let clients create or update records by sending metadata in a format other than the internal JSON. The REST API picks the import format by the Content-Type header of the request: the request body is passed to the format’s deserializer, and the result is then processed as if it had been sent as JSON.

A JSON import (application/json) is always registered, so the internal JSON format works without any configuration.

Creating a deserializer class

A deserializer inherits from flask_resources.deserializers.DeserializerMixin and implements deserialize(data). It receives the raw request body (bytes) and returns a dictionary in the internal JSON format of the record, i.e. the same structure a client would send as JSON:

equipment/deserializers.py


import csv
from io import StringIO
 
from flask_resources.deserializers import DeserializerMixin
 
 
class CSVDeserializer(DeserializerMixin):
    """Create a record from a CSV file with a header and a single data row."""
 
    def deserialize(self, data: bytes) -> dict:
        reader = csv.DictReader(StringIO(data.decode("utf-8")))
        row = next(reader, {})
        return {
            "metadata": {
                "name": row.get("name"),
                "serial_number": row.get("serial_number"),
                "manufacturer": row.get("manufacturer"),
            },
        }

The returned dictionary is validated by the model’s schema in the same way as JSON input, so the deserializer does not need to validate the values itself.

Registering the custom deserializer

equipment/model.py


from invenio_i18n import lazy_gettext as _
from oarepo_model.customizations import AddMetadataImport
 
from .deserializers import CSVDeserializer
 
my_model = model(
    # ...
    customizations=[
        AddMetadataImport(
            code="csv",
            name=_("CSV"),
            mimetype="text/csv",
            deserializer=CSVDeserializer(),
            description=_("A CSV file with a header and a single data row"),
        ),
    ],
)

Argument	Description
`code`	Code of the import format
`name`	Human-readable name of the format
`mimetype`	MIME type of the format; the format is used for requests with this `Content-Type`
`deserializer`	Deserializer instance that converts the request body to the internal JSON
`description`	Human-readable description of the format (required)
`oai_name`	Optional `(namespace, local name)` of the metadata element in OAI-PMH XML records; the format can be used for OAI-PMH harvesting only if set

Any other keyword arguments are accepted but ignored.

A client can then create a record by sending the data with the matching content type:


curl -X POST -H "Content-Type: text/csv" --data-binary @equipment.csv \
     https://<repository>/api/<model>/