Skip to content

Open Model Interface Spec

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Changelog

Date Version Notes
2021-07-26 1.0 Initial version of OMI Spec
2022-03-24 1.1 Updated OMI Spec to include new requirements

Requirements

To qualify as an Open Model Interface (OMI) compliant container, a container MUST pass the following tests.

OCI compliant

The container image MUST be an OCI compliant container image, for example it MAY be generated by docker build or Kaniko.

Origin tags

The container MUST be tagged (e.g. tag in registry/org/image:tag) with a unique ID which corresponds to the ID of the model in the upstream model management system. For example, the container image tag MAY be set to the MLflow Run ID which generated the model.

This is to enable provenance tracking from a given model which generated certain inference results to metadata about how the model was trained. The metadata referred to by the model tag SHOULD record the versions of the code, data and environment that were used to train the model.

Declares Openmodel Annotations

The container MUST have OCI annotations of the following form:

ml.openmodel.interfaces=["kfserving", "modzy"]
ml.openmodel.protocols=[["v1","v2"],["v2"]]
ml.openmodel.port="8080"
ml.openml.model_name="modelname"

Where ml.openmodel.protocol is list of lists such that the inner list's index corresponds to its interface index e.g. a model with the above configuration would support kfserving v1 and v2 protocols as well as the modzy interface with v2 protocol.

This is so that the serving system at runtime can determine whether it supports a given model without trying to run it so that it can give the user fast feedback.

Ports

When started, the container image MUST listen on the TCP port given as the ml.openmodel.port annotation. When a container starts listening on the prescribed port, it MUST adhere to the interface described below corresponding to its configuration.

Environment variables

OCI annotations are not available in a running Docker container. As a result, an OMI container MUST be configurable with the following environment variables to provide a copy of the values associated with the OMI labels: 

env:
  - name: INTERFACE
    value: <ml.openmodel.interfaces> # e.g. kfserving
  - name: HTTP_PORT
    value: <ml.openmodel.port> # e.g. 8080
  - name: PROTOCOL
    value: <ml.openmodel.protocols> # e.g. v2
  - name: MODEL_NAME
    value: <ml.openml.model_name> # e.g. mdodel name

  • INTERFACE MUST be the string interface supported, as defined in the interfaces section below.
  • PROTOCOL MAY be the optional sub-protocol of the given interface
  • MODEL_NAME MUST be some human-readable name of the model that is to be run inside the container.
  • HTTP_PORT MUST be the TCP port that the container should listen on when started

If a conflict arises between the OCI annotations and the environment variables, The OMI specification considers OCI annotations to be authoritative.

Interfaces and Protocols

The following interfaces are permitted in the spec:

  • kfserving: Supports
  • v1 REST API with ml.openmodel.interfaces=["kfserving"] and ml.openmodel.protocols="v1"
  • v2 gRPC API with ml.openmodel.interfaces=["kfserving"] and ml.openmodel.protocols="v2"
  • modzy: Supports
  • v1 REST API with ml.openmodel.interfaces=["modzy"] and ml.openmodel.protocols="v1"
  • v2 gRPC API with ml.openmodel.interfaces=["modzy"] and ml.openmodel.protocols="v2"

Compliant Implementations

OMI compliant container images MUST: * implement at least one of the kfserving or modzy API interfaces * be runnable and responsive on both KServe and Modzy platforms * be runnable independently in isolation

For example:

The Chassis reference implementation of the OMI specification produces containers * that implement the modzy.v2 API * that implement a KServe compatible architecture that follows the kfserving.v1 specification with one exception. Chassis-generated model containers expect binary input data instead of the json input data the kfserving.v1 spec designates. Therefore, for KFServing users must base64-encode their input bytes to be included within their json input submission. In this way, the model containers generated by Chassis will run on KFServing as well as Modzy. * that can run in isolation

Updates to this spec

If an OMI image declares or supports any strings other than the ones defined above, this document MUST be updated with the new interfaces (with links to their upstream protocol specifications), otherwise the container MUST NOT be described as Open Model Interface compliant.

This document can be updated via Pull Request on GitHub.

Any proposed PRs must be accompanied by a public comment period of no less than 2 weeks (14 days) before a merge can be approved.

Contributions welcome!