Data Types¶

The Runway Model SDK provides several data types that can be used to pass values to and from runway models and the applications that control them. The data types currently supported by the SDK are number, text, image, array, vector, category, file, and any, an extensible data type. These data types are primarily used in two places:

The options parameter in the @runway.setup() decorator
The input and output parameters in @runway.command() decorator

Note

This is example code for demonstration purposes only. It will not run, as the your_code import is not a real python module.

import runway
from runway.data_types import category, vector, image
from your_code import model

options = {"network_size": category(choices=[64, 128, 256, 512], default=256)}
@runway.setup(options=options)
def setup(opts):
    return model(network_size=opts["network_size"])


sample_inputs= {
    "z": vector(length=512),
    "category": category(choices=["day", "night"])
}

sample_outputs = {
    "image": image(width=1024, height=1024)
}

@runway.command("sample", inputs=sample_inputs, outputs=sample_outputs)
def sample(model, inputs):
    img = model.sample(z=inputs["z"], category=inputs["category"])
    # `img` can be a PIL or numpy image. It will be encoded as a base64 URI
    # string automatically by @runway.command().
    return { "image": img }

if __name__ == "__main__":
    runway.run()

Reference¶

class runway.data_types.BaseType(data_type, description=None)¶

An abstract class that defines a base data type interface. This type should be used as the base class of new data types, never directly.

Parameters

data_type (string) – The data type represented as a string
description (string, optional) – A description of this variable and how its used in the model, defaults to None

class runway.data_types.any(description=None)¶

A generic data type. The value this data type takes must be serializable to JSON.

import yaml
import runway
from runway.data_types import any
from your_code import model

# an example of passing your own yaml configuration using an "any" data_type and PyYAML
@runway.setup(options={ "configuration": any() })
def setup(opts):
    # parse the configuration string as yaml, and then pass the resulting
    # object as the configuration to your model
    config = yaml.load(opts["configuration"])
    return model(config)

Parameters: description (string, optional) – A description of this variable and how its used in the model, defaults to None

class runway.data_types.array(item_type=None, description=None, min_length=0, max_length=None)¶

A data type representing an array (list) of other runway.data_type objects.

import runway
from runway.data_types import array, text

@runway.setup(options={ "seed_sentences": array(item_type=text, min_length=5) })
def setup(opts):
    for i in range(5):
        print("Sentence {} is "{}"".format(i+1, opts["seed_sentences"][i]))

Parameters

item_type (runway.data_type) – A runway.data_type class, or an instance of a runway.data_type class
description (string, optional) – A description of this variable and how its used in the model, defaults to None
min_length (int, optional) – The minimum number of elements required to be in the array, defaults to 0
max_length (int, optional) – The maximum number of elements allowed to be in the array, defaults to None

Raises

MissingArgumentError – A missing argument error if item_type is not specified

class runway.data_types.boolean(description=None, default=False)¶

A basic boolean data type. The only accepted values for this data type are True and False.

import runway
from runway.data_types import boolean

@runway.setup(options={ "crop": boolean(default=True) })
def setup(opts):
    if opts["crop"]:
        print("The user has chosen to crop the image.")
    else:
        print("The user has chosen not to crop the image.")

Parameters

description (string, optional) – A description of this variable and how it’s used in the model, defaults to None
default (bool, optional) – A default value for this boolean variable, defaults to False

class runway.data_types.category(description=None, choices=None, default=None)¶

A categorical data type that allows you to specify a variable’s value as a member of a set list of choices.

import runway
from runway.data_types import category

# if no default value is specified, the first element in the choices
# list will be used
cat = category(choices=["rgb", "bgr", "rgba", "bgra"], default="rgba")
@runway.setup(options={ "pixel_order": cat })
def setup(opts):
    print("The selected pixel order is {}".format(opts["pixel_order"]))

Parameters

description (string, optional) – A description of this variable and how its used in the model, defaults to None
choices (A list of strings) – A list of categories, defaults to None
default (A list of strings) – A default list of categories, defaults to None

Raises

MissingArgumentError – A missing argument error if choices is not a list with at least one element.
InvalidArgumentError – An invalid argument error if a default argument is specified and that argument does not appear in the choices list.

class runway.data_types.directory(description=None, default=None)¶

A data type that represents a file directory. It can be a local path on disk or a remote tarball loaded over HTTP. .. code-block:: python

import runway from runway.data_types import directory

@runway.setup(options={“checkpoint_dir”: directory}) def setup(opts):

model = initialize_model_from_checkpoint_folder(args[“checkpoint_dir”]) return model

Parameters: description (string, optional) – A description of this variable and how its used in the model, defaults to None

class runway.data_types.file(description=None, is_directory=False, extension=None, default=None)¶

A data type that represents a file or directory. The file can be a local resource on disk or a remote resource loaded over HTTP. Instantiate this class to create a new runway model variable.

import runway
from runway.data_types import file

@runway.setup(options={"checkpoint": file(extension=".h5"))
def setup(opts):
    model = initialize_model_from_checkpoint(args["checkpoint"])
    return model

Parameters

description (string, optional) – A description of this variable and how its used in the model, defaults to None
is_directory (bool, optional) – Does this variable represent a directory instead of a file? Defaults to False.
extension (string, optional) – Accept only files of this extension.
default (string, optional) – Use this path if no input file or directory is provided.

class runway.data_types.image(description=None, channels=3, min_width=None, min_height=None, max_width=None, max_height=None, width=None, height=None, default_output_format=None)¶

A data type representing an image. Images represent PIL or numpy images but are passed to and from the Model SDK as base64 encoded data URI strings over the network (e.g. data:image/jpeg;base64,/9j/2wCEAAgGBgcG...).

When using an image as an output data type for a function wrapped by @runway.command(), return a PIL or numpy image from your wrapped function and it will automatically be serialized as a base64 encoded data URI.

import runway
from runway.data_types import image

inputs = {"image": image(width=512, height=512)}
outputs = {"image": image(width=512, height=512)}
@runway.command("style_transfer", inputs=inputs, outputs=outputs)
def style_transfer(result_of_setup, args):
    # perform some transformation to the image, and then return it as a
    # PIL image or numpy image
    img = do_style_transfer(args["image"])
    # The PIL or numpy image will be automatically converted to a base64
    # encoded data URI string by the @runway.command() decorator.
    return { "image": img }

Parameters

description (string, optional) – A description of this variable and how its used in the model, defaults to None
channels (int, optional) – The number of channels in the image, defaults to 3. E.g. an “rgb” image has 3 channels while an “rgba” image has 4.
min_width (int, optional) – The minimum width of the image, defaults to None
min_height (int, optional) – The minimum height of the image, defaults to None
max_width (int, optional) – The maximum width of the image, defaults to None
max_height (int, optional) – The maximum height of the image, defaults to None
width (int, optional) – The width of the image, defaults to None.
height (int, optional) – The height of the image, defaults to None

class runway.data_types.image_bounding_box(description=None)¶

An bounding box data type, representing a rectangular region in an image. It accepts four normalized floating point numbers [xmin, ymin, xmax, ymax] between 0 and 1, where [xmin, xmax] is the top-left corner of the rectangle and [xmax, ymax] is the bottom-right corner.

import runway
from runway.data_types import image, point

@runway.command('detect_face', inputs={'image': image()}, outputs={'face_bbox': image_bounding_box})
def detect_faze(model, inputs):
    result = model.run(inputs['image'])
    return {'face_bbox': result}

Parameters: description (string, optional) – A description of this variable and how it’s used in the model, defaults to None

class runway.data_types.image_landmarks(length, description=None, labels=None, connections=None)¶

An image landmarks data type, representing a fixed-length array of (x, y) coordinates, such as facial landmarks. Each (x, y) coordinate pair in the array should be expressed as normalized image points with values between 0 and 1, inclusive.

import runway
from runway.data_types import image, image_landmarks

landmark_names = [
    'nose',
    'leftEye',
    'rightEye',
    'leftEar',
    'rightEar',
    'leftShoulder',
    'rightShoulder',
    'leftElbow',
    'rightElbow',
    'leftWrist',
    'rightWrist',
    'leftHip',
    'rightHip',
    'leftKnee',
    'rightKnee',
    'leftAnkle',
    'rightAnkle'
]

landmark_connections = [
    ['leftHip', 'leftShoulder'],
    ['leftElbow', 'leftShoulder'],
    ['leftElbow', 'leftWrist'],
    ['leftHip', 'leftKnee'],
    ['leftKnee', 'leftAnkle'],
    ['rightHip', 'rightShoulder'],
    ['rightElbow', 'rightShoulder'],
    ['rightElbow', 'rightWrist'],
    ['rightHip', 'rightKnee'],
    ['rightKnee', 'rightAnkle'],
    ['leftShoulder', 'rightShoulder'],
    ['leftHip', 'rightHip']
]

command_outputs = {
    'keypoints': image_landmarks(17, labels=landmark_names, connections=landmark_connections)
}

@runway.command('detect_pose', inputs={'image': image()}, outputs=command_outputs)
def detect_pose(model, inputs):
    pose = model.run(inputs['image'])
    return {'keypoints': pose}

Parameters

length (int) – The number of landmarks associated with this type.
labels (list, optional) – Labels associated with each landmark.
connections (list, optional) – A list of pairs of logically connected landmarks identified by name, e.g. [[‘left_hip’, ‘left_shoulder], [‘right_hip’, ‘right_shoulder’]]. The names included in connections should correspond to the names provided by the labels property. This property is only used for visualization purposes.
description (string, optional) – A description of this variable and how it’s used in the model, defaults to None

class runway.data_types.image_point(description=None)¶

A point data type representing a specific location in an image. It accepts two normalized floating point numbers [x, y] between 0 and 1, where [0, 0] represents the top-left corner and [1, 1] the bottom-right corner of an image.

import runway
from runway.data_types import image, point

@runway.command('detect_gaze', inputs={'image': image()}, outputs={'gaze_location': image_point()})
def detect_gaze(model, inputs):
    result = model.run(inputs['image'])
    return {'gaze_location': result}

Parameters: description (string, optional) – A description of this variable and how it’s used in the model, defaults to None

class runway.data_types.number(description=None, default=None, step=None, min=None, max=None)¶

A basic number data type. Instantiate this class to create a new runway model variable.

import runway
from runway.data_types import number

@runway.setup(options={ "number_of_samples": number })
def setup(opts):
    print("The number of samples is {}".format(opts["number_of_samples"]))

Parameters

description (string, optional) – A description of this variable and how its used in the model, defaults to None
default (float, optional) – A default value for this number variable, defaults to 0
min (float, optional) – The minimum allowed value of this number type
max (float, optional) – The maximum allowed value of this number type
step (float, optional) – The step size of this number type. This argument define the minimum change of value associated with this number type. E.g., a step size of 0.1 would allow this data type to take on the values [0.0, 0.1, 0.2, ..., 1.0].

class runway.data_types.segmentation(label_to_id=None, description=None, label_to_color=None, default_label=None, min_width=None, min_height=None, max_width=None, max_height=None, width=None, height=None)¶

A datatype that represents a pixel-level segmentation of an image. Each pixel is annotated with a label id from 0-255, each corresponding to a different object class.

When used as an input data type, segmentation accepts a 1-channel base64-encoded PNG image, where each pixel takes the value of one of the ids defined in label_to_id, or a 3-channel base64-encoded PNG colormap image, where each pixel takes the value of one of the colors defined in label_to_color.

When used as an output data type, it serializes as a 3-channel base64-encoded PNG image, where each pixel takes the value of one of the colors defined in label_to_color.

import runway
from runway.data_types import segmentation, image

inputs = {"segmentation_map": segmentation(label_to_id={"background": 0, "person": 1})}
outputs = {"image": image()}
@runway.command("synthesize_pose", inputs=inputs, outputs=outputs)
def synthesize_human_pose(model, args):
    result = model.convert(args["segmentation_map"])
    return { "image": result }

Parameters

description (string, optional) – A description of this variable and how its used in the model, defaults to None
label_to_id (dict) – A mapping from labels to pixel values from 0-255 corresponding to those labels
default_label (string, optional) – The default label to use when a pixel value not in label_to_id is encountered
label_to_color (dict, optional) – A mapping from label names to colors to represent those labels
min_width (int, optional) – The minimum width of the segmentation image, defaults to None
min_height (int, optional) – The minimum height of the segmentation image, defaults to None
max_width (int, optional) – The maximum width of the segmentation image, defaults to None
max_height (int, optional) – The maximum height of the segmentation image, defaults to None
width (int, optional) – The width of the segmentation image, defaults to None.
height (int, optional) – The height of the segmentation image, defaults to None

class runway.data_types.text(description=None, default='', min_length=0, max_length=None)¶

A basic text data type. Used to represent strings. Instantiate this class to create a new runway model variable.

import runway
from runway.data_types import text

@runway.setup(options={ "flavor": text(default="vanilla") })
def setup(opts):
    print("The selected flavor is {}".format(opts["flavor"]))

Parameters

description (string, optional) – A description of this variable and how its used in the model, defaults to None
default (string, optional) – The default value for this text variable, defaults to ‘’
min_length (int, optional) – The minimum character length of this text variable, defaults to 0
max_length (int, optional) – The maximum character length of this text variable, defaults to None, which allows text to be of any maximum length

class runway.data_types.vector(length=None, description=None, default=None, sampling_mean=0, sampling_std=1)¶

A data type representing a vector of floats.

import runway
from runway.data_types import vector, number
import numpy as np

inputs={"length": number(min=1)}
outputs={"vector": vector(length=512)}
@runway.command("random_sample", inputs=inputs, outputs=outputs)
def random_sample(result_of_setup, args):
    vec = vector(length=args["length"])
    rand = np.random.random_sample(args["length"])
    return { "vector": vec.deserialize(rand) }

Parameters

description (string, optional) – A description of this variable and how its used in the model, defaults to None
length (int, inferred if a default vector is specified) – The number of elements in the vector
sampling_mean (float, optional) – The mean of the sample the vector is drawn from, defaults to 0
sampling_std (float, optional) – The standard deviation of the sample the vector is drawn from, defaults to 1

Raises

MissingArgumentError – A missing argument error if length is not specified