Record and Share Experiments#

In practical mathematical optimization, a workflow rarely ends by simply building a mathematical model and sending it to a solver. In many cases, you compare multiple formulations, relax some constraints, or try easier subproblems. In addition to managing modeled problems and solver results through adapters, OMMX provides APIs for recording these trial-and-error processes as experiments, then saving and sharing them.

experiment is the API for storing such experiment units as OMMX Artifacts.

Concept	Role
`Experiment`	The whole experiment. It can have experiment-level Attachments and multiple Runs. It is the sharing unit, and it always has a container-style name.
`Run`	One trial within an experiment and the comparison unit. Since complex workflows often call solvers more than once, a Run can contain multiple solver calls (Solves). A Run can also have scalar parameters used as comparison axes, making it easy to compare Runs across the Experiment.
`Solve`	One solver call within a Run. It stores the input `Instance` and output `Solution`. It also records which Adapter was used and which options were passed to the solver call.
Attachment	An arbitrary payload attached to an Experiment or Run. It can store data types such as JSON, `numpy.ndarray`, `Instance`, and `Solution`, as well as arbitrary bytes with an explicit Media Type.

In this tutorial, we solve a simple knapsack problem twice under different conditions, then save and read the execution history as one Experiment.

Prepare the Mathematical Model#

First, create the source data for a knapsack problem and a ParametricInstance whose capacity is a parameter. Like Instance, an OMMX ParametricInstance can define an objective function and constraints, but it can place parameters where constants would otherwise appear. This is useful when you need to prepare multiple models that differ only in constants.

from ommx.v1 import DecisionVariable, Parameter, Instance, ParametricInstance

v = [10, 13, 18, 31, 7, 15]  # value of each item
w = [11, 25, 20, 35, 10, 33]  # weight of each item
N = len(v)

x = [
    DecisionVariable.binary(
        id=i,
        name="x",
        subscripts=[i],
    )
    for i in range(N)
]

capacity = Parameter(N, name="capacity")

pi = ParametricInstance.from_components(
    decision_variables=x,
    parameters=[capacity],
    objective=sum(v[i] * x[i] for i in range(N)),
    constraints={
        0: (sum(w[i] * x[i] for i in range(N)) <= capacity).add_name("weight limit")
    },
    sense=Instance.MAXIMIZE,
)

Attachable Data Formats#

The ParametricInstance above is the OMMX-form mathematical model passed to solvers. To make the experiment easier to inspect later, you can also attach surrounding data such as the original modeling object or input files to the Experiment.

If the original model was written in a modeling package, keep that source model as an Attachment as well. For external payload types, OMMX defines only the attachment codec protocol and the log_with_codec / get_with_codec methods that invoke it. The concrete codec should live in the package that owns the object type. This tutorial defines a temporary ProblemCodec for JijModeling Problem; JijModeling is expected to provide an equivalent codec in the future.

import jijmodeling as jm


class ProblemCodec:
    media_type = "application/vnd.jijmodeling.problem+protobuf"

    @staticmethod
    def encode(problem: jm.Problem) -> bytes:
        return problem.to_protobuf()

    @staticmethod
    def decode(data: bytes) -> jm.Problem:
        return jm.Problem.from_protobuf(data)


@jm.Problem.define("Knapsack Problem", sense=jm.ProblemSense.MAXIMIZE)
def jij_problem(problem: jm.DecoratedProblem):
    N = problem.Length(description="Number of items")
    W = problem.Float(description="Capacity")
    w = problem.Float(shape=N, description="Weight of each item")
    v = problem.Float(shape=N, description="Value of each item")
    x = problem.BinaryVar(
        shape=N,
        description="Set x_i=1 iff item i is in the knapsack",
    )

    problem += jm.sum(v[i] * x[i] for i in N)
    problem += problem.Constraint(
        "weight limit",
        jm.sum(w[i] * x[i] for i in N) <= W,
    )

If the payload already exists as a file, attach that file directly instead. log_file copies the file bytes into the Experiment, and later readers can use get_blob to read the bytes or write_attachment to restore the file to disk. This is the usual path for Excel workbooks, solver logs, generated plots, and other files produced outside OMMX.

import io
from pathlib import Path

experiment.log_file("input-spreadsheet", "input.xlsx")

spreadsheet_file = io.BytesIO(loaded_experiment.get_blob("input-spreadsheet"))
# Pass `spreadsheet_file` to a library that accepts a binary file-like object.
Path("restored").mkdir(parents=True, exist_ok=True)
loaded_experiment.write_attachment("input-spreadsheet", "restored/input.xlsx")

Run the Experiment#

This time, solve the knapsack problem above with two different capacities.

from ommx.experiment import Experiment
from ommx_highs_adapter import OMMXHighsAdapter

# Start an experiment. If no name is specified, one is assigned automatically.
with Experiment() as experiment:
    # Store the model as experiment-level information.
    experiment.log_parametric_instance("instance", pi)

    # Store the original JijModeling Problem through the temporary codec defined above.
    experiment.log_with_codec(
        ProblemCodec,
        "jijmodeling-problem",
        jij_problem,
    )

    # This example does not need it, but model metadata can also be stored as JSON.
    experiment.log_json(
        "source-data",
        {
            "description": "knapsack demo",
            "values": v,
            "weights": w,
        },
    )

    # Create two Runs with different capacities.
    for c in [47, 56]:
        # Materialize the model parameter.
        instance = pi.with_parameters({capacity.id: c})

        # Start a Run. A Run has setup and finalization, so using with is recommended.
        with experiment.run() as run:
            # Record capacity as a Run comparison parameter.
            run.log_parameter("capacity", c)

            # Call the HiGHS Adapter. The input Instance and output Solution are stored automatically.
            solution = run.log_solve(OMMXHighsAdapter, instance, verbose=False)

            # Confirm that the solver succeeded.
            assert solution.feasible

            # Also record the objective value as a Run comparison parameter.
            run.log_parameter("objective", solution.objective)

            # Leaving the with block finalizes the Run.

    # Leaving the experiment with block finalizes the Experiment.

All data stored during the experiment is saved in OMMX’s Local Registry.

The OMMX Local Registry is storage for efficiently keeping OMMX Artifact components. You can change its location with the OMMX_LOCAL_REGISTRY_ROOT environment variable. APIs such as with_temp_local_registry() can create and use a temporary Local Registry.
log_json and log_solve store data in the Local Registry immediately. They do not keep everything in memory and save it all at the end of the Experiment. Since storage paths are determined from the content of the data (SHA256 hash), identical data is stored only once per Local Registry.
When the Experiment is finalized, OMMX stores JSON (the Artifact Manifest) that lists all data saved during the Experiment, and stores a tag in the Local Registry pointing to this Artifact Manifest under the Experiment name chosen at startup or generated automatically.

Inspect a Shared Experiment#

Since an Experiment is identified by name, a shared Experiment can be loaded by name with load().

loaded_experiment = Experiment.load("ghcr.io/jij-inc/ommx/tutorial/experiment:knapsack")

This first searches the Local Registry by name. If it is not found, OMMX pulls it from the container registry, stores it in the Local Registry, and then loads it.

load() and import_archive() load an Experiment in the same state as an Experiment whose finalization has already completed. In this tutorial, we use the Experiment created above directly.

loaded_experiment = experiment

Run Parameters#

From a loaded Experiment, you can inspect the experiment information. First, run_parameters_df() lists the parameters recorded with log_parameter() for each Run as a pandas.DataFrame.

loaded_experiment.run_parameters_df()

For example, it should look like this.

        capacity  objective
run_id
     0        47         41
     1        56         49

Attachments#

Experiment-level Attachments can be checked by name and retrieved by name. get_attachment() checks the saved Media Type and returns JSON as a Python value, ParametricInstance as that object, and so on. If you know the expected type, use type-specific methods such as get_json() or get_parametric_instance(); they raise an error if the Media Type does not match.

# Check the names of saved Attachments.
assert loaded_experiment.attachment_names == [
    "instance",
    "jijmodeling-problem",
    "source-data",
]

# Retrieve data saved as JSON.
source_data = loaded_experiment.get_json("source-data")
assert source_data == {
    "description": "knapsack demo",
    "values": v,
    "weights": w,
}

# get_attachment uses the Media Type to decode the payload.
pi = loaded_experiment.get_attachment("instance")
assert isinstance(pi, ParametricInstance)

# The codec validates the Media Type and decodes the original payload.
restored_jij_problem = loaded_experiment.get_with_codec(
    ProblemCodec,
    "jijmodeling-problem",
)
assert restored_jij_problem.name == jij_problem.name

Runs and Solves#

The list of Runs is available from runs. Finished Runs are ordered by creation time, and each Run exposes its Attachments and Solves.

If a Run was recorded with trace storage enabled, trace returns the stored Run trace. Trace storage is an advanced feature; see Storing Run Traces in Experiments for details.

from typing import Any
from ommx.v1 import Solution

for run in loaded_experiment.runs:
    # Run IDs are assigned in execution order.
    assert run.run_id in [0, 1]

    # This example does not save run-level Attachments, so the count should be 0.
    assert len(run.attachment_names) == 0

    # Each Run calls the solver once, so the number of Solves should be 1.
    assert len(run.solves) == 1
    solve = run.solves[0]

    # Solve IDs are also assigned in execution order; here each Run has one Solve, so the ID should be 0.
    assert solve.solve_id == 0

    # Adapter name used for this Solve.
    assert solve.adapter.endswith("OMMXHighsAdapter")

    # Load input and output.
    input: Instance = solve.input
    output: Solution = solve.output

    # The knapsack problem should have been solved.
    assert output.feasible

    # Adapter options are also loaded.
    options: dict[str, Any] = solve.adapter_options
    assert "verbose" in options and options["verbose"] == False

Fork an Experiment#

Once an Experiment has been saved, it becomes immutable. You can still start a new Experiment from a saved Experiment. This operation is called a Fork. A forked Experiment inherits the same information as the original Experiment, but it starts again in an unfinalized running state, so you can add new Runs and Attachments. Use fork() to fork an Experiment.

with loaded_experiment.fork() as forked_experiment:
    # The forked Experiment inherits existing Runs, so the new Run ID starts from 2.
    with forked_experiment.run() as run:
        assert run.run_id == 2

        c = 64
        instance = pi.with_parameters({capacity.id: c})

        run.log_parameter("capacity", c)
        solution = run.log_solve(OMMXHighsAdapter, instance, verbose=False)
        assert solution.feasible
        run.log_parameter("objective", solution.objective)

The original Experiment is not modified. The forked Experiment contains the original Runs plus the newly added Run.

assert list(loaded_experiment.run_parameters_df().index) == [0, 1]
assert list(forked_experiment.run_parameters_df().index) == [0, 1, 2]

forked_df = forked_experiment.run_parameters_df()
assert forked_df.loc[2, "capacity"] == 64

A forked Experiment inherits Solve and Attachment data, but the data itself is stored in the Local Registry based on its content. Forking does not duplicate that data. Only the Artifact Manifest, which lists the stored data, is duplicated, and the forked Experiment points to the same data as the original Experiment.

When you share a forked Experiment with save() or push(), what you share is the entire forked Experiment. Attachments, Runs, and Solves inherited from the original Experiment are also included in the forked Artifact’s layers, so reading the forked Experiment does not require the original Experiment.

Record and Share Experiments

Contents

Record and Share Experiments#

Prepare the Mathematical Model#

Attachable Data Formats#

Run the Experiment#

Share the Experiment#

GitHub Container Registry#

Google Cloud Artifact Registry#

Export and Import as a File#

Inspect a Shared Experiment#

Run Parameters#

Attachments#

Runs and Solves#

Fork an Experiment#