OCA Run a benchmark →

How a run works

  1. 1. You buy a bundle

    Pick a bundle on the pricing page and pay via Stripe. The webhook mints a license key of the form customer|hmac_sha256_hex and emails it to you.

  2. 2. You point at your model endpoint

    You stand up an HTTP service that accepts POST with a JSON challenge body and returns a JSON answer. You can use any framework. The challenge spec field tells you the expected shape (multiple_choice, json, free_text, …).

  3. 3. We score server-side

    We forward each challenge to your endpoint, collect your responses and score them against the held-out ground truth using the same reference scorer you saw in the public practice set.

  4. 4. You get a signed receipt

    We sign the canonical-JSON result with HMAC-SHA256 using RECEIPT_SECRET and seal it in a Durable Object. The receipt is immutable and verifiable forever.

API

Base URL: https://oca-api.example/v1 · All routes require Authorization: <license> except /v1/modules, /v1/bundles, and /v1/billing/checkout.

POST /v1/runs

Run an evaluation against your endpoint.

{
  "bundle":           "frontier",
  "seed":             1738801,            // optional; deterministic if set
  "n_per_module":     4,                  // optional; default 4
  "customer_endpoint": "https://your-model.example/v1/oca"
}

GET /v1/runs/:id

Fetch a sealed receipt.

GET /v1/runs

List your past runs (filtered by your license's customer_id).

POST /v1/billing/checkout

Create a Stripe Checkout Session for a bundle.

{ "bundle": "frontier", "customerEmail": "ops@your-lab.example" }

License keys

A license is customer_id|hmac_sha256_hex. The HMAC is computed over customer_id using the server's LICENSE_SECRET. Verify on your end by recomputing the HMAC and comparing — do not ship LICENSE_SECRET to clients.

Send it as either header:

Authorization: customer_id|hmac_sha256_hex
Authorization: Bearer customer_id|hmac_sha256_hex

What we keep, what we leak

We keep everything: challenges, ground truths, scorers, seeds, customer endpoints and responses. We leak nothing: customers see (a) the challenges during a run, (b) their own responses echoed back, and (c) the signed receipt. The held-out test set never leaves our Durable Object.