Replay decisions with the evidence available then.

Define the decision before revealing the outcome.

A prediction-agent backtest runs a question at a historical decision point and scores the answer against what happened later. Define the question, cutoff, and scoring rule before inspecting the result. Keep the outcome out of the agent's search query and input context.

Use a cutoff that falls before the event being predicted. Search dates resolve to the end of the selected UTC day: for a decision during a day, use an earlier completed UTC day or design the evaluation around daily boundaries.

Save the evidence for each decision date.

This Python example uses only the standard library. It sends two historical searches and saves the request and complete response locally. Those saved responses can become fixed inputs to your agent evaluation.

import json, os
from pathlib import Path
from urllib.request import Request, urlopen

# Choose questions and dates before inspecting their outcomes.
cases = [
    ("2026-08-13", "nvidia blackwell supply"),
    ("2026-08-20", "nvidia blackwell supply"),
]
output = Path("evidence")
output.mkdir(exist_ok=True)
for date, query in cases:
    body = {
        "query": query, "mode": "standard", "k": 5,
        "as_of_date": date, "recency": "7d",
    }
    request = Request(
        "https://api.ariselabs-search-api.com/v1/search",
        data=json.dumps(body).encode(),
        headers={
            "Authorization": "Bearer " + os.environ["ARISELABS_API_KEY"],
            "Content-Type": "application/json",
        },
    )
    with urlopen(request, timeout=60) as response:
        evidence = json.load(response)
    # Exclusive creation protects an earlier run from being overwritten.
    with (output / f"{date}.json").open("x") as saved:
        json.dump({"request": body, "response": evidence}, saved, indent=2)
    print(date, len(evidence["data"]), "sources saved")

Separate retrieval, prediction, and scoring.

1. Retrieve. Send every search through standard mode with the same case-specific as_of_date. Give the agent the indexed text returned by search.
2. Predict. Record the agent's prediction, confidence, cited evidence, and model configuration before exposing the outcome.
3. Score. Compare the recorded prediction with the later outcome using your chosen metric. For probability forecasts, a proper scoring rule such as the Brier score can measure calibration and accuracy.
4. Compare. Hold the evaluation cases and saved evidence fixed when comparing prompts, models, or agent policies. Keep training cases separate from held-out evaluation cases.

The search API supplies evidence. Your evaluation code defines the outcome labels, prediction format, metric, and any reward passed back into training.

Control the other paths to future information.

Standard search limits eligible documents by indexing time, through the end of the cutoff day. Publication time is separate. Coverage starts in 2025 and depends on what the corpus contains; see the time semantics reference.

Advanced search may use live web sources. Fetch and ordinary web browsing can expose pages updated after the decision date. Keep those tools out of the historical evidence path, and remember that a model's parameters can already encode later events. A retrieval cutoff alone does not establish a leakage-free backtest.

The live index and ranking can change. Save evidence for reproducibility, record empty or degraded results, and retain failed cases in your evaluation accounting. Dropping difficult cases silently can bias the comparison.

Use outcomes to improve the policy.

The same setup can support reinforcement learning: an agent searches within the historical boundary, makes a prediction, and receives a reward from a later outcome. Keep the held-out evaluation independent so a better training score can be tested against unseen cases.

Our research notes explain the ideas behind this approach. Start with the Search API reference and use per-request pricing to budget the retrieval calls in your evaluation.