Benchmarking Forecasting Agents with Point-in-Time Search

Arise PiT web search lets us evaluate forecasting agents on past events. Agents research with a fixed historical cutoff; we score their forecasts against what actually happened.

We use it here to benchmark 100 resolved prediction-market questions, with a research cutoff seven days before each event.

How search enables the evaluation

Live web search can reveal the answer to a past question. Arise PiT web search limits the evidence to what was available at the cutoff, so agents can research a question as of that date.

The agent has search, read, and bash tools to search for historical evidence and work with the retrieved documents.

One forecast, end to end

Here is one successful forecast from the benchmark. With research cut off a week before NVIDIA’s earnings, Astra makes 19 tool calls before returning its probability. The full sequence is shown below.

Where the 100 questions come from

The questions come from resolved binary markets on Polymarket and Kalshi, with explicit YES/NO resolution rules. Each cutoff is seven days before the verified event or first public release.

Eligible markets were already open and unresolved at the cutoff, with historical YES odds between 20% and 80%. From a reviewed bank of 195 questions, we fixed 100 with eligible market data and valid outputs from the initial comparison systems, without using prediction accuracy to select them.

The table contains the full question set. All times are UTC; outcomes are shown for readers and were withheld from the agents.

Benchmarking the forecasts

Each agent returns a probability of YES. We report accuracy, with probabilities of 50% or higher counted as YES, and Brier score, the mean squared error of that probability against the 0/1 outcome. Higher accuracy and lower Brier scores are better. The T−7 market uses the latest eligible odds at or before the same cutoff.

Fixed 100-question benchmark

Forecasting accuracy

OpenAIAnthropicDeepSeekAlibaba / QwenApodex AIMarket odds

Probability quality

Brier score · lower is better

Qwen3.8-27B has one missing answer: accuracy counts it as incorrect; Brier uses its 99 valid forecasts. Other systems have 100. Apodex uses the more accurate of the two tested workflows: workspace, at 59/100.

Astra scores 74%, Fable and Opus each score 71%, and the same-time market scores 63%. Astra also has the lowest Brier score, 0.181, compared with 0.226 for the market.

To test the importance of external evidence, we also run Astra and Opus without tools, documents, or market prices. These no-search runs may use any internal knowledge, including remembered outcomes.

Why search matters

PIT Search vs. No Search

Brier scoreLower is better

Model	No Search	PIT Search
Astra	0.263	0.181
Opus 5	0.315	0.208

PIT research scores 18 percentage points higher for Astra and 16 for Opus, with lower Brier scores for both. The comparison shows why search matters for these tasks, even when the no-search models are allowed to recall outcomes.

Different tasks, different evidence

Saved research lets us go beyond the scores and examine how agents used the evidence available at each cutoff.

By task category

Forecasting accuracy by task

Macroeconomics: reason about the threshold

Macroeconomic forecasts depend on mapping the available signals to an exact threshold. On a question about payroll growth above 60,000, Astra cites estimates of 55,000 and 65,000 and forecasts 49% YES. The outcome is YES. The historical record lets us inspect a concrete failure: the agent found forecasts on both sides of the threshold but put its final probability just below 50%.

Technology and business: use the right metric

Company forecasts depend on matching the evidence to the exact metric and reporting period. In the NVIDIA example above, the question asks about Data Center revenue, while the guidance and analyst forecasts cover total revenue. The saved research lets us inspect how the agent bridges that difference.

Politics and elections: a smaller sample

Political forecasts draw on polling, endorsements, and local reporting. Astra and Fable each answer 10 of 11 questions correctly, compared with seven for the market. A few outcomes can move this small category’s ranking substantially. The four sports and three culture questions remain in the overall scores but are also too few for broad category claims.

What search enables