One Shot Is Not Enough

Why your LLM forecaster needs agentic search

Note: this post is part of a series about building an autonomous forecasting agent. Part 1 can be found here.

When I entered the Metaculus AI forecasting tournament, I thought I signed up to build a forecasting agent. I was wrong. I signed up to build a research agent. I estimate 80%+ of my time has been dedicated to improving how the agent locates, ingests, and processes information to make a forecast possible.

This post (hopefully) offers lessons for anyone looking to build multi-step, dynamic, tool-enriched agent workflows (for forecasting or otherwise).

Look, if you had one shot or one opportunity / To research everything you ever needed in one moment / Would you capture it or just let it slip?

The death-knell for one-shot search strategy

Suppose you are asked: “Will the U.S. Supreme Court grant certiorari in at least 10 new cases before May 1, 2026?” (Never mind why you are asked or why you would care). How would you answer?

What a good forecaster will do: Check the time period (Feb 24-May 1) and how the question will resolve (what data source). Go to the official source first, and, on finding one PDF per court term, start with 2026 and work backwards, year-by-year, until the typical cadence for certiorari grants is clear.

If they didn’t already know, they’ll discover it’s quite seasonal and fairly predictable. Based on past history and the current status (zero cases), they’ll assign a very low probability, maybe 10-15%.¹

What my bot did: Absolutely none of this.

It didn’t do it because it couldn’t. Don’t blame the bot. Blame the maker. When this question posted on March 9, my bot could read it, make a search plan, execute all its queries simultaneously, and aggregate the results. It couldn’t complete a multi-step task like go to website, click on link, read linked page. It couldn’t read a PDF at all. It couldn’t do any of the things needed to answer this question or others like it.

Instead, my bot, given one shot to search for information, obtained the total number of cases granted certiorari in recent years, considered the number of days remaining, reasoned: “[t]he Court grants ~65-75 cases per term over ~40 active weeks = ~1.6-1.9 grants/week. Over 9.5 weeks = ~15-18 expected grants. The threshold of 10 is well below the expected value,” and assigned a probability of 76.4%.

That little orange dot is haunting me

Seventy-six point four percent!!! I was beside myself. I couldn’t believe it. It was even worse than the already-bad consensus of 55.5%. I was fed up with these recurring, grotesque, indefensible failures.²

Efforts to improve the research plan prompt were insufficient; giving the researcher more tools (PDF reader, CSV parser, stock lookup, etc.) was insufficient. The researcher needed more shots; it needed to become Agentic.

Agentic search is just human search on steroids

It is almost too obvious to notice, but the “good forecaster” approach is iterative: start with a basic search, orient yourself to the domain, do increasingly-targeted searches, and continually adjust based on what you discover and what you realize you don’t know yet, but need to.

Agentic search is just jargon for, “an LLM autonomously does multiple rounds of research.” It’s scary and potentially error-prone because the LLM is in control. Even if you provide instructions, each iterative step can and will deviate from them in ways that can be unpredictable. But it’s also powerful because the agent can consume extraordinary amounts of information in seconds, and potentially adapt its strategy as it learns. Most critically, it gives the LLM multiple chances to find the unknown knowns to which it is uniquely blind:

Studies of LLM-based forecasting confirm, generally, that one-shot research outperforms zero-shot, but agentic search outperforms both. My own bot’s deficiencies made plain why a one-shot strategy underperforms, and I used them as the basis for designing its replacement.

Understanding failure modes

First, I collected a dozen prior forecasts where, like the Supreme Court question above, the bot produced a poor response due to a lack of knowledge or context about the question, and where I believed agentic search would have helped. I classified the types of failures I saw and fed all the artifacts into Claude Code. “Help me understand what is going wrong here. Don’t jump to solutions.” I also gave it papers on best practices for LLM-based forecasting and RAG (retrieval augmented generation) search. This strategy helped me identify the key gaps between my implementation of search and what state-of-the-art deep research tools do.

With a rough idea of what state-of-the-art looked like, my next steps were to decide: what do I build vs. buy, and how do I approach prototyping?

Build vs buy

I 100% custom-built my research agent for a simple reason: because I wanted to learn how. The big labs have deep research tools you can call, and they are surely better than what I cobbled together, but you cede control and the costs can spiral. You also don’t get the full trace: what did the agent do, what did it find, how did it summarize it, how did it incorporate it into its final research product? To me, that visibility was far more important than the performance tradeoff. There are also paid frameworks like LangChain/LangGraph and open source ones like GPT Researcher and I’m sure these could help me at some point but I didn’t want to add more dependencies before I even knew what I was doing; I also wanted to maintain full control and not outsource design decisions to an external dependency.³

Backtesting vs A/B testing vs “good enough” testing

Backtesting is the gold standard for evaluating LLM forecaster performance. You build multiple models and have them answer questions that resolved after the LLM’s knowledge cutoff. Mantic, the leader in building AI forecasters, built a backtesting engine to tune their forecasts as described by their CEO in this interview. Bridgewater’s AI lab does the same thing.

For hobbyists, this approach is probably too resource intensive and costly. Further, if you’re building your bot around news ingestion, it is impractical to forecast resolved questions - the bot will find the answer.

A simpler approach is to A/B test against live markets, where the outcome isn’t yet known, but you have to wait for enough results to be confident whether one version beats another. You also pay for full multiple model runs each time.

Even if I had the time, money, and resources to build multiple models and set up backtesting, it would be premature to do so. The early versions of my bot produced too many obvious, major errors to justify such an investment. So, I do “good enough” testing: instead of full model runs of multiple variations, I took whichever discrete step in my pipeline underperforms most, prototype a few variants, re-run that one step, and select whichever candidate produces the consistently best-looking output, if one emerges (often my prototypes reach a dead-end and I put them aside).

Like forecasters, product builders need to exercise judgment under uncertainty. Every subsequent question posted to the tournament offered another chance for my bot to beclown itself against the competition - time was not on my side, so I settled for “good enough.”

Finding “good enough”

I tried many different approaches to question decomposition, reference class identification, and other forecasting best practices. At one point I had the bot trying to identify, all at once, key drivers, background conditions, resolution mechanisms, base rates, and contrarian views—and then produce a research plan to execute in multiple steps.

In general, everything underperformed my expectations. The more I asked it to do at once, the worse the bot seemed to perform on each discrete task. I simplified the initial research planning step to

1. Identify uncertainties (and number them, as U1, U2, etc.) that are important to resolve to forecast the question, and

2. Plan research steps, in order of importance, tied explicitly to these uncertainties

By focusing on just these two things, the planning step consistently produced an output that looked good enough - that sufficiently resembled what I as a forecaster would do.

This plan is handed over to a research agent operating in ReAct (Reasoning and Acting) iterative loop pattern. At each step, the agent gets the question, the plan, and any prior research and is instructed to follow these canonical steps:

Thought: The agent internally reasons, decomposes goals, and develops subplans via natural language.

Action: The agent invokes a tool, queries an API, or executes a domain-specific command.

Observation: The environment’s reply is received and added to the agent’s context, enabling subsequent thoughts to be conditioned on actual data.

The agent continues this loop until it has addressed all its original uncertainties, or until it runs out of steps. It is encouraged to deviate from the original plan as it learns new information.

I experimented with different numbers of turns, actions per turn (how many tool calls it could make), more or less summarization/compression of the information gathered, until I had reached “good enough” - materially better research content than the status quo one-shot search output across a diversity of forecast questions.

Briefing the forecasters

The output of the ReAct research can easily run 25,000 words (about 70 pages of a Word doc), and this is already after summarizing articles rather than including the full text of each. It would be cost prohibitive to send this output to each of the 10 forecasters in my ensemble, so it is instead sent one time to a Brief Writer, who pulls out the key findings and produces a finished product for each forecaster to review. Again, I experimented with 6 different prompts for this step, pulling the best parts from each until I had a single output that was consistently high quality.

The research pipeline now looks like this:

I'm not claiming to have invented anything here; I am simply trying to implement what others have identified as best practice, and continually tweak it until I can’t find any further meaningful improvements.

Better research; too much consensus?

The Supreme Court question posted on March 9. I already had a very light prototype of iterative agentic search, but it was very far from a production-ready product. After four feverish days of building, I launched it into production. It remains, by far, the biggest change to my bot system since its inception. Of ~600 questions forecast, half predate and half postdate this change. Whether the bot is better at forecasting now remains an open question. My median peer score (how I performed relative to all other forecasters on a question) has improved >70%, but they are different question sets. Anecdotally, reading the research output, it’s vastly more relevant and complete. It’s embarrassing to admit but I sometimes feel proud of just how well it surfaced and contextualized the key information.

The most evident change in forecasting performance is that the level of disagreement within the ensemble has halved - the forecasters independently assess the evidence and come up with, substantially, the same result. My supervisor, which is triggered above a pre-defined divergence level, is left to twiddle its thumbs. With one-shot research, the Supervisor was triggered 21% of the time. With ReAct, it’s down to 6%. I am contemplating lowering the threshold.

I’m now concerned the forecasters agree too much - that the research and its presentation is herding them toward one possible conclusion, constraining their ability to independently assess the question. At the moment I’m exploring prompting strategies to try and make the Research Brief more neutral, or to explicitly highlight diverging information and interpretations. I believe there is a goldilocks zone for ensemble disagreement and I may have unintentionally leapfrogged from one extreme (wild disagreement based on poor information) to another (groupthink).

Expert forecasters can look at the same evidence base and come to vastly different conclusions. That my forecasters are not doing so suggests I have upgraded to a higher class of problem.

1. This was at the question opening date of March 9. As of today there are 3 cases granted certiorari with 14 days left; the probability is probably closer to 2%.

2. Sadly, this is far from the worst forecast my bot produced, particularly in the early days; it’s just the one that convinced me it was time to rebuild search, that I had improved as much as I could with my existing architecture and it wasn’t enough.

3. Implications for the SaaS-pocalypse narrative: yes, it was relatively easy to build what I have, but past a certain point, I’m pretty confident I will turn to a paid solution. “Everyone will write all their own software” seems misguided. Hobbyists like me will build from scratch and create future demand for paid offerings.