Close Menu
    Business

    NVIDIA PivotOPD Teaches Multi-Turn AI Agents to Recover From Pivotal Mistakes

    ◀


    NVIDIA researchers, with Princeton University and the University of Maryland, have introduced PivotOPD, an on-policy distillation method for multi-turn LLM agents. PivotOPD on-policy distillation trains an agent to avoid its most damaging early mistake, and to recover when it happens anyway. Against 13 baselines, it posts the best average on ALFWorld, WebShop and Search-based QA for Qwen3-1.7B and Qwen3-8B students. The takeaway: recovery is learnable, and standard OPD rarely teaches it.

    TL;DR

    • Size: A training method, not a model. Tested on Qwen3-1.7B and Qwen3-8B students, plus a Nemotron-3.5-SFT student on SWE-Bench Verified.
    • Runs on: Trained on NVIDIA H100 nodes. Adds 0 inference cost, so the trained agent runs wherever its base model runs.
    • Performance: First on all 8 per-benchmark averages against 13 baselines, across 3 seeds.
    • Best: Recovers from 72.7% of replayed pivotal mistakes, vs 20.3% for standard OPD.
    • Worst: 55.9% on ALFWorld “Look” tasks with the 1.7B student, vs 83.9% for SOD.
    • Bottom line: Best: teaches recovery that outcome-only RL cannot reach. Worst: depends on replayable environments and a teacher whose pivots match the oracle in 77.8% of failed rollouts.

    What is a pivotal mistake in a multi-turn agent?

    A pivotal mistake is an action that lengthens the shortest remaining path to finishing a task, or makes it unsolvable. ALFWorld’s symbolic oracle measures this at every turn.

    Across Qwen3-8B, Qwen3-30B-A3B and Qwen3-235B-A22B, 59% of failed rollouts (155 of 262) contained one. The first pivotal turn arrived early, at a median of turn 8 to 12 out of 30. The agents then wasted 18 to 21 more turns without recovering.

    In replays of Qwen3-8B failures, correcting the pivotal turn raised success from 8% to 59%. Leaving the mistake in place and forcing the right action for the next 2 turns still reached 58%.

    Why does standard on-policy distillation miss it?

    Standard OPD lowered the held-out failure rate from 79% to 56%. Failures after a pivotal turn only fell from 51% to 49%. The correct action stayed below 1% probability at every pivotal turn, so 8 rollouts rarely sample it.

    Outcome-based RL shares the blind spot: if every rollout fails, the group-relative advantage is 0.

    How does PivotOPD work?

    PivotOPD adds 3 components to group-based RL, combined in a single PPO update.

  • Pivot detection: A larger teacher model reads each rollout and its outcome in hindsight. It picks candidate turns and names a gold action at each. A turn counts as pivotal when the student’s action differs from the gold action. On ALFWorld, detected pivots land within 1 turn of the oracle’s pivot in 77.8% of failed rollouts on average.
  • Preventive distillation (reverse KL): A frozen copy of the student, hinted with the gold action, re-scores the student’s own response. This pushes the student away from the committed mistake.
  • Recovery distillation (forward KL): After each pivot, the teacher names a recovery action for up to K turns. The hinted self-teacher writes recovery responses, and the unhinted student trains on them. Forward KL is mass-covering, so it lifts actions the student almost never samples.

    • The teacher only names actions. Token-level targets come from the student’s own hinted distribution.

    How does PivotOPD perform on agent benchmarks?

    With the 1.7B student, PivotOPD averages 73.7% on ALFWorld, 5.5 points above SDAR. It averages 44.5% on Search-based QA, 5.9 points above RLSD. On WebShop it beats RLSD by 1.2 in score but by 14.1 in success rate (76.6%).

    With the 8B student, it reaches 93.0% on ALFWorld, 47.4% on Search-based QA and 81.9% WebShop success. Margins are smaller, at least 1.8 points.

    With Qwen3-8B as its own teacher, PivotOPD still wins all 3 benchmarks by at least 1.5 points, 3.9 on average.

    On SWE-Bench Verified, a Nemotron-3.5-SFT student taught by Nemotron-3-Super went from 62.8% to 66.0%. Standard OPD reached 63.0%, and the teacher scores 73.0%.

    Recovery is the standout. Across 72 replayed pivotal mistakes, PivotOPD recovered 72.7% of the time, vs 8.3% for the base model, 20.3% for standard OPD and 45.8% for preventive-only. It averaged 9.7 turns to recover, against an optimal 6.2.

    #mtp-pivotopd-wrap iframe{width:100%!important;border:0!important;display:block;background:#111!important;border-radius:14px;}#mtp-pivotopd-wrap hr,#mtp-pivotopd-wrap p:empty,#mtp-pivotopd-wrap del,#mtp-pivotopd-wrap s{display:none!important;}
    <iframe id="mtp-pivotopd-frame" title="PivotOPD interactive explainer" loading="lazy" height="600" style="height:600px;" srcdoc="<!DOCTYPE html>
    <html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
    <title>PivotOPD Explainer</title>
    <style>
    html,body{margin:0;padding:0;background:#111!important;}
    #mtp-pivotopd{max-width:880px;margin:0 auto;background:#111!important;color:#e8e8e8!important;font-family:Inter,"Segoe UI",system-ui,sans-serif;border:1px solid #2a2a2a!important;border-radius:14px;overflow:hidden;box-sizing:border-box;}
    #mtp-pivotopd *{box-sizing:border-box;}
    #mtp-pivotopd hr,#mtp-pivotopd p:empty,#mtp-pivotopd del,#mtp-pivotopd s{display:none!important;}
    #mtp-pivotopd .hd{padding:20px 24px 14px;border-bottom:1px solid #222!important;background:linear-gradient(135deg,#151a0c,#111)!important;}
    #mtp-pivotopd .kick{font-size:11px;letter-spacing:.14em;text-transform:uppercase;color:#76B900!important;font-weight:700;}
    #mtp-pivotopd h2{margin:6px 0 4px;font-size:22px;color:#fff!important;line-height:1.25;}
    #mtp-pivotopd .sub{margin:0;font-size:13.5px;color:#a8a8a8!important;}
    #mtp-pivotopd .tabs{display:flex;gap:6px;padding:12px 16px;background:#141414!important;border-bottom:1px solid #222!important;flex-wrap:wrap;}
    #mtp-pivotopd .tab{background:#1c1c1c!important;color:#cfcfcf!important;border:1px solid #2c2c2c!important;border-radius:999px;padding:7px 14px;font-size:13px;cursor:pointer;font-family:inherit;transition:all .2s;}
    #mtp-pivotopd .tab:hover{border-color:#76B900!important;color:#fff!important;}
    #mtp-pivotopd .tab.on{background:#76B900!important;color:#0b0b0b!important;border-color:#76B900!important;font-weight:700;}
    #mtp-pivotopd .panel{display:none;padding:20px 24px 22px;animation:mtpfade .35s ease;}
    #mtp-pivotopd .panel.on{display:block;}
    @keyframes mtpfade{from{opacity:0;transform:translateY(6px)}to{opacity:1;transform:none}}
    #mtp-pivotopd p{font-size:14px;line-height:1.55;color:#d6d6d6!important;margin:0 0 12px;}
    #mtp-pivotopd .muted{color:#8f8f8f!important;font-size:12px;}
    #mtp-pivotopd .turns{display:flex;gap:4px;margin:14px 0 8px;flex-wrap:wrap;}
    #mtp-pivotopd .t{width:22px;height:22px;border-radius:5px;background:#2a2a2a!important;display:flex;align-items:center;justify-content:center;font-size:10px;color:#777!important;transition:background .25s,transform .25s;}
    #mtp-pivotopd .t.ok{background:#4a5a7a!important;color:#e6ecff!important;}
    #mtp-pivotopd .t.pv{background:#e0603a!important;color:#fff!important;transform:scale(1.15);}
    #mtp-pivotopd .t.wa{background:#3a2a2a!important;color:#b88!important;}
    #mtp-pivotopd .t.rc{background:#76B900!important;color:#0b0b0b!important;}
    #mtp-pivotopd .legend{display:flex;gap:14px;flex-wrap:wrap;font-size:12px;color:#aaa!important;margin-bottom:10px;}
    #mtp-pivotopd .legend i{display:inline-block;width:10px;height:10px;border-radius:3px;margin-right:5px;vertical-align:-1px;}
    #mtp-pivotopd .btn{background:transparent!important;color:#76B900!important;border:1px solid #76B900!important;border-radius:8px;padding:7px 14px;font-size:13px;cursor:pointer;font-family:inherit;margin-right:6px;}
    #mtp-pivotopd .btn:hover{background:#76B900!important;color:#0b0b0b!important;}
    #mtp-pivotopd .stats{display:grid;grid-template-columns:repeat(3,1fr);gap:10px;margin-top:14px;}
    #mtp-pivotopd .stat{background:#181818!important;border:1px solid #262626!important;border-radius:10px;padding:12px;}
    #mtp-pivotopd .stat b{display:block;font-size:24px;color:#76B900!important;}
    #mtp-pivotopd .stat span{font-size:12px;color:#aaa!important;line-height:1.35;display:block;}
    #mtp-pivotopd .steps{display:grid;grid-template-columns:repeat(3,1fr);gap:10px;margin:6px 0 14px;}
    #mtp-pivotopd .step{background:#181818!important;border:1px solid #262626!important;border-radius:10px;padding:12px;cursor:pointer;transition:all .25s;opacity:.55;}
    #mtp-pivotopd .step.on{opacity:1;border-color:#76B900!important;box-shadow:0 0 0 1px #76B900 inset,0 0 18px rgba(118,185,0,.18);}
    #mtp-pivotopd .step .n{font-size:11px;color:#76B900!important;font-weight:700;letter-spacing:.1em;}
    #mtp-pivotopd .step h4{margin:4px 0 4px;font-size:14px;color:#fff!important;}
    #mtp-pivotopd .step small{font-size:12px;color:#9a9a9a!important;}
    #mtp-pivotopd .detail{background:#151a0c!important;border:1px solid #2f4200!important;border-radius:10px;padding:14px;min-height:96px;}
    #mtp-pivotopd .detail h5{margin:0 0 6px;font-size:14px;color:#9fe02a!important;}
    #mtp-pivotopd .flow{display:flex;align-items:center;gap:6px;margin:10px 0 2px;flex-wrap:wrap;font-size:12px;}
    #mtp-pivotopd .chip{padding:4px 9px;border-radius:6px;background:#222!important;color:#ddd!important;border:1px solid #333!important;}
    #mtp-pivotopd .chip.g{background:#76B900!important;color:#0b0b0b!important;border-color:#76B900!important;font-weight:700;}
    #mtp-pivotopd .chip.r{background:#e0603a!important;color:#fff!important;border-color:#e0603a!important;}
    #mtp-pivotopd .arr{color:#666!important;}
    #mtp-pivotopd .tog{display:inline-flex;border:1px solid #333!important;border-radius:8px;overflow:hidden;margin-bottom:12px;}
    #mtp-pivotopd .tog button{background:#1a1a1a!important;color:#bbb!important;border:0!important;padding:6px 12px;font-size:12px;cursor:pointer;font-family:inherit;}
    #mtp-pivotopd .tog button.on{background:#76B900!important;color:#0b0b0b!important;font-weight:700;}
    #mtp-pivotopd .bench{margin-bottom:14px;}
    #mtp-pivotopd .bench h6{margin:0 0 6px;font-size:12.5px;color:#cfcfcf!important;font-weight:600;}
    #mtp-pivotopd .row{display:flex;align-items:center;gap:8px;margin:4px 0;font-size:12px;}
    #mtp-pivotopd .row .lb{width:92px;color:#aaa!important;flex:none;}
    #mtp-pivotopd .row .tr{display:block;flex:1;height:16px;background:#1d1d1d!important;border-radius:4px;overflow:hidden;}
    #mtp-pivotopd .row .bar{display:block;height:100%;width:0;background:#4a5a7a!important;border-radius:4px;transition:width 1s cubic-bezier(.2,.8,.2,1);}
    #mtp-pivotopd .row.me .bar{background:#76B900!important;}
    #mtp-pivotopd .row.me .lb{color:#76B900!important;font-weight:700;}
    #mtp-pivotopd .row .v{width:40px;text-align:right;color:#ddd!important;flex:none;}
    #mtp-pivotopd .case{display:grid;grid-template-columns:1fr 1fr;gap:10px;margin-top:12px;}
    #mtp-pivotopd .col{background:#181818!important;border:1px solid #262626!important;border-radius:10px;padding:10px;}
    #mtp-pivotopd .col h6{margin:0 0 6px;font-size:12px;color:#fff!important;}
    #mtp-pivotopd .ln{font-size:12px;padding:5px 7px;border-radius:6px;margin:3px 0;color:#ccc!important;background:#202020!important;opacity:0;transform:translateX(-6px);transition:all .3s;}
    #mtp-pivotopd .ln.show{opacity:1;transform:none;}
    #mtp-pivotopd .ln.bad{background:#3a1d16!important;color:#f3b5a2!important;}
    #mtp-pivotopd .ln.good{background:#1f2c08!important;color:#c6f07a!important;}
    #mtp-pivotopd .ft{padding:10px 24px;border-top:1px solid #222!important;font-size:12px;color:#777!important;display:flex;justify-content:space-between;flex-wrap:wrap;gap:6px;background:#0f0f0f!important;}
    #mtp-pivotopd .ft b{color:#76B900!important;}
    #mtp-pivotopd a{color:#76B900!important;text-decoration:none;}
    @media (max-width:640px){
    #mtp-pivotopd .hd,#mtp-pivotopd .panel{padding-left:16px;padding-right:16px;}
    #mtp-pivotopd h2{font-size:19px;}
    #mtp-pivotopd .stats,#mtp-pivotopd .steps,#mtp-pivotopd .case{grid-template-columns:1fr;}
    #mtp-pivotopd .t{width:18px;height:18px;font-size:9px;}
    #mtp-pivotopd .row .lb{width:74px;}
    }
    </style></head><body>
    <div id="mtp-pivotopd">
    <div class="hd">
    <div class="kick">NVIDIA Research · Interactive explainer</div>
    <h2>PivotOPD: teaching agents to recover from pivotal mistakes</h2>
    <p class="sub">On-policy distillation that prevents early agent errors and trains recovery from them.</p>
    </div>
    <div class="tabs" role="tablist">
    <button class="tab on" data-p="0">1. The problem</button>
    <button class="tab" data-p="1">2. How it works</button>
    <button class="tab" data-p="2">3. Benchmarks</button>
    <button class="tab" data-p="3">4. Recovery</button>
    </div>

    <div class="panel on" data-p="0">
    <p>In ALFWorld, 59% of failed rollouts across 3 Qwen3 models contain a <b style="color:#e0603a">pivotal mistake</b>: an action that lengthens the shortest path to finishing the task. It lands early, then the agent wastes the rest of the 30-turn episode.</p>
    <div class="legend"><span><i style="background:#4a5a7a"></i>Correct turn</span><span><i style="background:#e0603a"></i>Pivotal mistake</span><span><i style="background:#3a2a2a"></i>Wasted turn</span><span><i style="background:#76B900"></i>Recovery</span></div>
    <div class="turns" id="mtp-turns"></div>
    <div style="margin:10px 0 2px"><button class="btn" id="mtp-play1">Play failed rollout</button><button class="btn" id="mtp-play2">Play with recovery</button></div>
    <p class="muted" id="mtp-cap">Illustrative 30-turn episode. Pivot shown at turn 10, inside the reported median range of 8 to 12.</p>
    <div class="stats">
    <div class="stat"><b data-c="59">0%</b><span>replayed success after fixing the pivotal turn (from 8%)</span></div>
    <div class="stat"><b data-c="58">0%</b><span>success when only the 2 turns after the mistake are guided</span></div>
    <div class="stat"><b>51% → 49%</b><span>pivotal-turn failures after standard OPD barely move</span></div>
    </div>
    </div>

    <div class="panel" data-p="1">
    <p>PivotOPD adds 3 parts to group-based RL, all applied in one PPO update. Click a step or use the buttons.</p>
    <div class="steps">
    <div class="step on" data-s="0"><div class="n">STEP 01</div><h4>Pivot detection</h4><small>Teacher finds the turn and the gold action</small></div>
    <div class="step" data-s="1"><div class="n">STEP 02</div><h4>Preventive distillation</h4><small>Reverse KL, on the student's own response</small></div>
    <div class="step" data-s="2"><div class="n">STEP 03</div><h4>Recovery distillation</h4><small>Forward KL, on self-teacher responses</small></div>
    </div>
    <div class="detail" id="mtp-detail"></div>
    <div style="margin-top:12px"><button class="btn" id="mtp-prev"> Prev</button><button class="btn" id="mtp-next">Next ▶</button></div>
    </div>

    <div class="panel" data-p="2">
    <div class="tog" id="mtp-tog"><button class="on" data-m="s">Qwen3-1.7B</button><button data-m="l">Qwen3-8B</button></div>
    <div id="mtp-bench"></div>
    <p class="muted">Averages over 3 seeds from Table 1 of the paper. Compared with the strongest baselines. PivotOPD ranks first on all 8 per-benchmark averages against 13 baselines.</p>
    <div class="bench"><h6>SWE-Bench Verified resolve rate (Nemotron-3.5-SFT student)</h6><div id="mtp-swe"></div></div>
    </div>

    <div class="panel" data-p="3">
    <p>72 oracle-labeled pivotal mistakes, replayed 8 times per policy. How often does each policy finish the task anyway?</p>
    <div id="mtp-rec"></div>
    <p class="muted">Average turns to recover: PivotOPD 9.7, standard OPD 12.3, base 13.4, optimal 6.2.</p>
    <button class="btn" id="mtp-case">Replay case study</button>
    <div class="case">
    <div class="col"><h6>Base model</h6><div id="mtp-ca"></div></div>
    <div class="col"><h6 style="color:#76B900">PivotOPD model</h6><div id="mtp-cb"></div></div>
    </div>
    </div>

    How does PivotOPD compare with other agent distillation methods?

    What does it cost to train, and can you run it?

    PivotOPD changes only training, so inference costs nothing extra. On ALFWorld with the 1.7B student, on 4 H100 GPUs, the overhead over GRPO is 12.4% at K = 1. It jumps to 94.2% at K = 2, because later recovery turns need environment replay. The selected budget is K = 2 on ALFWorld and K = 1 on WebShop and Search-based QA.

    Key Takeaways

    • Over half of failed agent rollouts hinge on 1 early, recoverable mistake.
    • Standard OPD barely touches these failures: 51% to 49%.
    • PivotOPD recovers from 72.7% of replayed mistakes, vs 20.3% for OPD.
    • Best average against 13 baselines on ALFWorld, WebShop and Search QA.
    • 0 inference overhead, but code is not yet released.

    Check out the paper and the project page. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

    [Sponsored] The web is the one API most agents are missing. Databases, calendars and repos have APIs. The open web mostly doesn’t. The TinyFish MCP server gives any MCP client four tools: TinySearch, TinyFetch (full pages as markdown, JavaScript included), TinyBrowser for logins and forms, and TinyAgent for multi-step jobs. Search and Fetch are free.



    Source link

    Related Posts