Daily intelligence brief
FinEvo-Bench tests whether retained experience improves later professional work without increasing compliance failures—and finds that structured skills can beat accumulated memory.
- Report date
- Aug 10, 2026
- Status
- published
Memory Has to Earn Its Keep
Persistent memory is often treated as an obvious upgrade: if an agent can retain more of its experience, it should perform better the next time. That assumption skips the more useful questions. Which experience deserves to survive? Does it improve later work? And can performance rise without compliance deteriorating at the same time?
FinEvo-Bench: A Longitudinal Benchmark for Self-Evolving Agents in Professional Financial Workflows turns those questions into an evaluation design. Rather than scoring an agent once, the benchmark follows repeated professional procedures as retained experience accumulates, compares each evolving system with a state-reset control, and measures task quality separately from compliance failures.
The paper was submitted to arXiv on August 6 at 15:14:54 UTC and appeared in the August 7 cs.AI batch. It is included through Keelbase Signal's August 10 catch-up review; it is not presented as a new weekend publication. The August 8–10 scan windows produced no separate qualifying event.
Improvement has to survive a reset control
FinEvo-Bench contains 120 tasks grouped into 20 recurring professional-finance scenes. The tasks cover workflows such as financial analysis, investment research, risk assessment and compliance-sensitive reporting. Each scene unfolds as a sequence, allowing an agent's retained experience from earlier encounters to affect later ones.
The authors evaluate four self-evolving agent scaffolds while holding the Qwen3.7-Max backbone model fixed. For each scaffold, an evolving condition is paired with a state-reset control that receives the same interleaved task stream but does not preserve the accumulated memory or skills between tasks. This matters because a rising score over time can otherwise be confused with differences in task order, model choice or harness design.
Across the four scaffolds, the paper reports that retained experience increased task scores by 9.33 to 19.37 points and reduced compliance issues by 0.12 to 0.44 per task relative to the reset controls. Gains were larger on later encounters with the same professional procedure, and every measured capability dimension improved. The gains were not uniform: report structure improved more readily than evidence use and conclusion quality.
The evaluation still depends substantially on automated scoring. To test that dependency, the authors compare the rubric judge with a financial expert across 120 outputs and report an absolute-agreement intraclass correlation coefficient of 0.95. That supports consistency with the selected expert on this sample; it does not establish that the rubrics capture every real institutional or regulatory requirement.
Persistent experience should be judged by the quality and compliance of later work, not by the amount of history an agent can recall.
Source: FinEvo-Bench (opens in a new tab), submitted August 6, 2026.
A procedure can be more useful than a recollection
The paper's most useful internal comparison asks what form retained experience should take.
In a Claude Code carrier experiment, the authors compare memory-only, skill-only and combined memory-plus-skill persistence. Skill-only evolution produces the highest reported task score and the fewest compliance issues among those conditions. Rubric-based feedback also outperforms feedback based on reference answers.
That result should not be generalized into a universal claim that memory is harmful or skills are always superior. The carrier comparison uses one system, and implementation details affect what each condition can express. But it sharpens an architectural distinction: a record of what happened is not the same as a reusable procedure for what to do next.
Raw history can preserve examples, errors, local context and obsolete assumptions together. A structured skill compresses selected experience into a procedure that can be inspected, tested and reused. If retained state is expected to guide consequential work, the transformation from experience into procedure—and the evidence that the procedure improves future outcomes—becomes part of governance rather than a convenience feature.
Longitudinal governance needs two scoreboards
FinEvo-Bench is distinct from research that asks whether memory remains authoritative, whether external evidence has been laundered into trusted state, or whether stale history can misdirect tool use. It asks a later-stage question: after a system has decided what to retain, did that retained state create measurable improvement over time?
The separate compliance measure is essential. A self-evolving system could learn shortcuts that improve task scores while violating policy more often. A single aggregate reward could hide that tradeoff. The benchmark instead treats performance and compliance as related but independently visible outcomes.
That does not make the benchmark a production assurance. It uses one backbone model, finance-focused tasks and non-parametric evolution rather than model-weight updates. The memory-versus-skill carrier comparison is limited to Claude Code, and the cross-scene diagnostic covers five scenes in one order. Its compliance rubrics are benchmark constructs, not proof of legal or regulatory compliance in a deployed financial institution.
The operational lesson is narrower and durable: persistent state should have an evidence burden. A system should be able to show not only what it retained, but whether that retention improved later execution, preserved compliance, and remained useful outside the exact episode that produced it.
Sources and limitations
FinEvo-Bench is an author-reported preprint and benchmark, not an independent production evaluation. The catch-up synthesis also considered adjacent work on active error propagation, misleading histories and agentic economies; those papers remain supporting research rather than additional structured records.