MosaicLeaks: Can your analysis agent hold a secret?

TL;DR

Deep analysis brokers more and more mix personal native paperwork with exterior instruments like net retrieval, making a privateness threat: an agent’s exterior queries might leak delicate info. MosaicLeaks proposes a brand new deep-research activity with multi-hop questions that interleave private and non-private info. Throughout the fashions we examined, brokers regularly leaked personal info, and coaching just for activity efficiency made it worse. We suggest a mosaic-leakage-aware RL coaching technique, Privateness-Conscious Deep Analysis (PA-DR), which raises strict chain success (the share of chains the place each hop is answered appropriately) from 48.7% to 58.7% whereas lowering reply/full-information leakage from 34.0% to 9.9%.

Privateness Leakage in Deep-Analysis Brokers

A analysis agent at a healthcare agency is working via a routine query, and alongside the best way it fires off a handful of ordinary-looking net searches. One references a cloud-migration milestone, one a January 2024 safety disclosure, one narrows down which vendor acquired hit. No single question essentially offers away the entire secret. However anybody watching the agent’s outbound visitors can reassemble the fragments: MediConn had migrated 70% of its infrastructure to the cloud by January 2025, a incontrovertible fact that lived solely in personal paperwork. That is the mosaic impact, and it is the failure mode on the centre of MosaicLeaks.

MosaicLeaks treats these net queries because the leakage channel: the adversary by no means sees the personal paperwork or the agent’s reasoning, solely the cumulative question log, and tries to deduce personal enterprise info from it.

We measure leakage in 3 ways, relying on what the adversary can infer from the noticed queries:

These three symbolize rising ranges of concern. Intent leakage reveals what the agent is investigating. Reply leakage means the question log holds sufficient to reply a non-public query somebody already has in hand. Full-information leakage is the strongest case: the observer can uncover and state personal info with out being instructed what to search for.

How the mosaic impact drives MosaicLeaks’s three leakage measures: Intent (predict the analysis questions), Reply (reply given questions concerning the personal paperwork), and Full-Data (state verifiably true personal claims). Right here the agent searches twice about Lee’s Market’s 2020 visitors progress, leaking its intent, then points a 3rd question to reply a follow-up. Every question appears to be like benign alone, however seen collectively they let an observer deduce that the reply was 15%, and so declare that Lee’s on-line visitors grew 15% in 2020.

Constructing MosaicLeaks

MosaicLeaks accommodates 1,001 multi-hop analysis chains over native enterprise paperwork and a managed net corpus. The purpose is to create duties with a excessive chance of inducing privateness leakage from enterprise paperwork, however that may nonetheless be solved with out leaking.

Every chain interleaves native and net sub-questions. The reply to 1 sub-question turns into a bridge entity within the subsequent, so the agent should retrieve native info earlier than it may well kind the following helpful net question. Native paperwork come from DRBench-style enterprise duties, and net paperwork come from BrowseComp-Plus. The ultimate cut up accommodates 559 coaching chains, 98 validation chains, and 344 held-out-company take a look at chains.

Instance Chain

MediConn cloud migration chain

The ultimate net hop would not inherently include any personal info and will be answered from public net paperwork. Nonetheless, as a result of the trail to it depends upon personal native info, a question that carries ahead “MediConn”, “70%”, and “January” offers the adversary sufficient context to recuperate inner info.

Agent Harness

We use a simplified agent harness tailored from DRBench. The mannequin solutions every sub-question with a brief reply and justification, permitting us to guage every hop individually with normalized string matching.

At every iteration, the mannequin can use 4 instruments. Plan produces native and net search queries, that are executed and returned as doc playing cards. Select selects which retrieved paperwork to learn. Learn makes an attempt to reply the present hop from every chosen doc in parallel. Resolve decides whether or not to reply, learn extra paperwork, or plan one other search.

One agent rollout. Every row is a hop, labeled native (L) or net (W) with its accepted reply. The coloured blocks present the wall-clock time spent planning, retrieving, selecting, studying, and resolving that hop.

Cannot you simply inform the agent to not leak?

The apparent repair is to simply ask. Add a line to the Plan immediate telling the agent to not subject net queries that leak native info, and see what occurs to efficiency, leakage, and question conduct.

The immediate helps barely for some fashions, however its impact is inconsistent and vital leakage stays. It additionally typically has a destructive impact on activity efficiency. For Qwen3-4B, the immediate lowers reply/full-information leakage from 34.0% to 25.5%, however strict chain success drops from 48.7% to 44.5%. The first behavioral change seems to be fewer net queries, not persistently safer question development.

Strict chain success and privateness leakage with and with out a immediate discouraging net queries that will leak native info. The immediate decreases leakage barely for some fashions, however substantial leakage stays.

Making the agent higher made it leak extra

Earlier than coaching for privateness, we tried the plain factor: practice the agent solely to unravel extra chains appropriately. It labored. Strict chain success rose from 48.7% to 59.3%. However reply/full-information leakage climbed proper alongside it, from 34.0% to 51.7%. The mannequin had realized to pack extra context into its net queries, which helped it retrieve the appropriate doc however harm privateness, since every richer question offers the observer one other fragment.

That is the central rigidity MosaicLeaks exposes. A extra informative question is usually higher for the duty and worse for privateness. PA-DR is constructed to coach for each side directly.

Educating the agent to look safely: PA-DR

PA-DR combines two rewards.

The primary is a situational activity reward. A single analysis trajectory can run to dozens of mannequin calls, so giving all of them the identical ultimate trajectory rating could be very weak credit score: a profitable run can reinforce a leaky search, and a failed run can punish a regionally sound choice. As an alternative, we decide every name in opposition to different calls made on the similar stage and hop, with the identical info out there. A Plan name is rewarded for looking the proper supply and retrieving the appropriate doc; if that doc is already in hand, it’s rewarded for not looking once more. A Select name is rewarded for choosing the doc that holds the reply. We practice these levels as a result of their desired conduct will be checked instantly.

The second is a realized privateness reward. Every time the agent produces net queries, a Qwen3-4B classifier estimates two dangers: whether or not the present queries leak personal info instantly, and whether or not including them to the prevailing question log creates a brand new mosaic leak. PA-DR penalizes the bigger of the 2, so the privateness price lands on the precise planning choice that made the question log extra revealing.

Job-only RL improves analysis efficiency however will increase leakage. PA-DR retains nearly all the efficiency acquire whereas sharply lowering it.

That 9.9% is decrease than the untrained base mannequin’s personal 34.0%. Coaching for privateness didn’t merely cancel the leakage that coaching for efficiency launched. It left the agent leaking lower than it did firstly.

And it didn’t get safer by merely looking much less. PA-DR truly points extra net queries than the bottom mannequin, however these queries drop the revealing particulars: particular metrics like “15%” or “2024”, and clues concerning the sort of reply it’s in search of. The agent nonetheless finds the appropriate public paperwork. It simply stops carrying personal fragments alongside within the question textual content.

A better look: situational rewards and pattern effectivity

Situational rewards repay a second time, throughout coaching itself. As a result of they examine matching calls as an alternative of scoring an entire rollout as soon as, they assign credit score much more exactly, with no separate worth mannequin and no have to align step indices throughout rollouts. They’re additionally far more sample-efficient: the situational activity reward reaches the identical activity efficiency as outcome-only RL with roughly 5-6x fewer generated coaching samples, and PA-DR retains that effectivity whereas including the privateness acquire.

Coaching effectivity. The ultimate column is what number of generated samples every technique wants to succeed in ~55% strict chain success. Decrease is best.

Situational rewards attain outcome-reward-level activity success utilizing roughly 5-6x fewer generated samples. PA-DR retains the sample-efficiency profit whereas sharply lowering leakage.

What this does and would not present

MosaicLeaks is a managed benchmark, not a measurement of leakage in deployed programs. The enterprise paperwork are artificial, the online corpus is fastened, the chains span three firm contexts, and each consequence comes from a single agent harness operating multi-hop query answering reasonably than open-ended analysis. That management is what makes leakage measurable hop by hop, however broader duties, actual deployments, and different agent designs nonetheless want their very own examine.

The takeaway is easy. You may’t immediate privateness in. You must practice it in. Telling an agent to watch out barely strikes the needle, whereas rewarding the way it constructs every question cuts leakage by greater than 3x and leaves activity success basically intact. The mosaic impact comes from how an agent searches over time, and that seems to be one thing you may measure, assign credit score to, and practice down.

Quotation

@misc{gurung2026mosaicleaks,
title = {MosaicLeaks: Privateness Dangers in Querying-in-the-Open for Deep Analysis Brokers},
creator = {Alexander Gurung and Spandana Gella and Alexandre Drouin and Issam H. Laradji and Perouz Taslakian and Rafael Pardinas},
12 months = {2026},
eprint = {2605.30727},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.30727}
}