A Fireplace Chat with Cat and Thariq from the Claude Code crew
twenty first July 2026
Earlier this month I hosted a hearth chat session on the AI Engineer World’s Truthful with Cat Wu and Thariq Shihipar from Anthropic’s Claude Code crew. We talked about Claude Code, Claude Tag, Fable, coding agent safety, evals, software design, and the way Anthropic use these instruments themselves.
The complete video of the session is now out there on YouTube. Under is an edited copy of the transcript, with further hyperlinks and my very own bolded highlights.
A number of top-level notes in the event you don’t wish to watch the video or wade by way of the entire transcript:
Claude Tag (Claude’s new collaborative Slack integration) now lands 65% of the product engineering PRs for the Claude Code crew.
Claude Code ships options to Anthropic workers first, and solely ships the options that reveal consumer retention with that cohort
Crucial modifications to Claude Code are nonetheless reviewed manually, however the crew more and more depends on automated code evaluate for the “outer layers” of the product.
Including examples to a system immediate is now not finest observe for fashions like Fable 5 and even Opus 4.8. The Claude Code system immediate just lately shriveled by 80%.
Likewise, lists of “don’t do X and don’t do Y” can scale back the standard of outcomes from the newest fashions.
Dogfooding inside Anthropic is named “ant fooding”.
Anthropic actually consider of their auto mode, and see that as an enabling expertise for Claude Tag.
Thariq advises offsetting coding-agent-induced Deep Blue by “being extra bold” with the work you tackle.
Fable is competent at modifying video, and Thariq used it to edit its personal launch video.
Anthropic’s tradition of working (internally) in public is vital to their success, as demonstrated by the way in which they use Claude Tag of their public Slack Channels.
How has what you do day-to-day modified up to now 12 months?
1:05
Simon: Claude Code got here out in February of final 12 months — it’s beneath a 12 months and a half previous, and it was initially only a bullet level on the Claude Sonnet 3.7 launch. How has what you do on a day-to-day foundation modified up to now 12 months, now that we’ve these coding brokers that truly work for us?
Cat: I bear in mind once we first got here out with Claude Code and Sonnet 3.7, you’d give it a job and you would need to intently monitor each single little factor it tried to do. I might learn each permission immediate extraordinarily fastidiously. I might ceaselessly say no — no, no, no, did you test this file? Did you test that file? And now it’s been unimaginable with each mannequin technology. I really feel like we’ve all gotten an opportunity to take a step again and delegate much more of the menial implementation to Claude. It’s freed up numerous our time to consider extra inventive work, like: what’s the proper expertise that we needs to be offering to our customers, now that we all know Claude Code can implement numerous it? And now with Fable it’s a completely completely different step change enchancment. We see for lots of our use circumstances you can really one-shot a ton of options with Fable now.
Thariq: I bear in mind the primary textual content I bought about Claude Code. One in every of my finest pals was like, “You could go strive Claude Code.” It was about when Opus 4 got here out, and I attempted it and I used to be like, “Oh, shit. I must work at Anthropic now.” And that was Opus 4 — nice mannequin, however you had been studying permission prompts. It’s sort of loopy how a lot amnesia we’ve, the place I’m like, oh, auto mode has at all times been right here, proper? I don’t even bear in mind urgent sure and permit. For me, the massive factor I’m attempting to push myself on is that we’ve to do greater high quality work than we’ve ever performed earlier than. The outputs are extremely prime quality. I’ve been utilizing it to edit movies a bunch, and I’m like, okay, it has to satisfy the very exacting calls for of our model crew in a few hours or we simply can’t do it. That’s how I’m attempting to shift with Fable: the most effective work we’ve ever performed, sooner than we’ve ever performed it earlier than.
What piece of typical software program engineering now not holds?
3:39
Simon: What’s a chunk of typical software program engineering that was true a 12 months in the past that you just don’t assume holds anymore on this new world?
Cat: One of many largest shifts we’re seeing within the eng ability set: two years in the past it was fairly typical for a product supervisor to go discuss to a bunch of shoppers, align over the course of six months with cross-functional groups on some PRD, and write a radical spec on precisely how we’ll implement this earlier than the primary line of code will get written. Now issues are utterly turned the alternative approach. For lots of engineers, the push I might give to people within the room is to develop extra of your corporation sense and product sense on what it’s we must always construct, as a result of the timeline between having an thought and constructing it’s so a lot shorter — it’s down from six to 12 months to possibly even per week. Which means all of us must have higher style on what’s price constructing, what is going to really inflect the companies we’re engaged on. So it’s a rise in worth on product style and enterprise sense, and a bit decrease on execution in most product domains. After all, for infra there’s nonetheless a really heavy emphasis on ensuring all the small print are proper.
Thariq: For me, it’s that rewrites at the moment are good.
Simon: The worst factor you would do is now really wonderful!
Thariq: Precisely. All of the Legendary Man-Month stuff — by no means rewrite — I’m pro-rewriting now. If in case you have check suite — and I feel the rewrite really forces you to ensure you have check suite — however I feel what folks undercount is {that a} codebase is a spec, and possibly it’s the one copy of the spec that you’ve got, as a result of nobody is aware of each branching a part of the codebase. You’ll be able to take this as an artifact and distill it or create different variations of it. We rewrote Bun in Rust and it really works nice — it’s reside for me proper now.
Simon: You’re not transport Claude Code on Bun-in-Rust but, proper?
Thariq: Internally we’ve.
(Really it appears like Anthropic began transport Claude Code on Bun-in-Rust to everybody on June seventeenth.)
What sort of issues are non-engineers doing with Claude Tag?
6:36
Simon: The opposite huge launch just lately was Claude Tag — that’s what, per week previous now, a minimum of for the remainder of us. I perceive it’s getting used at Anthropic by non-engineers a terrific deal. What sort of issues are non-engineers doing with Claude Tag?
Cat: Claude Tag is a Claude that lives in your crew’s collaboration instruments. We launched it final week inside Slack. The factor that’s completely different about Claude Tag is it’s multiplayer by default. When you add Claude Tag to a Slack channel, you possibly can chime in, your teammates can chime in, and you may collaborate collectively on the PR. The opposite huge distinction is that it’s proactive as an alternative of reactive. You’ll be able to inform Claude Tag, “Hey, monitor each bug report on this channel, put up a PR to repair it, and tag the engineer who final touched this a part of the codebase,” and it’ll do it for the lifetime of the channel with out you having to manually tag it in. And the third huge shift is that we’ve added crew reminiscence into this. In the event you inform Claude Tag your preferences within the channel, it’ll bear in mind them for each future publish. In the event you at all times need it to debug outages however you don’t need it to debug warnings, simply inform it that in pure language within the channel and it’ll bear in mind it for you and everybody else in your crew.
Internally, we see Claude Tag because the evolution of Claude Code. We see this as a big shift in how we work internally. Claude Tag at the moment lands 65% of our product eng PRs.
Simon: For all of Anthropic, or simply for Claude Code?
Cat: That is only for our product engineering crew — our inner model of Claude Tag lands 65% of our product PRs proper now. And it is a big shift; that is greater than 50% of our PRs. The best way we see folks cut up work between Claude Code and Claude Tag is: Claude Code remains to be the most effective place to your most complicated duties, if you’re interactively iterating with the agent. However Claude Tag is nice for having it work proactively in your behalf, so that you now not must manually kick off Claude Code for all of the bug stories that come up for options you’re engaged on.
Thariq: And for non-coding circumstances: for instance, earlier than this discuss we requested Claude Tag, “Hey, when is Fable releasing?” We needed to ensure we’d line it up with the announcement. Claude Tag would search our Slack and take a look at who’s been saying what. As a search engine to your firm, it’s actually priceless. It has all of the context to your product, so you possibly can ask it metrics-related questions — usually if you’re making selections you need them knowledgeable by what the metrics say, so that you hook it as much as your occasion retailer. I’ve seen our advertising and marketing crew do issues like, “Hey, inform me about this characteristic.” They’re not programmers, however Claude is a programmer — it could actually clone the codebase and say, “That is the characteristic, that is what it appears like, it is a recording of me utilizing the characteristic.” It allows a complete huge number of issues, and I feel we’re nonetheless early in figuring that out.
Claude Tag because the crew collaborative layer
10:06
Simon: One of many issues I’ve had with coding brokers is that I get learn how to use them as a person, however I’m probably not clear on learn how to use them in a crew setting. It feels like Claude Tag is your present reply to that crew collaborative layer for these items.
Cat: Precisely. And a big proportion of our classes are literally multiplayer proper now. Perhaps I say, “Hey, I feel we must always implement this new characteristic in Cowork,” and I’ll tag in Claude Tag to do a primary cross at it. Then I’ll inform Claude Tag, “Share a recording of your remaining implementation,” and I’ll tag in design to have a look. They’ll nudge it, then cross it on to eng to take it to the end line and get it out to prod. It’s been this very fluid expertise. We’re nonetheless attempting to iron out what the social dynamics are for steering the identical session, however we’ve discovered that individuals simply observe how others use it and comply with these social norms — it’s been fairly intuitive for us to combine Claude Tag into our groups.
Thariq: It’s nice for educating folks, and in addition for lowering slop, as a result of the truth that everyone seems to be seeing you employ Claude collectively type of ranges up how you employ Claude as properly.
This jogged my memory of how Midjourney solved the problem of educating folks superior picture prompting by implementing prompting in public of their Discord channels.
How do you determine which options are price constructing when constructing is a lot cheaper?
11:41
One thing I’ve discovered actually arduous myself is understanding when a characteristic is price transport now that the price of really constructing options has dropped a lot.
Simon: How do you take care of the toughest downside in all of engineering — prioritization? How do you determine which options are price constructing and transport when constructing a characteristic is a lot extra cheap now?
Cat: That is the arduous factor. There are a number of methods we strategy it. One is we dogfood our merchandise each single day. At any time when there’s one thing we wish to have the ability to do in our merchandise that we’re not capable of, as an alternative of discovering a unique answer we repair our product so it could actually assist that case. Now we have a really heavy dogfooding tradition internally. Earlier than we share our merchandise with everybody on this planet, we share them with everybody inside Anthropic, and with some early clients who give us very sincere suggestions about it — the extra brutal the higher — and we iterate till folks find it irresistible. Now we have an inner bar for the variety of energetic customers and the quantity of retention a characteristic has to have earlier than we share it with the world. As a result of this bar could be very clear, each engineer is aware of what they’re attempting to hit. I feel this additionally ranges up our polish, as a result of if the characteristic isn’t polished, folks will churn — after which we shouldn’t ship that characteristic.
Utilizing inner user-retention to determine if a characteristic ought to ship makes a complete lot of sense to me.
Do you might have an instance of a characteristic which shocked you?
12:54
Simon: Do you might have an instance of a characteristic which shocked you? You rolled it out and the engagement was off the charts — one thing unlikely to be shipped that changed into an actual product factor.
Cat: I do have one. A number of people on our crew love distant management. Distant management enables you to use your cell gadget, or Claude within the internet browser, to connect with an area Claude Code session operating in your CLI. I by no means have this want, as a result of I simply kick off the duty straight on cell and it runs in a cloud session with out utilizing my native setting — I feel as a result of I’m doing very straightforward coding duties. It was one thing I didn’t completely perceive; I used to be like, hey, folks ought to simply arrange distant dev environments. However in observe, as soon as we rolled out distant management, so many individuals I discuss to informed me that what they do each night time is plug their laptop computer into an influence charger, open a bunch of distant management classes, lock the display, after which use their cell phone from their sofa to manage Claude Code. So this has develop into a circulate we’re now leaning into that I didn’t initially get — however now I do.
Does a human evaluate each line of manufacturing code in Claude Code?
14:20
One of many over-arching themes of the convention was evaluate: how a lot consideration to folks spend to reviewing code written for them by coding brokers. I used to be very eager to listen to the Claude Code crew’s tackle this!
Simon: How does code evaluate work? Does a human being evaluate each line of manufacturing code that makes it into Claude Code? And if not, what are you doing — how do you retain the standard up?
Thariq: It varies on the duty quite a bit. For vital areas we’ve code house owners. The system immediate is an instance the place we’ve a code proprietor — you really want to get their approval.
Simon: So the code proprietor is straight liable for the standard of that space of the code.
Thariq: That’s proper.
Cat: And they should approve any PR that touches it.
Thariq: Now we have our code evaluate GitHub bot evaluate every little thing — that goes on each PR, and infrequently it’s doing the majority of the evaluate. One thing I’ve seen on the crew is that for extra complicated PRs you would possibly make an artifact to elucidate the PR in order that different folks can then evaluate. And we make investments quite a bit into verification, CI/CD, issues like that, to ensure that any time something fails we’ve a check. Now we have a very strong setting the place Claude can management Claude Code and check it. So there’s a multi-pronged strategy to code evaluate.
Cat: Generally, we are attempting to maneuver to a world the place people don’t must be within the loop. For probably the most vital modifications to the core of Claude Code, and the cores of different merchandise, there’s at all times a code proprietor and so they do manually evaluate all of the modifications. However more and more, for the modifications on the outer layers, we even have Claude code evaluate totally evaluate these. That sounds fairly scary, however we’ve had a six-plus-month-long course of to get right here, and there are child steps that you just take to construct up belief with code evaluate. At first we had human evaluate for every little thing, after which more and more we might say, okay, for code modifications that contact these recordsdata, code evaluate is catching 100% of the problems there — so we really don’t want a human manually reviewing these. And when we’ve incident evaluate, we take a look at the PRs that triggered the incident and say, okay, how will we replace code evaluate to catch that? — and we take these PRs and add them to an eval set to ensure our future modifications to code evaluate by no means regress that metric. Eradicating people from the code evaluate loop is an enormous step ahead. It might probably sound scary, and it’s not one thing you are able to do in a single day, however it’s one thing you are able to do by way of many months of funding within the infrastructure to provide the confidence that code evaluate is catching every little thing you care about.
So the important thing appears to be continually iterating on the automated evaluate programs themselves, with a view to construct belief in them over time.
How does a brand new mannequin have an effect on your instinct for what it could actually and might’t do?
17:20
We bought deep into evals—one other sizzling matter all through the broader convention.
Simon: I do know that Opus 4.8, if I ask it to construct me a JSON endpoint that runs a SQL question and outputs JSON, is simply going to get it proper — that’s not one thing I’ve to evaluate intently. However then a brand new mannequin comes alongside and I don’t know learn how to construct belief in Fable shortly, that it’s not going to mess issues up that Opus didn’t. How does the brand new mannequin have an effect on your instinct for what it could actually do and what it could actually’t do?
Cat: The primary purpose we’re increase this eval base over time is in order that new fashions generally is a drop-in alternative. When we’ve a brand new mannequin, we run the entire eval set and ensure that, for instance, Fable is strictly higher than Opus 4.8 — and that provides us the boldness to drop it in.
Simon: Are these mannequin evals for Anthropic as a complete, or Claude Code team-specific?
Cat: Now we have each. Now we have evals on our crew, and we run code evaluate throughout each repo inside Anthropic, so we’ve evals for that. And for issues like auto mode, we not solely have evals throughout each consumer inside Anthropic — we’ve additionally commissioned a number of exterior testers to purple crew it, to create environments with immediate injections and malicious inputs, and ensure that auto mode doesn’t let any of these cross.
How do you construct confidence {that a} system immediate tweak leads to higher output?
18:41
Simon: I wish to know if the system immediate enchancment I made really improved the product — that’s probably the most primary type of product-specific eval, and I nonetheless don’t have a terrific really feel for the way to try this. Is that one thing you’re doing such that you’ve got full confidence {that a} tweak you’ve made to the system immediate leads to higher output?
Cat: We don’t have full confidence, however we do quite a bit to ensure that we don’t regress efficiency. The start line is a collection of exterior evals that we belief, and we complement that with an excellent bigger suite of inner evals that we belief. To begin, we primarily optimize for functionality: given an entire definition of a job and the complete codebase, does Claude make the fitting selections, totally repair the bugs, and cross all of the exams? That’s the start line and the factor we optimize for, as a result of it’s most straight what customers need. However there are numerous behaviors that affect how customers really feel after they work with Claude Code. For instance, folks actually don’t prefer it when Claude Code says it’s time to fall asleep. Or folks actually don’t prefer it when it says, “Hey, I completed two out of 5 elements — would you like me to proceed?” Sure, please proceed. So we’re increase a set of behavioral evals to catch these. And as we get consumer suggestions — please be loud with us about your consumer suggestions — we rank the precedence points and go down one after the other and construct evals for every of them. It’s not 100% protection, however it’s a precedence for us to extend the protection.
How a lot interplay is there between the Claude Code crew and the mannequin coaching groups?
20:21
Simon: How a lot interplay is there between the Claude Code crew and the groups at Anthropic who’re coaching the fashions within the first place? Is that fairly a detailed collaboration?
Cat: Throughout Anthropic, all of us work fairly intently collectively. We meet usually to speak about what we count on the following technology of fashions to have the ability to do. Our analysis crew has additionally been wonderful about displaying this publicly — we frequently discuss in our weblog posts about how we’re concentrating on ever-increasing longer-horizon work, and the way we practice Claude itself to be sincere, innocent, and useful. We additionally put numerous effort into ensuring it’s aligned together with your intent, even when your intent is expressed in a fuzzy approach. After all, strive your finest to be particular about what you need, so Claude has all of the context — however even if you’re not particular, we train Claude to make good assumptions. It’s been a productive partnership.
The system immediate has been lowered by 80% — what have you ever been capable of drop?
21:24
So many helpful prompting suggestions on this part!
Simon: Thariq, you talked about this morning that the system immediate for Claude Code has been lowered by 80% due to Claude Fable. Are you able to go into a bit of extra element? What sort of issues have you ever been capable of drop?
Thariq: It wasn’t simply Fable — it was Opus 4.8 as properly, and going ahead, future fashions. Now we have completely different system prompts for various fashions now. One of many patterns we noticed is that we had been over-constraining Claude. The preliminary, possibly Opus 4-ish fashions needed numerous examples, and eradicating examples was extraordinarily useful, as a result of it was simply extra inventive than the examples we gave it.
Simon: That’s actually fascinating, as a result of one of many high prompting suggestions I give folks is: give it examples. If that’s now not true, that sort of breaks my prompting mannequin a bit of bit.
Thariq: Similar right here — I used to be shocked to listen to that. I feel now it’s extra concerning the form of what you give it — the instruments you give to Claude, your system immediate, issues like that. The opposite factor we did is attempt to give it extra context and fewer “don’t do that” directions, as a result of that’s a really robust impulse for Claude, and particularly if it conflicts with consumer directions afterward, that may be extraordinarily complicated to Claude — “I’ve bought this ability that claims this and the system immediate says this.” So we attempt to have fewer arduous constraints, extra context, and fewer directions general. It’s positively a science — it took a bunch of evals to construct.
Cat: Generally, if you’re prompting these fashions, it is best to at all times assume: are there edge circumstances to the instruction that I’m giving it? After we went again and reviewed all of the directions within the Claude Code system immediate, we discovered a number of circumstances the place sure, this assertion is 90% true, however there’s an actual 10% of circumstances the place it’s not true. We didn’t wish to constrain the mannequin, or confuse it into considering it ought to at all times do that. One good instance is verification. Everybody right here desires Claude to confirm its work, and we had some directions within the immediate that stated: in the event you make a front-end change, at all times confirm. However there’s a restrict to it. If it’s altering copy from one string to a different string, and the consumer says “simply make a fast repair and replace the check,” possibly you don’t wish to confirm. So we’ve adjusted our wording from “at all times confirm, confirm, confirm” to one thing like: more often than not if you’re doing front-end work you possibly can’t totally perceive the expertise by hitting the backend endpoints, so if you make bigger modifications to the consumer expertise, please run the app domestically. And actually, that instruction in all probability isn’t even good both, as a result of what’s a big change? Perhaps it ought to check small modifications too. Generally, everytime you give a immediate to the mannequin, it is best to take into consideration the methods during which it may very well be misinterpreted by a well-intentioned human, with a view to higher perceive how the mannequin would possibly interpret it — and soften the immediate in order that it’s really 100% correct, since you’re giving this immediate to the mannequin 100% of the time.
Simon: What’s fascinating about that’s you’re counting on the mannequin’s judgment — and that’s bought to be an Opus/Fable-level factor. Fashions a 12 months in the past didn’t have the extent of judgment essential to determine whether or not they had been going to check a change or not. However that does break down in the event you’re constructing for a variety of fashions and attempting to run the cheaper fashions for cheaper duties.
Cat: We even have a unique system immediate per mannequin now, for this very purpose. It’s solely our most frontier fashions which have this 80% token lower — the older fashions nonetheless have the complete system immediate.
Simon: Do you assume Fable and Opus are good sufficient to immediate Haiku with extra particulars, as a result of they perceive that Haiku has much less judgment, much less style?
Cat: We haven’t been capable of eval it — we don’t have any arduous information to indicate it.
Thariq: There’s a troublesome factor with smaller fashions typically, as a result of typically the bigger fashions may be extra token-efficient on a tough downside than the smaller fashions. So there’s a little bit of instinct to construct there — typically you actually simply need frontier intelligence nearly on a regular basis. The Pareto curve shifts, and it’s arduous to seek out.
Simon: A 12 months in the past I didn’t belief a mannequin to put in writing a immediate. At the moment the nice fashions are superb at prompting — numerous my prompts are written by fashions, which feels absurd however works very well. What helped me come to phrases with that was fascinated by subagents, that are fully a couple of Claude mannequin organising a immediate for one more Claude mannequin.
Thariq: Workflows are literally a very good instance of this, as a result of it’s Claude not simply prompting a single subagent, however prompting the orchestration of many subagents, and every one in every of them will get a really detailed immediate. It’s nearly a degree above simply spawning a subagent. I’ve additionally been utilizing it on my private machine, giving it the Gemini API and saying: right here, generate photographs. It’s approach much less lazy than I’m at prompting a picture mannequin. It’s simply Claude prompting Claude all the way in which down.
Cat: I feel Claude additionally wrote the immediate for the workflow software.
Simon: I’ve learn that immediate — it’s immediate. That’s really a frustration I’ve with Anthropic typically: you publish the prompts for Claude Chat, however you don’t embrace the software prompts and the Claude Code prompts. I nonetheless need to run a proxy to intercept them. I might find it irresistible if the Claude Code prompts had been intentionally revealed — they’re the documentation. They’re how what the software can do and the way it works.
Cat: I’ll write down that characteristic request. I’ll have Claude Tag do it.
Fascinating to notice that OpenAI’s prompting finest practices for GPT-5.6 contains comparable recommendation for his or her newest fashions:
Favor leaner prompts
Eradicating repeated directions and examples and simplifying software descriptions can enhance job efficiency and token effectivity. In a pattern of inner coding-agent eval runs, configurations with leaner system prompts improved analysis scores by roughly 10–15% whereas lowering whole tokens by 41–66% and price by 33–67%.
What’s your bar for introducing a brand new software?
28:06
Simon: Claude Code is principally an enormous bag of instruments. What’s your bar for introducing a brand new software? How do you determine when it’s price doing that further engineering at that degree?
Cat: Do you wish to take it? You launched among the best instruments we’ve.
Thariq: My profession peaked after I launched the ask consumer query software. It’s actually arduous. Particularly for some instruments — ask consumer query is Claude’s software to ask you — so it’s arduous to eval, and typically it’s extra of a consumer desire factor. Again then we had fewer evals, so it was very dogfooding based mostly — or “ant fooding,” our ant model of that. However general we’ve been attempting to development in direction of fewer instruments. The final set of instruments we launched was the duty software, I feel — and we attempt to give Claude extra common variations to do issues.
What’s the newest evolution of your file modifying software?
29:03
I’ve a long-running fascination with file modifying instruments—they had been the topic of the previous Aider code modifying leaderboard, and I’ve watched with curiosity as they’ve developed in several coding brokers from search-and-replace based mostly to line-number-based to extra sophisticated patterns.
The Claude API docs describe a textual content modifying software that’s advisable for constructing in opposition to the API, however Claude Code appears to make use of barely completely different approaches right here.
Simon: One of the crucial fascinating instruments is the file modifying software — you possibly can have file modifying as a software, or you possibly can inform it to make use of sed and grep and do issues that approach. What’s the newest evolution of your file modifying software?
Thariq: We nonetheless have one, however for instance we eliminated our grep and different search instruments — glob instruments — in favor of native bash. Like I stated in my discuss earlier, the fashions are sort of extra of a biology than a physics, and power design particularly is kind of arduous. I’m undecided if Cat disagrees and thinks there’s a science to the eval of it, however I feel software design is extra of an artwork, possibly — or a biology.
Cat: I largely agree, however usually as we introduce extra instruments, we attempt to maintain the cardinality fairly low and ensure that each software we add has a definite operate from each different software, in order that Claude can very simply distinguish when to name every. For file edit, the rationale we’ve it is usually because we are able to render it. We present folks when Claude makes a file change, and there’s this good devoted UI that claims: do you approve this edit to this file? The explanation we had a devoted file edit software was in order that we may deterministically know that Claude was making a file change, so we may present folks this good UI. A number of new customers onboarding nonetheless actually like this expertise, so we’ve stored it round. However for lots of us who’re on auto mode proper now — hopefully you’re not on YOLO mode — I don’t assume it really issues, and we may in all probability simply take away file edit and be completely wonderful.
What’s the recommendation inside Anthropic for safely operating Claude Code?
30:58
It’s the immediate injection query! Who higher than Anthropic workers to elucidate how Anthropic sees the danger of immediate injection assaults inflicting their Claude Code cases to run amok?
It seems they actually belief their auto mode—and see that because the characteristic that enabled Claude Tag.
Simon: Let’s speak about security and safety. I’m deeply conscious of the dangers of immediate injection, and there are such a lot of dangerous issues that may occur if any individual else tells my Claude Code what to do. I nonetheless principally run Claude Code in YOLO mode and really feel extremely responsible about it. What’s the recommendation inside Anthropic for safely operating Claude Code?
Cat: Why not auto mode?
Simon: I’m beginning to use auto mode, however I don’t perceive it sufficient to get how protected it’s. As of possibly three weeks in the past, I’m defaulting to auto mode.
Cat: Broadly inside Anthropic, nearly each single particular person makes use of auto mode. It’s the easiest way to do long-running work in Claude Code whereas being protected. We’ve performed intensive bashing. Now we have 1000’s of evals. We’ve commissioned many purple teamers to create adversarial environments with a view to trick Claude Code into doing dangerous actions, and we’ve mitigated each single concern that they discovered. We’re going to publish some evals within the coming weeks, however we’ve just about mitigated each assault.
Simon: That may be a huge declare.
Cat: We’ll share the evals for it so people can assess, however we’ve been extraordinarily diligent about figuring out all of the methods during which Claude would possibly mess up after which updating auto mode to counter it. It doesn’t catch 100% of issues — that might be approach too robust a declare. However for the primary classes of dangers that we’re involved about, like immediate injection and information exfiltration, the dangers are far decrease than the typical human reviewer.
I’m very a lot trying ahead to studying extra about their evals and strategy to verifying auto mode.
Thariq: A bit of on how auto mode works — it’s helpful to construct this psychological mannequin. At any time when Claude is doing a flip, or a bash name, there’s a Sonnet classifier that’s judging the software name and in addition the context of the dialog — your instruction. There are some issues round permissions which can be dependent in your request: you don’t wish to give git push permissions on a regular basis, however in the event you say “push this to GitHub,” you need it to do it — and in the event you say “don’t push,” you need it to disclaim it. Auto mode will try this. That exact factor occurs to me quite a bit, the place Claude tried to do one thing as a result of it’s very useful and proactive, and auto mode noticed “don’t do that” and surfaced it. So it’s good on the dynamic permissions that you just your self give contained in the immediate, which I feel is admittedly vital. It additionally works properly with our sandboxing infrastructure, as a result of sandboxing is a kind of issues the place there are such a lot of completely different edge circumstances that it’s arduous for us to deterministically comply with them. Now we have a sandbox, and when one thing wants to flee the sandbox — like a community request — auto mode can take a look at that request and ask: does this make sense? — and permit it.
Simon: I hadn’t realized auto mode is interacting with the networking sandbox as properly.
Cat: It interacts with any permission immediate the consumer would in any other case see.
Simon: How previous is auto mode? As a characteristic I had entry to, it’s solely a few months previous, proper?
(It was first made out there to the general public on March twenty fourth.)
Cat: We’ve been utilizing it inside Anthropic since January, so we’ve been hardening it for fairly some time. Anthropic is extraordinarily targeted on security and safety, and we’ve been working broadly throughout our alignment and safeguards groups to allow the rollout internally, construct out these evals, and make auto mode much more strong earlier than sharing it with the world.
Thariq: That is additionally the rationale Claude Tag is so good — Claude Tag makes use of auto mode. I’ve heard numerous build-versus-buy questions on a Slackbot, and I’m like: please, you in all probability shouldn’t construct your individual AI Slackbot. There are such a lot of assault vectors. You could have a suggestions channel that customers can publish suggestions into, and now your bot is studying it. The work we’ve put in with auto mode — and we’ve a common Swiss cheese protection for safety; we additionally RL in opposition to these items — I feel that is actually what makes Claude Tag work. It really works seamlessly together with your permissions, and also you don’t wish to be immediate injected in your Slack.
Are there extra safety issues within the pipeline past auto mode?
35:54
Simon: Are there any extra safety issues within the pipeline that transcend auto mode?
Thariq: I feel we’re very safe. With Claude Tag you possibly can provision your individual credentials for Claude, so it doesn’t must act in your behalf — you possibly can have Claude as an identification, and that additionally makes it simpler to audit and examine what Claude is doing.
Simon: As a result of Claude Tag is influenced by anybody who can discuss to it — it’s bought a a lot wider pool of individuals telling it what to do.
Thariq: That’s proper. And naturally we’ve probes as properly with Fable, which is a downstream impact of our security and analysis work. I feel that is the second the place you see Anthropic being an AI security firm actually paying off: we actually need Claude to have the ability to run in an aligned approach over lengthy durations of time, and auto mode must be principally flawless for this to work — it’s all downstream of our being an AI security firm.
Cat: We additionally launched trusted gadgets for the distant management customers on the market who wish to be safer. And for all of our distant environments, we assist credential injection. If you need Claude Code to have the ability to entry Datadog, however you don’t need Claude Code itself to carry the Datadog credential, you possibly can arrange our identification and credential administration system in order that the Datadog credentials are solely usable by the agent however not accessible by the agent — we insert them on the fly when the agent tries to make a Datadog request.
I actually like that credential injection sample, the place Claude Code can entry an API through a proxy and that proxy each audits the request and injects the related API key—so Claude can entry authenticated endpoints with out gaining access to the API credentials itself.
How has the previous 12 months and a half modified how you concentrate on your individual craft?
37:53
Thariq talked a couple of sense of grief introduced on by Fable-class fashions in his keynote within the morning, and we dived additional into that as a part of our dialog. I’ve been calling this Deep Blue.
Simon: Let’s discuss a bit of bit concerning the human component. Lots of people are feeling a way of loss now that a lot of what they thought of to be their function in constructing software program is being subsumed by the fashions. How do you concentrate on that? How has the previous 12 months and a half modified the way in which you concentrate on your individual craft and the worth that you just add?
Thariq: Cat and Boris are such good reminders that it’s important to be extra bold. They’re at all times like: we’re rising so quick, we’ve to be on the sting, we’ve to do the most effective work we are able to. That’s a relentless reminder for me — any time I’m sluggish on one thing, I’m like, okay, can I do it sooner? Can I be extra bold right here? And oftentimes the reply is Claude, as a result of Claude is getting higher as you go — the final time I attempted this, it was with the earlier mannequin. In your level about loss: I feel that is actual. In the event you’re solely attempting to do the identical work you had been doing earlier than LLMs, and now it’s a immediate, it’s, I feel, sort of a tragic feeling. And the way in which you offset that’s by being extra bold. I feel Jared is such instance — he hand-wrote all the Zig code in his Oakland house in a couple of 12 months, barely left his home, and had a lot enjoyable doing that. Now I see him rewrite all of Bun into Rust and he’s having a lot enjoyable doing that — it’s a lot extra bold, and that’s how he offsets it. Typically it’s asking how do I do the larger factor and do extra — I feel success is enjoyable. It’s altering your ambition.
“The best way you offset that’s by being extra bold” neatly captures the place I’ve landed on this concern myself as properly.
Simon: And Cat, what does that appear like from a product administration perspective?
Cat: I really feel just like the product function simply modifications each single month. All of the PMs on our crew are this mixture of engineer, designer, PM — most of them really was once full-time engineers. For us it actually means plugging in at any time when there’s any sort of hole. If we’ve an thought and we didn’t encourage any engineer to go construct it, then we must always simply construct it, put it right into a pocket book, and encourage folks to take it to manufacturing. If the designs look a bit of off, let’s take a web page that’s comparable, do a first-pass design, and tag in somebody who’s very detail-oriented to fill within the gaps. Or if we discover that our crew and product adoption is larger inside the firm, and extra folks must know what’s coming down the pipe for Claude Code, Claude Tag, and Cowork — let’s automate determining our complete launch calendar, let’s automate getting these standing updates asynchronously so we’re not bugging folks, and ensure our updates in our inner announce channels are totally detailed and to the purpose. For us it’s very a lot understanding what the hole is true now between a terrific thought and getting one thing to our clients, and the way will we automate it as a lot as potential.
This displays one thing I’ve seen: when you possibly can produce code a lot sooner, time spent blocked awaiting a choice from another person turns into a way more notable bottleneck. Engineers who could make product selections can transfer a complete lot sooner, and the price of getting a kind of selections unsuitable is far much less prohibitive.
What’s a second when Claude has shocked you?
41:50
Simon: What’s a second when Claude has shocked you? When the mannequin did one thing you didn’t assume it could be capable to do?
Thariq: I’ve posted quite a bit about Claude video modifying, however most just lately I gave a chat on the ACM Agentic convention, and I requested, “Hey guys, do you might have the edited video? I’d like to publish it and share it with my comms crew.” They stated, “Oh, it’s taking so lengthy.” So I requested for the uncooked recordsdata. They despatched me the video of me speaking on stage, the video of the deck, and the audio file, and stated, “Good luck.” I gave this to Claude, together with my HTML deck, and stated, “Hey, are you able to simply edit this collectively?“ And what it does is truthfully unimaginable — I’m able to ship it. It transcribes all the video. It notices that typically the video of my deck is a bit of bizarre — there’s a popup of an auto-update within the center — and it goes, ”Oh, I in all probability shouldn’t use the video of your deck. What I’m going to do is slice it up, determine which slide you’re on, and use the HTML supply as an alternative.” So it shows the HTML supply. Then it’s bought video of me, however I’m solely taking on a small a part of the stage, so it’s cropping dynamically to the place I’m on the stage — and I’m pacing, so it’s monitoring me as I tempo. And it’s transcribing what I’m saying.
Simon: This was Fable, proper?
Thariq: This was Fable, yeah. It was immediate, however it was a one-shot immediate. Then I requested it so as to add some fascinating animations and graphics, and I used to be simply blown away. It does ffmpeg, it does Remotion.
Right here’s Thariq’s video on how he used Fable to edit Fable’s personal launch video, and right here’s that launch video.
What can’t it do but?
43:36
I’m embarrased to confess that I’ve been discovering it fairly arduous to provide you with duties that frontier fashions like Fable 5 and GPT-5.6 are unable to perform.
Cat nonetheless doesn’t price its UX design expertise:
Simon: What can’t it do? What are the issues the place you’re nonetheless dissatisfied — the place you’re ready for Claude Fable 6 to determine it out for you?
Cat: I would like it to have higher design and UX style. It’s now on the level the place if I write out a immediate with an in depth spec of how I need a characteristic to behave, it’ll often behave that approach. However the paddings is likely to be off, or the interface simply isn’t pleasant but. It leans on present finest practices for the way apps are designed, however for frontier AI merchandise, there are such a lot of new interplay experiences that we’ve but to design.
Simon: There’s an Opus aesthetic — you possibly can take a look at one thing and go, “Yeah, that was designed by Opus.” It’d be good if we may transfer past that.
Cat: Yeah. I’m very excited for future fashions to hopefully be interplay design thought companions.
Thariq: What can’t it do? I might like to see it work together extra with the true world. Can it resolve science? Can it orchestrate the experiments? There’s some quantity of coding that goes into that, however there’s additionally this different style of the broader world that it wants.
Which elements of Anthropic’s tradition ought to different corporations steal?
45:11
I figured this is able to make a terrific closing query:
Simon: Which elements of Anthropic’s firm tradition do you assume uniquely assist Anthropic be productive with these instruments, that different corporations ought to steal? What are the cultural hacks folks needs to be adopting from you?
Cat: I’ll share one for Claude Tag. Claude Tag works finest when you might have it in a public channel, and when most of your channels are public. Claude Tag is ready to search throughout all public channels to get as a lot context as potential to provide the highest-accuracy reply — and it’s solely ready to do that if it has entry to every little thing.
Thariq: I discussed this in my keynote, however it’s so vital to me I wish to re-emphasize it. The co-founders say we don’t negotiate in opposition to ourselves, and I feel that is actually vital. You’ll be able to think about trade-offs in your head and discuss your self out of doing one thing bold — or you possibly can simply attempt to do the bold factor. We’re so usually asking: what if we simply did it? Is that this an actual trade-off or not? And if that’s the case, why — the place’s the proof that it’s an actual trade-off, and never simply one thing that sounds cheap? Make the trade-offs present themselves to you. Be as bold as you possibly can.
What’s your favourite absurd factor you’ve constructed with Claude, simply since you may?
46:46
I couldn’t resist throwing on this one as properly.
Simon: What’s one in every of your favourite absurd issues that you just’ve constructed with Claude, simply since you may construct it?
Thariq: I’m engaged on a 2D Avenue Fighter preventing sport with me as a personality — and my pals as properly. It makes use of Claude Code to immediate Gemini — and truthfully the Seedance mannequin is fairly good — to make video animations. It really works nice; it’s so good at prompting, and it could actually confirm the frames to test whether or not an animation was good.
Simon: Is that this Avenue Fighter 2-level 2D sprites you’re producing?
Thariq: Yeah, precisely — 2D sprites. The animation appears wonderful. And it could actually additionally determine hitboxes — it may be like, “Oh, your fist is right here, I’ll draw the JSON hitbox.” It’s unimaginable.
Cat: Mine is rather more easy. I’m an enormous rock climber and numerous my pals climb, so we’ve this little app we constructed with Claude Code the place we log all of the initiatives we’re engaged on. We additionally go outside collectively quite a bit, so we’ve Claude do all this analysis with workflows. Workflows is wonderful — we model it as a coding software, however it’s wonderful for doing deep analysis for journey. I additionally plan our crew offsites, and it’s good at discovering venues that may match all of us. I exploit workflows to analysis all of the climbing locations we’d wish to go to, and what has direct flights from the place all of us are situated. It goes to Mountain Mission and finds all of the climbs at our grade degree. It finds the Airbnb. And I don’t like mountain climbing, so I care quite a bit about it having a really brief strategy — very brief strolling distance from the place the automobile parks to the place the rock really is — and it filters for this. With present apps I’ve to manually click on by way of Mountain Mission, however with this I simply put in all of our preferences and it’s a customized app for us.
Simon: So that you’re principally vibe coding Jira for mountaineering.
Cat: Precisely.
Viewers: Any plans for eval-building instruments and agent observability?
49:23
We had a couple of minutes on the finish for questions from the viewers.
Viewers: Do you might have any near-term plans to construct extra eval instruments for us to construct eval datasets, and extra observability instruments to watch the efficiency of brokers and workflows?
Cat: We’ve thought of constructing eval instruments, however I feel the limiting issue really tends to be that it takes a very long time for patrons to construct actually high-quality evals. So I feel the tooling is much less of the constraint, and extra the ability set of the way you construct a terrific eval. That’s an space the place we’re excited to each make investments internally and hopefully share some finest practices externally.
Viewers: How is reminiscence designed right this moment — and would you progress from recordsdata to a knowledge retailer?
50:08
Viewers (Sai): I’m within the reminiscence and the multiplayer. How is reminiscence being designed right this moment? I assume it’s round recordsdata. And second, have you considered an orthogonal route the place you’d really want an information retailer for these reminiscences, as an alternative of recordsdata, to scale it higher?
Thariq: Proper now for Claude Tag the reminiscence is channel-specific. Each Claude in that channel has a shared reminiscence, and the cases have a session — however the session can contribute again to predominant reminiscence. We do numerous reminiscence analysis, and it may be sort of unintuitive what the fitting method to do reminiscence is. We’re at all times operating reminiscence experiments. The way it works proper now in Claude Tag is a markdown file per channel.
