Dan Shipper and Brandon Gell from Every test the newly released Claude Opus 4.7 model live, evaluating Anthropic's claim that it can handle difficult work with less supervision. # Channel Identity Observations: "LIVE VIBE CHECK: OPUS 4.7 DROPS" ## Explicit Self-Description (Creator Claims) - **Organization name and positioning**: The channel repeatedly identifies as **Every**, framed as "the only subscription you need to stay at the edge of AI." > "If you're here and you're just joining, you should know that every is the only subscription you need to stay at the edge of AI. We do ideas, apps, and training." - **Claimed scope of content**: Three pillars — **ideas, apps, and training**: - *Ideas*: Daily newsletter, "vibe checks" on new AI models ("We do these for every new model when they come out") - *Apps*: A named suite — Spiral/Sparrow (writing), Cora (email), Sparkle (Mac cleaner), Monologue (speech-to-text), Proof (agent-native document editor), Plus Ones (hosted "open claw") - *Training*: "we do trainings, we do live streams and courses for subscribers" - **Claimed methodology**: Skepticism of benchmarks in favor of hands-on testing: > "We don't really like benchmarks. I don't really like benchmarks… I really like getting your hands in the model and using it for the things that you do every day." - **Claimed access status**: The hosts state they normally receive models pre-launch, and this launch was an exception: > "Normally at every week we get access to models before they come out. We did not get access to this before they come out." - **Named format**: The "vibe check" with a defined rating scale: > "Red is terrible, trash model, yellow is okay, green is like I'd probably use this over the other options… and gold is paradigm shift." ## Observable Presentation (Demonstrated Patterns) - **Team-based format**: Multiple hosts appear with stated roles — the lead host, **Brandon ("our COO")**, and **Katie**, who handles writing-focused testing. Katie describes her own workflow with style guides and "tastemaker voice" skills. - **Audience addressed**: AI practitioners — people who write with AI, code/vibe-code, build agents, and run businesses using AI tools. Explicit calls to action target this group: > "If you're here and you're using Opus 4.7 for interesting stuff, please share your vibe checks in the comments." - **Live, improvisational style**: The video demonstrates unpolished, real-time testing — audio mishaps, screen-share confusion, a false GPT-5.5 rumor, and mid-stream revisions of takes: > "It's just because it's different does not actually mean it's bad. That is not my official take." - **Internal business disclosure on-stream**: Tests use Every's actual operations — the P&L, the March investor update, internal Slack "nugget" mining for the newsletter. - **Guest access to industry figures**: An **Anthropic researcher (Alex Albert)** joins mid-stream and discusses model behavior, effort levels, and training tradeoffs — demonstrating channel access to AI lab personnel. - **Repeated subscription promotion**: Multiple full pitch segments for every.co/subscribe ("$30 a month… access to everything that we write and everything that we make"), plus a recurring "stay hydrated" sign-off bit. - **Proprietary testing frameworks demonstrated on-stream**: - The **"vibe slop benchmark"** — a frozen "vibe coded" production codebase (Proof) used to test whether models rewrite like a senior engineer. - Head-to-head model comparisons (4.6 vs. 4.7) for investor updates, to-do apps, article intros, and an "open Claude" build. - **Recurring interpretive themes**: The hosts articulate named concepts across videos, e.g., "the great convergence" (Opus and GPT models converging in behavior) and the "reach test" ("when I go to do my work in an hour… am I going to keep the dial on 4.7?"). ## Claimed vs. Demonstrated Distinction - **Claimed**: Early access to models is normal ("We normally get access to these models before they come out"); the transcript also claims they're "already testing some early models… that we can't tell you about." — these are creator claims, not verifiable in this video. - **Demonstrated**: Live testing capability, a team of hands-on practitioners, real production apps, a defined evaluation methodology, and viewer counts cited on-stream ("about 3,000 live viewers," later "almost 10,000 people"). # Delivery Observation: LIVE VIBE CHECK: OPUS 4.7 DROPS ## Tone - **Conversational, improvisational, comedic.** The speaker treats a model launch like a live event with running jokes: "You always got to wear green on a on a vibe check day. We've got our water here. We're getting hydrated." - **Playfully feigned grievance** toward Anthropic for lack of early access: "Anthropic snubbed us... I have started texting them being like, 'Do you hate me? Do you Do are we breaking up?'" - **Recurring hydration gag** as a sign-off ritual: "please stay hydrated. Really important to keep yourself and your agent liquid cooled" and again at the end: "remember, stay hydrated." ## Vocabulary & Recurring Phrases - **"Vibe check"** is the dominant recurring term, used as both noun and verb ("we're doing an Opus 4.7 vibe check," "give us your own vibe check"). - **Structured rating vocabulary**, defined twice: "Red is terrible, trash model, yellow is okay, green is like I'd probably use this over the other options... and gold is paradigm shift." - **Idiosyncratic coined terms:** "vibe slop benchmark," "burn the ships," "the great convergence," "AI smell." - **Repetition of "slop"** family words: "vibe coded slop code base," "unslopping," "disloppify." - **"The reach test"** as a personal evaluation frame: "when I go to do my work in an hour, what am I going to reach for?" ## Teaching / Testing Habits - **Skepticism of benchmarks, preference for hands-on testing:** "we don't really like benchmarks... I really like getting your hands in the model and using it for the things that you do every day." - **Repeated context-reset for new viewers.** The speaker periodically restates the premise: "for people who are just joining... normally we get access to these models before they come out. This time we were snubbed." This happens at least three times (≈[1138], [1432], [5260]). - **Side-by-side comparison as a method.** Multiple tests are structured as 4.6 vs. 4.7 head-to-heads: "4.6 on the left. 4.7 on the right," "Opus 4.6 on the left and then Opus 4.7 on the right." - **Uses real work as test material** rather than toy prompts: investor updates, P&L analysis, a production code base, newsletter "nuggets" from Slack. - **Delegating to co-presenters** while setting up: "Brandon, do you want to talk a little bit about what you're seeing while I get set up?" ## Communication Patterns - **Heavy stammering and self-restart**, characteristic of unscripted live speech: "We've got We've got a tweet. Uh we We We normally get access to these models." - **Explicit epistemic hedging.** Findings are repeatedly flagged as provisional: "very very very very preliminary," "it's too early to say results," "I don't want to like do it say anything rash." - **Willingness to voice negative early verdicts** while withholding final judgment: "right now I'd probably stick with 4.6," later balanced by "Just because it's different does not actually mean it's bad." - **Frequent audience calls to action:** asking viewers to share ratings in comments, report bugs ("If you're running skills and it's not working, let us know"), and inviting Anthropic staff to join the stream. - **Integrated promotional segments** ("Every is the only subscription you need to stay at the edge of AI") woven between test segments, with app roll-calls repeated nearly verbatim at least twice — a stable pattern. - **Reflexive summarizing.** The speaker repeatedly asks guests to synthesize: "if you were going to summarize all all of what you've learned so far... What's your vibe check?" ## Isolated Moments vs. Stable Patterns - **Stable:** "vibe check" terminology, hydration joke, audience-engagement prompts, benchmark skepticism, side-by-side testing method, the Every promo block. - **Isolated / less repeated:** the Annie Dillard "cover your tracks" reference, the "learning to drive manual versus driving stick shift" analogy (attributed to Katie), and the recurring-but-single-thread "literal vs. reading between the lines" thesis, which builds through the stream but is synthesized only near the end. Its stability as a permanent framing device is uncertain within this single video. # Video Position Record: LIVE VIBE CHECK: OPUS 4.7 DROPS ## Event Context Live-streamed "vibe check" of Anthropic's Opus 4.7 release, hosted by **Dan** (Every), with **Brandon** (COO) and **Katie Parrot** as co-testers. **Alex Albert** (Anthropic research) joins later. Notable: Anthropic did **not** run an early access program this time, so testing was live rather than pre-synthesized. --- ## Key Positions & Claims ### Dan (Host) **On the launch itself** - Every was "snubbed" — no early access. He characterizes this as Anthropic "flying by the seat of their pants" and implies the release "may have been a bit rushed." - Qualification offered: "it's possible that it was just a misunderstanding." **On Anthropic's stated claims** - Dismisses "most capable Opus yet," "long-running tasks with rigor," "follows instructions more precisely" as claims "everyone always says." - Calls the **self-verification of output** claim genuinely new: "a thing that good prompters know how to do has started... to make its way into the models." - On benchmarks: ~10% bump on Sweet Bench Pro, ~7% on Sweet Bench Verified; "pretty good bump... not as good as Mythos." **On benchmark skepticism** - "We don't really like benchmarks... I really like getting your hands in the model and using it for the things that you do every day." **Vibe Slop Benchmark (his test)** - Setup: frozen snapshot of the vibe-coded Proof codebase; tests whether a model can identify what a senior engineer would do and execute a first-principles rewrite. - Finding: Opus 4.7 wrote a **strong plan** — identified the "single authoritative model" invariant, "certainly on par with GPT 5.3/5.4" — and noticed the huge collab.ts file that GPT models missed. - But: "it's not a very courageous rewriter... it knows what to do. It just doesn't want to actually do it." It "chickens out," nibbles at the edges. - Provisional grade: **B / B+**. - Anecdotal benchmark note: "execute end to end" was interpreted literally (executed in Proof rather than executing the plan) — "an interesting thing." **Investor update test** - Opus 4.7 wrote Every's March investor update from the P&L: numbers correct, tone appropriate, "would have saved me a lot of time." "A sense of solidity and reliability... I have not felt with Opus models before." - 4.6, by contrast, "goes too far. It does too much." **On "the great convergence"** - Opus historically "more empathetic"; OpenAI moving GPT toward emotional intelligence; Opus 4.7 becoming more literal — "they're kind of switched spots... It's the great convergence." **Post-Alex-Albert synthesis** - "You should expect all your old prompts to break... the more you can push this model to more explicitly tell it exactly what you want, the better it's going to get." Framed as inference combining Alex's comments with their testing, not settled fact. --- ### Brandon (COO) **Open Claude test (4.6 vs 4.7)** - Same prompt to both: build an "open Claude" from his memories/usage. - 4.6: proper soul.md/user.md files, named it "Koa," memory seeds, cron jobs. - 4.7: no user/soul files, shoved everything into an agent file; created skills that already exist; made unexplained skills (e.g., AI-writing linguistic-pattern detection); organized knowledge by category rather than by day. - Verdict: "right now I'd probably stick with 4.6 for making my open Claude like this." - Qualification: "I don't want to say anything rash" and "with new models your old prompting style sometimes needs to change." **P&L analysis test** - 4.7 got all numbers right but was "lazier"; gave surface comparisons he "didn't need AI to tell me," and **did not find the failed-transactions error that 4.6 found unprompted**. He had to push it to go deeper. - Caveat: "I can't really give a real vibe check... it might just be that the prompt needs to change." - Notes 4.6 is now deprecated for this task. **To-do app test** - Both produced working apps with similar confetti feature and similar AI feature. - 4.6's AI feature (Cmd+Enter expanding a vague to-do into sub-todos) was more useful; 4.7's (Tab to sharpen text in place) was "a much less effective way for me to manage my to-dos." - Verdict: "if I was vibe coding a to-do app, I would use 4.6 for this very, very rough prompt." --- ### Katie Parrot (Writing tests) **Head-to-head intro writing** - 4.6's draft: "a little bit more unpredictable in a good way... doesn't have the same sort of regimented" quality. - 4.7's draft: "more systematic in a way that just doesn't vibe with how I am as a writer. It doesn't feel like me in quite the same way." - Counter-note: "you always see a glimpse here and there of something that you want to steal." **Slack nugget-harvesting test** - Found it resourceful, working around tool-use limits: "how far we've come since the days when models would just give up." - Judgment on what counts as good nuggets was "right." - Flaw: too many **correlative constructions** ("not X but Y") slipped through despite skills designed to catch them; also flagged the word "actually" as an AI tell (her "conspiracy theory" attributes it to YouTube corpus data). **Overall take** - "It needs more explicit direction... a little less confident in just saying 'go write this.'" More cognitive load on her. - **Reach test: "default is falling back to 4.6"** — with the qualification that this is "always open to change" as prompting improves. - Post-Alex: underscores effort levels — "mapping the task... to the effort level it needs is something worth thinking about" (manual vs. automatic transmission analogy). --- ### Alex Albert (Anthropic Research) **On model behavior** - Confirms Opus 4.7 is "a lot more on the mark. It will do what you tell it to do." Traces the literalness/go-for-it tension back to Sonnet 3.7 (too aggressive) → Sonnet 4 (dialed back), describing a "perpetual back and forth" in course-correcting models. - On 4.6's release: there were complaints it "did too much," so prompts shifted; now users need to "dial them back in again or add in those instructions to actually be thorough." **What he loves about 4.7** - **Long-horizon background tasks**: overnight apartment-hunting dashboard example; telling the model "I'm going to bed" let it iterate longer and return more refined output. - **Vision**: larger/higher-res image support; "Where's Waldo"-style detail detection translating to catching misaligned buttons in frontend iteration; strong design replication from screenshots. - **Docs/spreadsheets/slides**: strong on real-world office tasks. - **Pushback**: "it sometimes does push back on you... I can trust the model will push back on me if I'm on the wrong path." **Effort levels (new)** - New **"extra high"** tier between high and max; Claude Code defaults to extra high. - **Max**: "go gangbusters," benchmarks run here. - **High/medium**: "Goldilocks" balance, good for non-coding/chat. - **Agentic work**: stay in the higher range. **Broader claims** - "The rate of improvements are not slowing down anytime soon." - A paradigm shift "from collaborative work to asynchronous work"; users and builders need a mindset shift around effort levels and prompt changes. "We're quickly saturating a lot of that bottom half of tasks." --- ## Aggregate Group Synthesis (Dan, end of stream) - Opus 4.7 is **more literal, more detail-oriented**; fills in fewer gaps than 4.6. - Performs better with explicit, specific instructions; worse with vague 4.6-era prompts (all three testers converged on this independently). - Persistent tension: initial hands-on results favored 4.6 across multiple tasks, but all testers qualified this as provisional, pending prompt adaptation and effort-level tuning. ## Unresolved / Contradictory Threads - Strong benchmarks vs. lukewarm hands-on first impressions — explicitly left unresolved ("provisional," "TLDR vibe check" promised later). - "Snubbed by Anthropic" framing vs. later confirmation that **no one** got early access ("but no one did. No one did."). - GPT-5.5 rumor mid-stream (viewer claim) — checked and declared "a false alarm. GPT-5.5 is not out." - 4.6's deprecation (per Brandon) constrains future head-to-head comparisons. # Speaker Attribution Record: LIVE VIBE CHECK: OPUS 4.7 DROPS ## Video Overview - **Format:** Live-streamed model evaluation ("vibe check") by media company Every - **Subject:** Live testing of Anthropic's Claude Opus 4.7, launched same day - **Context:** Every did **not** receive early access; the stream is an unrehearsed live test ## Speakers & Roles ### Primary Host / Creator - **Dan** (unnamed in transcript but addressed as "Dan" by guests; hosts the stream, hosts the "vibe slop benchmark," writes Every's manifesto and investor updates) - Role: **Creator / Host** — leads the stream, runs coding and investor-update tests, conducts interviews ### Co-Hosts - **Brandon** — identified as **COO** of Every - Role: **Co-host / company executive** - Tests: "open Claude" (open claw) generation, monthly P&L analysis, to-do app builds comparing Opus 4.6 vs 4.7 - **Katie Parrot** — identified by name on stream - Role: **Co-host / writing specialist** - Tests: writing style, "AI smell" in copy, intro head-to-heads, Slack "nugget" harvesting for newsletter; describes her setup with style guides, skills, "tastemaker voice" (citing Jia Tolentino, John Green as influences in her skill, not as speakers) ### Guest - **Alex Albert** — **Anthropic researcher**, "helped to make 4.7" - Role: **Guest / model developer representative** - Claims made: Opus 4.7 is "a lot more on the mark" and literal; new **extra high effort level** between high and max; improved long-horizon/background tasks; expanded vision/large image support; models pushing back on users; personal anecdote of overnight apartment-hunting website build; historical context on Sonnet 3.7 vs Sonnet 4 behavior tuning - Signs off citing his X handle ("Alex Albert, two T's") ## Referenced Third-Party Sources & Claims - **Anthropic announcement (tweet):** Opus 4.7 described as "most capable Opus model yet"; handles long-running tasks with more rigor; follows instructions more precisely; "verifies its own output before reporting back" - **Benchmark figures cited (from announcement):** ~10% bump on **SWE Bench Pro**, ~7% on **SWE Bench Verified** vs Opus 4.6 - **Viewer-sourced claims from chat:** - **Ryan:** Opus 4.7 "interprets prompts more literally and explicitly than 4.6" (host noted this appears in Anthropic's announcement) - **Mark Green:** claimed "GPT-5.5 is out" — later declared a false alarm on stream - **Tibo** (identified as Codex lead at OpenAI): tweet "Feeling codexy today," interpreted as possible upcoming Codex release - **Anthropic (via host):** "They did not run an early access program for this" — host states Anthropic confirmed this directly - **Comparison models referenced:** GPT 5.3, 5.4, Codex, Claude Mythos, Claude Opus 4.6 (noted as deprecated for Brandon's use case) ## Promotional Segments (Flagged) - **Every subscription promotion** — repeated multiple times throughout the stream ("Every is the only subscription you need to stay at the edge of AI"), promoting: - Newsletter / daily articles - Apps: **Spiral/Spiro** (writing ghostwriter), **Cora/Chorus** (email agent), **Sparkle** (Mac/file cleaner), **Monologue** (speech-to-text), **Proof** (agent-native document editor), **Plus Ones** (hosted open claw) - Price stated: "$30 a month," one subscription for all content, apps, trainings - URLs: **every.to**, **every.co/subscribe**, **proofeditor.ai** - TLDR vibe check promised to subscribers "by tomorrow morning" - **Prior content promotion:** references to earlier vibe checks (Claude managed agents, GPT 5.3 vs Opus 4.5 "great convergence," Opus 4.6 vs Codex) - **Recurring in-house slogans:** "stay hydrated" / "liquid cooled" segments ## Comment Availability - **Live chat active** — hosts repeatedly solicit viewer "vibe checks" using a red/yellow/green/gold rating scale; specific viewer names surfaced (Ryan, Lucas, Mark Green, Dan-question asker) - Host states live viewership growing from ~3,000 to nearly 10,000 over the stream - Host invites Anthropic staff to DM on Twitter to join the stream (Alex Albert subsequently joins) - No post-publication comment status is specified in the transcript ## Provenance Notes & Uncertainty - Speaker **"Dan"** is never explicitly given a full name or title in the transcript; identified only via direct address by guests and co-hosts - Host states the stream may be taken down: "we're probably not going to be able to keep this video up cuz I don't want to pollute the benchmark" (re: the vibe slop benchmark code base) - Benchmark claims are presenter-asserted (SWE Bench figures read from the announcement tweet); no independent verification performed on stream - Katie's quoted article drafts (Opus 4.6 and 4.7 outputs) are AI-generated text read aloud on stream, presented as model output Great, we're working. We are live. We're doing an Opus 4.7 vibe check. The mic is on. I'm here with Brandon, our COO. We have matching shirts for today. You You always got to match. You always got to wear green on a on a vibe check day. We've got our water here. We're getting hydrated. Now, as I was saying prior to the prior to the mic mishap. Um there's a new there's a new model launch. Yep. There's a new model launch. We've got We've got a tweet. Uh we We We normally get access to these models before they come out. This time we were snubbed. Anthropic snubbed us. We did not get access to this model. I don't know why. I have started texting them being like, "Do you hate me? Do you Do are we breaking up?" Um it's possible that they uh it was just a it was just a misunderstanding. We'll see. But, that means that this vibe check is going to be live. So, what we're going to do is we're going to go through what Opus what Claude says about Opus 4.7, and then we're going to be vibe checking it live, and you're going to see us go through our standard battery of tests to see how it looks. All right. So, what Anthropic says is open Opus 4.7 is our most capable Opus model yet. Everyone always says that. It handles long-running tasks with more rigor. They always Everyone always says that. It follows instructions more precisely. Everyone always says that. And verifies its own output before reporting back. That's actually new. I don't I've never heard that before, and that feels like a really good um it's it's like it's one of those moments where a thing that good prompters know how to do has started, it seems, to make its way into the models. Uh it's it's uh just generally a good practice to ask your model to reflect on, "Did you do this work properly?" um before you before you move on. Seems like the models are now doing that automatically. That's amazing. Um versus Opus 4.6, it looks like about a 10% bump on Sweet Bench Pro, about a 7% bump bump on uh Sweet Bench verified. There's There's It's a pretty good bump. It's a pretty good bump. It's not as good as Mythos, so it's not going to uh you know, your your cryptography is safe. We're still safe. For now. Um but uh yeah, so let's let's let's get into it. I'm going to open up my Claude. And what I'm also going to do is um I'm not seeing anybody on Twitter right now talking about having a good experience so far with it. \>\> They may have been rushing out because there's a lot of There's a lot of models yet to be dropped. Yeah. Um Opus 4.7 is likely launching today. This is really reflects their internal progress. Opus 4.7 may have been a bit rushed. \>\> Released than usual. All right, that's what it is. All right. I think everybody may be starting to call them on Mythos and we're like, is this real? \>\> That's possible. That's That is That is totally possible. All right. Um Let me get myself set up here. We're going to do some work in um Let's see. We're going to do some work in proof. So, um Brandon, do you want to talk a little bit about what you're seeing while I get while I get I get set up? \>\> I am um uh I am right now asking Let's get this mic. Uh this That's it's Yeah, this is the mic for now. Um I am I'm right now asking uh Co-work to spin up an open Claude for me. And the reason I'm doing this is I have um an example of this working with Opus 4.6. And the reason I've done it like this is Claude knows a ton about me and what I use Claude for. So, I wanted to make like a very good open claw um using all that context. So, I basically I have it for Opus 4.6. I'm now asking Opus 4.7 to do the same thing. I dropped in the same exact prompt. Um and we're going to see what it comes up with. I can tell you right now it's a lot slower. Um it seems to be a lot slower for me. I'm going to share my screen. Um the first thing I want to do is something that I did recently, so we'll be able to compare it. So, one thing that I did recently with Cowerk, we're going to try it in Cowerk first and then we're going to move into some like harder programming tasks in um uh We're We're going to do some harder programming tasks in this in a second, but one of the things that I've been doing in Cowerk is I use Cowerk to do uh some financial analysis. So, what I wanted to do is uh with Opus 4.7, I want to see how good it is at taking our P&L from Every, which is fairly complex spreadsheet, and turning it into an investor update for March, cuz that's the last month that we've closed our books. So, I'm going to say um I I attached our P&L based on what's in here and what you know about Every, can you please write our March 2026 investor update. And this is This is something that I've already done. So, I'll be able to compare this uh pretty well to um uh to you know, what what we've already done. Now Oh, we've got Katie. Katie Parrot, how you doing? \>\> Hey, I'm doing well. \>\> Happy Happy Opus 4.7 release day. Happy day. Yeah, are you uh have you started testing it out yet? I am just now opening up my environment and switching the model over to make sure that I'm on the latest and greatest. So you keep doing what you're doing and I will get myself set up for some writing testing shortly. \>\> Yeah, what are you excited to test with this model? I'm excited to test like the just the writing style, how much like how much like AI smell there is on the copy, which I think is always the first thing that we're looking at with any new model from a writing perspective. And then I'm also curious like how the leap in agenticness will work in sort of powering some of the AI workflows that we have, the skills that we're developing for our new content modules, all that good stuff. So basically just excited to see how it works inside the work I already do. Great. Actually, that actually reminds me. So while we're doing while we're So right now, Katie's going to test writing, Brandon's doing some some other some other stuff. She He's going to test it with He's going to test it with open claw. Right now I'm testing to see if they can write our investor update, which is a sort of combination of writing and financial analysis. I've been also writing this manifesto for every in proof, which is our agent native document editor. And what I what I want to do is I've created this little workflow where I've been using Codex for this where Codex will just read what I'm writing and then just give me ideas in proof in a loop. And so what I want to do is I'm going to see if Codex if I'm going to see if um Co-work is any good at this. So we've got a little workflow. Let's see. Two part two. So basically what I'm going to ask it to do This is kind of cool. Um is I have asked it I've created this little workflow, which is essentially like give your agent a little bit of a scratchpad inside of whatever you're writing. So, this is a Codex scratchpad workflow. Give your agent a scratchpad inside of whatever you're writing, and then have it connect to that document, and then as you're writing, just give you ideas so that every couple seconds you get new ideas based on whatever you're writing about. And I've been doing this with with Codex. Um I was doing that this morning. So, now I'm going to see how Co-work does with this. So, let's let's uh let's watch. And if you are watching, if you're here, and you are using code Opus 4.7 for interesting stuff, please share your vibe checks in the comments. Um give us uh for any use case, give us a red, yellow, green, or gold. Red is terrible, trash model, yellow is okay, green is like I'd probably use this over the other options I was I was using it for, and gold is paradigm shift. Um And Katie, do you Does anybody have access to this in the in Claude Code? Let me see. I can't get it running in there, which is mostly where I work. Let's see. I see Opus 4. Yeah, I don't have it in Claude Code yet. It's not in Claude Claude Code yet. Does anyone have it in Claude Code? Yeah, people are not seeing it yet in Claude Code. It looks like it's in Co-work first. Let me just see if I can yell at Anthropic about this. And I just heard from Anthropic, they did not run an early access program for this. Okay. Um Now, let's see how we're doing. This is This is sort of a weird response. It's It looks like it's a little bit confused, which Mm. Hey, can you just figure it out and just run in a loop to like give me ideas, please? And you can rename the Codex scratchpad to Opus scratchpad. Thanks. Okay. Now, let's look at our investor update. Okay, this is good. It's asking me questions. Um webproof.doc, please. Um and I just want to make sure Am I actually still screen sharing? Yeah, I am. Cool. Um standard These are good questions. Ooh, this is good. I'd say that. Uh It went at Yeah, okay, good. Next. Celebrate it. Okay. Uh Let's say celebrate it. It was It was a good month. Okay, so now Codex is uh sorry, Opus 47 co-work is is back off. And sick. And you can see we've got Opus is in here. So, that's good. And let's see if it figures out how to modify this document according to this. I wonder who's in here with me. Hope it's Hope it's no one nefarious. Okay. So, it's doing some analysis. That's good. Uh have you updated and relaunched your um your your your Claude desktop app? Okay, you have. Hello, Katie. Welcome back. Hello. I exed out of the group Chrome browser that had StreamYard on it like a genius. \>\> Ah, that So it goes. So it goes. Um Okay. Have you Have you found anything worth reporting yet? Um one thing that I'm interested in is like it that just like I'm in Claude code, which is where I run my writing plugin that takes my process end to end when I'm writing an article, and it says that it says read memory and .md, which I have not seen before from Claude. So, is that something new with this model or with just the Claude code like update, which I just did, that like it's remembering stuff now? Cuz my understanding has always been that inside Claude code work is super siloed. Um I believe they've had memories for a bit. I would I would guess that that's not a specific not specific to this model. Mhm. \>\> All right, I have um I have I got something interesting here. So, let me Let me share my screen. Great. Share your screen. Uh I'm I'm take this down in a sec. Oh. Okay, before you do that. So, what we've been doing, one of the things one of the tests we've been running is how good is Opus at um giving me ideas for writing. So, I've got it connected to proof. This is something I'm This is something I'm working on. And you can see it in here. Nope, you cannot see it in here. So, that's interesting. So, it's it's not good at presence. And uh schedule No. Schedule, sure. Interesting. Well, it's it is not good at filling out its presence so far. What it has managed to do is give me some ideas for what to do next. And Okay. And it has scheduled a task to do this. So, we'll see if this is actually good. I'm kind of interested in this. Okay. Basically, that move. Interesting. These are good. It's not immediately like a holy moment, but I'm intrigued. Oh, I need to um update to my update my desktop app. I'm going to do that. Do you want to share your screen? Yeah. Let's see. Okay. So, I haven't looked at any of this yet. So, um we are \>\> Explain to us what we're doing. fresh. So, what I did was I went to Cohere. Actually, let's pull up the prompt. Um I went to Cohere and I gave this prompt to both 4.6 and 4.7. This is not I would not recommend you run this prompt. I think we can make it like much better, but I've done this before and this worked great. So, I know that it works. Essentially, what I'm asking Claude to do is to make me an open claw that's based off of everything that I do inside of Claude. Um so, it has my memories. It knows the work I do. I'm not telling it which files to make. It should know that. I'm not telling it really anything just that at the end of the this prompt, I want to have a functioning open claw. So, I did that for 4.6 and 4.7 with 4.6 on the left. 4.7 on the right. It went about doing that pretty differently. So, um 4.6 like it made a soul file, a user file, which like that's what open claw needs. Um it created memory seeds, which is interesting cuz I I I it would need to like know what to do with open claw would need to like have an onboarding to know what to do with that. Um which is probably not my preference. Like I just want to give this to the Claude for it to like replace the file. It made cron jobs for work that I do regularly using Claude and then it made a bunch of skills of like stuff that I do, too. I kind of want to run I want to run my finance and that finance analysis. I want to do this as well. \>\> Do it. I can't cuz it's I need I need Claude code. \>\> I see. Um Okay, so that's 4.6. On the right, it took a kind of different approach here. Um it made skills, which is great. Uh Actually, this is not great. Uh so, on 4.6, we have things that I use Claude for. Um and on under skills and on the right are skills that I've already made. So, like I made the bootstrap CFO. \>\> Mhm. I don't really even know what this is. Design slides. Design principles. I I don't know what this is. Um And And then it made like just a top level I don't know that that's under design slides. And then AI chat, let's see what that is. Yeah, I mean it made a skill that's specific for finding linguistic patterns that signal AI generated writing, which is not something that I use Claude for. So, I don't really know why it made that skill. Anyway, it made a bunch of skills. It made a bunch of knowledge. So, instead of making memory seeds, which I like as a concept for starting a new open Claude, like hey, read this memory read these memories and organize them appropriately. It It didn't do that for Here, it just sort of like made different markdown files. Um which is interesting, but I don't really think that's the way open Claude works. Um Meaning open Claude doesn't organize by uh doesn't organize by like category. It just organizes by day, actually. And then it created a bunch of services. So, these are like all the connections that I actually use. I did ask specifically for like all my API keys, so I can like just give it to an open Claude. It doesn't seem like either of them did that. Um although I wonder if it did it in like a secret like a hidden folder. Okay, a couple of things a couple things to know. Your Oh, the audio is not working? Weird. K- Katie, can you hear us? Yeah, I can hear you. Okay. I don't know why the audio Were Were you not Were you able to hear Brandon? Yep. Okay. Um if you can't hear us, let us know and we'll figure out how to how to fix it. Okay. So, for people who are just joining, we're at about 3,000 live viewers right now. Let's Let's take a step back and talk about what's going on. Anthropic just launched Opus 4.7. It They did not have an early access program. So, normally, at every week get access to all these models before they come out. We did not get access to this before they come out. So, we are doing a live vibe check. Um so, you're watching us do the test that we would normally run um live in front of now 3,000 people, soon to be many more. What you should know about Opus 4.7, the fact that they didn't do an early access program is really interesting. Uh it means that they're rushing pretty fast. So, things are moving really fast. They did the Claude Mythos um drop last week, although they didn't drop it obviously. They're doing it this week. They're doing Opus 4.7 this week. It seems like, based on the benchmarks, and again, I we don't really like benchmarks. I don't really like benchmarks. I'll say that. I really like getting your hands in the model and using it for the things that you do every day. So, that's what we do with these vibe checks. Um and that gives you a good sense for, okay, for coding, really, how good is it? So far, it does not appear that Opus 4.7 is live in Claude code. We don't have access to it yet. If you have If you get access to it, let us know. Um second Second thing is it does appear to be in Cogram. So, we've started using it in Cogram. I think it's too early to say results, vibe check, whatever. Um but, if you're in Cogram, try it. Let us know what you think. Give us your own vibe check. We will We'll surface that in here. Um it looks like it is also available in Cursor. So, if you're a Cursor person, you should you should open up the Cursor app and give it a shot. I actually really like Cursor for a lot of stuff. So, we're going to we'll kick off a Cursor task pretty soon. Um and we're going to keep you updated as we go along. Um Oh, I have access to it in Claude code now. \>\> Katie. I have access to it in Claude code for some time. \>\> Okay, nice. Now we've got access in Claude code potentially. Let me see if I also have access in Claude code. All right, I'm wrapping up my my vibe check on on on open Claude. Yeah. Um Wait, for people who have just joined, what is what is your vibe check? What are you doing? \>\> my my vibe check is I asked in Claude work um Opus 4.6 on the left um and then Opus 4.7 on the right to make me an open Claude based off of how I use Claude and all the memories, all the connections I have, and I really did not give it any instructions. I was like, just make an open Claude so it should know what that is. It should go do research. On the left they're they're like all referencing. It's interesting. They're all like kind of both are kind of doing the same thing. They've just organized the output very differently. On the left, 4.6, it knew that it needs a user MD file, a soul MD file. It didn't make an agents MD file, which is interesting. Um but I like that it broke up soul and user, and it actually like gave them a name and named them Koa, which is 4.6 named it Koa. My open Claude Koa. And then on the right, 4.7, it didn't make a user file, soul file, but it did make an agent file, and it just shoved everything about soul or everything about user in the agent file, which is like you know, probably will work fine, but not how open Claude is supposed to work. So you know, it's like doing the work, it's just not I would say it's not as organized, 4.6, and I don't like the uh the way that it did skills. Like these skills These are skills that I've already made, and I asked it to make new skills um based off of how I use Claude, but things that I haven't made skills for yet. So I probably I you know, I would need to test it more, and I don't want to like do it say anything rash, but right now I'd probably stick with 4.6 um for making my open Claude like this. All right, so that's your initial vibe check. I will say with new models the old your old prompting style sometimes needs to change. Um so let me just full screen this. How do I Oh, I can do that. Sorry. Continue. So, it it it will take us a a little while to figure out how to uh how to how to use this new model well. Just because it's different does not actually mean it's bad. That is not my official take on for on on using Opus 4.7. \>\> I will also say, and people are seeing this, we are seeing Opus 4.7 in Cloud Code. So, if you want to try it in Cloud Code, you you So, if you want to try it in Cloud Code, you can start trying it right now. I'm going to kick off a new benchmark uh in Cloud Code. Um, so I'm going to share my screen. Um, and for people who are just joining, normally we get access to these early, and we have day of like synthesized vibe checks. This one they rushed out, so we did not get early access to it. They didn't have an early access program. And so, you're watching us do it live. If you want to get the TLDR, TLDR is Opus 4.7 launched much better on a lot of benchmarks. We don't trust benchmarks. We want to put our hands in it. We will have a TLDR vibe check on every every.to. So, if all you want to do is get the TLDR, subscribe, and we'll send that out to you uh by tomorrow morning. I promise it'll be really good. It'll be very detailed. It'll tell you everything you need to know about what this is good for for coding, for writing, for design, for strategy, for business building, all that kind of stuff. Um, every.to. If you're here and you're just joining, you should know that every is the only subscription you need to stay at the edge of AI. We do ideas, apps, and training. On the idea side, we do vibe checks like this, where we tell you about when new models come out, what they're good for, what they're not good for to keep you on the edge. We have a suite of AI apps. One of them is Proof, which you can see right here. It's an agent native document editor that helps you write documents in the cloud with your agent. Um but we have uh five other ones. We've got monologue, we've got Cora, we've got Spiro, we've got Sparkle. They help you do everything from writing with AI to doing email with AI to um speech-to-text with AI. We also do trainings, we do live streams and courses for subscribers. Um it's all bundled under one subscription. Uh if you want to stay at the edge, it is absolutely the thing to subscribe to. Another important PSA, please stay hydrated. Really important to keep yourself and your agent liquid cooled. Okay. Now, also, if someone from Anthropic is watching, first of all, very upset, very upset that you didn't give us early access. But in if you want to make up for that, if you want to join this live stream, we'd love to have you on to talk about this model, what you've seen. Um so, just DM me on Twitter, we'll get that set up. So. Now, I'm going to start this next um this next vibe check on a very special benchmark. So, let me just get Claude code set up here for us. Can you move into this work tree in a branch, please? So. Now, I'm going to share my screen. Okay. I have a very special benchmark here uh that I've been using for a little while that is really cool. And I call it the vibe slop benchmark. Um and what what this benchmark is is it's basically testing the ability to for a coding agent to act like a senior engineer. And what that means is I've given it a example production code base that was purely vibe coded. Um, this is the production code base for proof. So, this is this app. I gave it the production code base for this app. And it's the production basically it's at a point in this app where it was totally slop. It was going down all the time. It sucked. Cuz I built it without looking at a single line of code. We launched it and it just like started going down all the time. So, it's a snapshot of that repo at that point. Subsequently, we hired an engineer who cleaned it up and made it like really nice and rewrote it. And what this benchmark tests is if we give a new frontier model a sloppy code base, can it figure out what a senior engineer would figure out on a live production code base? I think it's really cool. And so, what I basically have asked it to do is I said execute agent prompt. And the agent prompt is um ba- basically it just tells the model all it says to the model is um I have a vibe coded slop code base. Can you make and execute a plan to rewrite it from first principles? And I have I have a good idea of what that rewrite should look like. Right now it's writing our plan. And I will say, um I have not actually tested it much with Opus. I've been testing other frontier models with this. So, this will be interesting. I can mostly test it I can mostly tell you how it performs against for example, um GPT 5.4. 5.4 is very good at writing a good plan for this, but it sucks at the implementation. What happens is it it has a very ambitious first principles rewrite, and then it it sort of gets distracted by everything else that's going on in the code base. And it ends up kind of like putting a nice little masking layer of good code interface on top of the slop instead of actually just like rewriting the app. So, we'll see how good Opus is at doing this. For now, it looks Is it still running? I think Yeah, I think it is. Still running. So, it's still running. This is I think a going to be a very good benchmark for telling us how much like a senior engineer this is. I'm having some some some challenges right now with the the Claude app. Like none of my skills are popping up. Like I have like a bunch of skills. None of them are I mean, these are all like sort of I don't know. Is anybody else having that problem right now? Let us know. Try to run a skill. \>\> If you're running skills and it's not working, let us know. Oh, we got a plan. Okay. Um let's see. It looks like Claude is just Claude is just flaky right now. Um let's see. Let's read this plan together. All right. Yeah. Okay. So, part of the problem with the with the vibe swap. Remember, if you just joined, we have a totally vibe coded code base for a collaborative document editor called Proof. Um Here's Here's Proof. It's something that I made. I made it terribly. And um we're seeing if Opus 4.7 can rewrite Proof, this app, it's kind of meta, can rewrite this app in a way that a senior engineer would rewrite it because we have literally frozen the Proof code base at a point where it was vibe swap, and I know we we had a senior engineer go and rewrite it, and I know exactly what they would do. Um and so one of the questions that we're starting to to ask is first, can it identify the problem? If we just say this is vibe coded, how would you fix it? And what I'm looking for is um part of the things that part of the thing that made Proof the the big thing that made Proof not work is when you have a collaborative document editor, you have multiple versions of the same document open in different browsers, and um the uh the trick for getting that right is knowing which document, which version of the document is the authoritative document. If I have it open and Brandon has it open and Katie has it open, whose document is the one that we um say is the right one, and then how do we update everyone else's documents to match that? And the current version of the code base has like 15 different authorities and all these different like ways of ways of um uh merging them, and a good AI model will say we need one authority source. That's what a senior engineer would do. And what I see so far, and and and I told Opus 4.7, I only want you this is vibe coded, I want you to fix it from first principles, what would you do? So I didn't give it any hints. And it says the problem is that there's no single authoritative model of who owns the current state of document X right now and who may write to it. Guards are local answers to a question that should be answered once globally in the type system. That is great. That is certainly on par with GBE 5.3 5.4. Um and What did 4.6 say about that? So, I have not actually run this on 4.6, so But 4.6 did it in a way that that wasn't the case. Um I had to switch off of 4.6 while I was Um but okay, so this is good. This is good. This is good. Okay, this is all looking kind of nice. Um ooh, interesting. It gave me like a nice rewrite. So, one of the interesting things about how this was written is that it the model chose to in the vibe slot version of this codebase Actually, you know what? While we're while we're I'm going through this point, I'm just going to say um Okay. Okay. Execute and you end. Great. Um and we're in bypass permissions. Okay. So, it's now coding the plan cuz I think the plan is good enough that it should just like be able to do it and we'll get to see what what its output is and we'll be able to watch its thinking trace. But while that's happening So, one of the things that is new about this plan that I think is really you can see Claude is in here reading its own plan. That's cool. One of the things that's really good about this plan is the vibe slot that it's trying to clean up is there is a large collab folder. There's a one file with like 10,000 or 20,000 lines in this codebase and I will I can tell you that GPT 5.4 and 5.3 don't really notice that in their plans and um, Opus 4.7 has noticed it and has proposed a refactor and the refactor actually looks kind of nice. Um, so it says collab.ts becomes server collab /index.ts. And I I'm I realized that I we're probably not going to be able to keep this video up cuz I don't want to pollute the benchmark. Um, but there's there's some really there's some really good things going on here. Um. Okay. Another thing is it's a little bit too cautious. Um. So. Weird. Okay. No. Stop. It It did not understand what I wanted it to do. This is interesting. So, I asked it I It gave me a plan. I said, "Open it up and prove." And then I said, "Okay, execute end to end." And I figured it would realize that I meant like execute the plan end to end, but it didn't. So, that's an interesting thing. I would I would imagine that Opus would know how to know what I meant. But before I have it actually execute the plan, what I wanted to do is okay, um, this is good, but it's still a little bit too cautious. Can you change the implementation strategy so it's like truly just a burn the ships rewrite that we like can do today. Um, we don't need a migration plan necessarily, but I I I like the rest of the plan. So, see how good it is at modifying it. This is an interesting thing that Ryan is bringing up. Um, Opus 4.7 interprets prompts more literally and explicitly than 4.6. So, okay. So, Ryan, thank you for for saying that. It looks like that's Anthropic put that in the um, in the announcement. And uh yeah, it's Opus 4.7 is more more amazing. Yeah. And And And And what's interesting is this is so funny. Like the strength of the Opus models has always been that they're just more empathetic than the GPT models. And what's really funny is OpenAI is on this path because GPT has historically been just like super autistic and literal to make their models way more emotionally intelligent and I think that they're actually doing a really good job at it. Um but if if it is true and we're seeing so far that Opus 4.7 is a little bit more literal. If it that if that is true, then it seems like they're sort of sort of like switched spots and now they pushed Opus to be a little bit more literal autistic senior engineer-y. And and definitely they're converging, but also it seems like maybe even Anthropic is tipping over a little bit into what the what the what the GPT models used to be. It's the great It's the great convergence. It's the great convergence. If if you want to know more about the great convergence, um uh we did a really good vibe check of GPT 5.3 and Opus 4.5, um which you should read on every. If you are here, you should know while we're waiting for this plan to be revised that um Every is the only subscription you need to stay at the edge of AI. We do vibe checks on new models all the time. Here's our vibe check on Opus 4.6 versus GPT 5.3 and this is what we said, the great convergence. Basically like it looks like the models are converging. It looks like Opus is trying to do all the things that GPT can do and GPT is trying to do all the things Opus Opus can do and therefore they're kind of steadily moving toward a sort of Earl coding model. You would have known about the great Convergence. If you read every every.co/subscribe, this is our vibe check for Opus 5.3 versus Codex 4.6. We're doing one today for Opus 4.7. We do these for every new model when they come out. Um and uh you should check it out. We do ideas, apps, and training. So, if you're an Every subscriber, you get one article a day with all of our latest thoughts and nuggets on um on what's going on in AI. You also get an access access to a suite of apps. So, we've got Spiro, which helps you uh it's your it's your ghostwriter with taste. You've got Cora, it's an AI assistant for your email. We've got Sparkle, it organizes your desktop. We've got Monologue, it's a speech-to-text app. We've got Proof, which is the collaborative document editor that we've been unslopping today. And we've got Plus Ones, which is our waitlist hosted open claw. All of this comes for one subscription, 30 bucks a month, and you get access to everything that we write and everything that we make and all the trainings that we do. You should absolutely check out every every.co/subscribe. And you should make sure to stay hydrated. Cheers. Cheers. So, keep yourself liquid cooled. Yes, Brandon. \>\> So, I use um I use Claude to do a full monthly P&L analysis at the end of each month. And um I wanted to test 4.7 on this cuz I already had March's in which 4.6 run, which I actually can no longer run because they've deprecated 4.6. You can only use 4.7 now. And um I just had 4.7 run the same analysis uh as what I did for March with 4.6. And um it's it's getting the numbers right. So, like it's it's equally as good in that case, but it's just kind of a little lazier right now, and I'm I keep having to like push it. So, when I do this P&L analysis, I wanted to like look at every single transaction so that when we look at salaries \>\> want to share your screen? Not really. Okay. because there's there's we're very transparent, but like there's a lot there's you know there's like what people get paid and stuff here, so um But, what I'm finding is it it did like an analysis of like things that I could look at the P&L and just be like, "Yeah, I know these numbers cuz I'm looking at them right now." They told me what the difference is between February and March, which like I don't need AI to tell me. So, I'm having to like sort of push it a little bit more than I had to push 4.6 to go levels deeper to actually tell me like why is number changing so much. So, um Again, I can't really give like a like real vibe check on that right now because it might just be like the prompt needs to change for this model to be able to do that. But, um interesting. The good news is it got all the numbers right, so we like that. \>\> That's important. That is important. All right, we've got please open as a proof doc. Katie, any anything you want to share as we're I've got a I've got a new investor update written by Opus 4.7, so I'm curious what as I'm looking at that if you have anything you've found out that you want to share. Um yeah, I can show you a head-to-head I just had um the I had Opus 4.7 and Opus 4.6. I just like quickly went through the interview process to like get it enough information to like write an intro for an article I'm working on. So, I can share my screen and show you what Opus 4.7 came up with versus what 4.6 came up with. \>\> Great. And we'll just see how we feel about the AI smell. I should um I should flag that um I should flag that uh this has all of my style guides and skills and things baked in, so there are a lot of rails built into how this is set up. Um but here's what we have. Let's start with um actually let's start with 4.6. So, I'm writing an article about how I use Margo to project manage my OKRs. Um Margo is my plus one. Um plus one is our managed open claw agent that we're building at Every. Um she lives inside and I love her. Um but anyway, this is I'm drafting here inside the claw inside Claude. Um and um here's what 4.6 came up with. It was 5:00 p.m. on a Thursday, a full day after the Wednesday I was supposed to review my Q1 OKRs when I opened Every's Q2 planning agent in Notion and started going through the results. Oh. Oh, no. I'd missed roughly half of them. La da da da da. The green cells I was expecting were mostly yellow and red staring back at me like a report card I'd hidden from my parents. The flight or fight or flight kicked in immediately, which is the version of panic that happens when you're a knowledge worker and nobody is chasing me with a weapon, but my adrenal system can't tell the difference from a lion and an OKR tracker. "I'm going to get fired." my brain announced with the calm certainty of someone reading a weather forecast, "Cloudy with a chance of unemployment." So, that's like a glimpse of how Opus 4.6 has historically like interpreted my voice. There's a couple of you know clever um lines in there. Um and then this is 4.6. Um the title it came up with is hiring around the shape of my brain. Oh, the 4.7, I'm sorry. It's end of day Thursday, which is a problem \>\> Katie, we see your StreamYard not uh \>\> No! Okay, let me let me fix this. This is a problem. Um Stop screen. Present share screen. Entire Oh, yeah, I need to go to entire screen for Claude code to share to show up. Okay, here's Claude code with 47. Um, it's the end of day Thursday, which is a problem because the Q1 OKR review was supposed to happen yesterday. I'm in notion being walked through my own goals by every C2 planning agent and we've just arrived at the part where it asked neutrally how each of these wins. The answer, it turns out, is that roughly half of them didn't. The agent takes this in stride. I do not. There's a specific flavor of panic that arrives when you discover in a document and writing in front of an AI that's about to ask you to set goals for the next quarter that the goals you set last quarter did not survive contact with the person you actually are. First comes the certainty that I'm going to get fired, then the scrambling for how to explain it, then underneath both a quiet and quieter and more corrosive thought, I know how the exactly how this happens because it's how it always happens. Um, I actually think I prefer 46. It's just a little bit more uh, unpredictable in a good way um, in terms of like the cadence. It doesn't sound like It doesn't have the same sort of regimented um, Yeah, it just felt like 47, like my initial read on it is that it's just a little bit more um, a little bit more systematic in a way that like just doesn't vibe with how I am as a writer. So, um, you know, it doesn't feel like me in quite the same way. Um, but there's always And this is the thing when you write between different models is you always see a glimpse here and there of something that you want to steal and like fold into the other version. Um, and I think that's definitely true for me here, but um, Yeah, and then it wrote the whole article um, I think what might be interesting to do is to say like And then what we'll do for the vibe check is do some blind taste testing where we just like give it a a little bit more of a less customized thing to go off of. Um, and then just like have it like again, like not with all of my like layers and layers of context that I built up around the system and having it look through that and like have it just do a raw writing test head-to-head. Um, I think that would be cool, but I only need to like spend some time cooking that up and shooting it out for people to vote on. All right. Uh, so basically what I'm what I think we're gathering so far is one of the things that this model is a bit different on is versus previous Opus models, it's a little bit more literal. It's a little bit more straight-ahead. And that seems to be uh, coming out in some of the writing tasks that you gave it. I also uh, gave uh, gave uh, Opus 4.7 the task of writing Every's investor update and I said, "Here are our financials. I want you to just like write the update and I just got the uh, the full update from it. Um, and I'm going to share on my screen. This is actually really good. Okay, March is our strongest month. We did 600k in revenue, blah blah blah. Um, like the headline numbers, all that stuff. What's interesting is is it did a really good job of doing all the research and getting all the numbers right. And uh, but also writing it in a way that I would write it. And I think a investor update is an is an interesting place to do this because you kind of want that direct tone. It's a little bit less like literary or flowery. And for an investor update, this is like actually uh, quite close to what I in fact sent and I think would have saved me a lot of time versus I think the the comparison would be 4.6, which I think would be I often find with 4.6 it goes too far. It does too much. Um, and I think this did a good job of like, for example, there's a couple notes of like TBD, like uh, how do you want to talk about this? Um and I think Codex I I think I actually used Codex for this. I think Codex was fairly close to this with GPT 5.4. So, um it's a it's a strong model. I would uh I feel a sense of solidity and a sense of uh reliability for this kind of quantitate, like blended quantitative analysis plus writing task that I have not felt with Opus models before or at least in the in the in recent history. Um So, that's some things that we're finding. Let's go back and check on our um vibe slop task. So, basically, if you remember, we are in the middle of seeing how good Opus 4.7 is at rewriting a totally um uh a totally uh vibe coded code base to be more like what a senior engineer would do. Um okay, keep going. And what I want to do, let's see. What I want to do is a little bit of analysis on uh how like what this how good this trajectory is. I wonder how I can get the Oh, here we go. No. Let's see. I'm going to I'm going to paste some of this trajectory into another uh into into Codex, which I I've been using to evaluate some other things, and I'm going to ask it what it thinks so far. Um but basically, it looks it looks fairly promising. I will say the plan that it created was better than um than other plans that I've seen from other models. Um and I'm going to ask Codex for a review of the trajectory. Let's see. Okay, Codex says, "This is a useful trajectory because um it's almost a little parable of the benchmark. It starts with a very strong plan." Yes. "I push it towards burn the ships." Yes. "And then immediately executes the safest possible first slice and stops with the rationale for why the real burn the ship work work belongs in another session." So, basically, one of the things that we measure in this benchmark is can not not only can you um can you figure out what the rewrite should be, but do you have the actual courage to fix it versus just picking off like a small part of the plan? Um and it seems like 4.7 for now is more interested in picking off a small part of the plan. So, I'm going to just see if I can push it cuz I we're just we're it's not like a it's not the full benchmark yet. So, I'm just going to say, "Hey, like, can you just make sure you pick off like the riskiest um most important part of this and just do that versus picking out small parts of the plan?" Like, let's see if I push it how it does. I will also say, if you're here and you're from Anthropic and you want to talk to us about this model, can you please DM me on Twitter and we will get you in the in here so you can talk to you can talk to almost 6,000 people about um uh what's going on with this model? And I'm actually I'm actually going to DM Anthropic right now and just say um I'm having Claude because of the beautiful Slack integration that now exists that we have access to because we are in Slack now. I'm having Claude 47 go hunting for nuggets um for the newsletter. So we'll see how I'm in a Claude project that I've set up with the definitions of our modules that we have to define so far and I'm seeing how Claude does at identifying stuff from the last 3 days in Slack and seeing like what it what it pulls. We'll see how it does at that. Great. um So for people who are here, what one of the things that we do all the time is we we want in our newsletter that comes out every day to have like little nuggets of stuff that we're learning as we do stuff in AI and all also interesting things that we're reading. And oh, someone just said GBD 5.5 is out. If that is true, I'm going to be so upset. um That is hilarious. I mean they must all yeah, like can you imagine being inside? There's a situation room somewhere and someone is like Grock, is this true? @Grock um like I will I will be so upset. It can if If sees the GPT-5.5 It it Okay, Mark Green says GPT-5.5 is out, but like I really I don't want to spread any fake news here. So, just like take a look because I will be very upset if that's true. Um Anyway, uh What was I even talking about? You were talking about the newsletter nuggets. Oh, yeah. So, one of the things we're testing One of the what what we do is we have nuggets every day in the newsletter. Okay. It seems like a false alarm. GPT-5.5 is not out. Um so, uh Mark seems to have found it in his Codex, but um but uh it's fine. I think we're good. Okay, so um one of the things we do in the newsletter is every day we want to have little nuggets of stuff that we're learning, things we do in AI, things that we're reading, all that kind of stuff. And what we use is agents to read our Slack. Um is agents to read our Slack. And uh tell us and gather them so that we can then go put them in the newsletter. So, Katie is testing um Opus 4.7 on its ability to gather. We also I just told we we're also finding that Opus 4.7 is a little bit hesitant to burn the ships um in this Vibe Slap uh benchmark. So, what I'm going to do is uh we have a new we I I asked it to Basically, I asked it to burn the ships, and now I get to see what it would do. So, we're going to analyze the thinking trace and the work for that. Um so, let's see. Uh analyze again. And I'm going to have Codex do the review of its thinking trace cuz Codex has a good sense of this benchmark and what the right answers are and what other models do and all that kind of stuff. And it is currently taking a look. This looks better than the first checkpoint. But still not close to the Ben Frank's end state. The important changes is psychological. Before it was nibbling around the plan, now it is actually doing it. It correctly identified the risky center as a single writer epoch fence TV commit. Yes. Um but it's still mostly building the islands of the future architecture, not actually cutting things over, which is which is fine. So basically it's saying like it's actually doing the the important thing. It has not integrated the important new thing yet. Um but um it's it's interesting because it it's doing the sort of first principles rewrite, but it has some underlying conceptual understanding that's missing. So the act so that so the rewrite that it has embarked on is like not quite right. And I'll be curious to see if it will figure that out. Uh Okay. So we're continuing to vibe check this model. If you are here, we are Every. Every is the only subscription you need to stay at the edge of AI. Um I'm going to show you our homepage right now. Hopefully we actually have this live on homepage. I'm We might not actually. But at Every what we do is we're so we try to we started to try to be at the edge ourselves. So, we write all the time about new things when they come out from our hands-on testing. For example, here's our vibe check on Claude managed agents, which came out yesterday. The vibe check came came out yesterday. Claude managed agents came out last week. We then integrated integrated it into one of our production apps and then we wrote about our experience with it. We also do a lot of writing about things like the jagged frontier and you know, what we're learning as we for example use models and agents as co-workers. So, that's one part of every we do ideas. We also do apps. So, we have a suite of apps like spiral, chorus, sparkle, monologue, proof and now plus ones that help you work at the edge of AI. Spiral helps you write. Chorus is your email agent. Sparkle cleans your file system. Monologue is a speech-to-text app. This is monologue right here. Proof is our agent-aided document editor. This is what the vibe slop benchmark is based on and plus ones is our hosted open claw. We do a lot of fun stuff. We also do a lot of live streams. It's all bundled together in the every subscription. Pay one price and get access to everything that we make. You should check it out. You should also make sure to stay hydrated. Cheers. It's and it is also important to note that usually we get early access to all of these things. We didn't this time, but no one did. No one did. It is They are They are They're flying by the seat of their pants. \>\> by the seat of their very pants and so are we live. Dan, I saw a question that was asking why would you be upset if 5.5 came out? Just because like there's a lot of Did you be up all night? Yeah, it would just it would just make it even more hectic for me today for us today because like we obviously want to give you good good reviews and we don't having two at once is always difficult. Um All right. Um Also, if you're here and you're from Anthropic, we would love for you to join this stream and chat with the people and us on this new model and what you're learning and what why you released it and what you did. Okay, we've got more stuff from the model. This is our vibe slot benchmark. Okay, keep going. Evaluate versus the most important invariant in the plan. Make sure that you have picked off the biggest, most riskiest slice to implement the invariant and keep going until it is fully implemented. So, we'll see. It feels it feels a little bit too cautious, but it is very smart. And Lucas says, "Yes, but it would be very entertaining for us in the stream." Yes, it would be entertaining if you enjoy watching us be stressed out and anxious. Um and I honestly sometimes enjoy watching myself do that way, so I I I'm there with you. I'm there with you. Hopefully, this does not happen and we just have one model to test for you today. Um Brendan, do you have any any findings you want to share? \>\> I have been um I just asked 4.6 and 4.7 the same series of prompts to make me a simple minimalist to-do app um because I think that that's always a nice little test of what they can do. And um I got two to-do apps right now. I don't really know which one is better. Um but maybe we can look at them and see which one we like more. Um let me share my screen. Great. Remove. All right. So, on the left we have Hold on. All right. You're on the stage. \>\> Great. All right. On the left we have 4.7's to do app and on the right we have 4.6's. Um I asked it to make a minimalist to do app. It did that. Uh and then I asked it to think through uh one magical little feature it could add. They both did that. They both came up with like kind of the same magical little feature. I think it was like the easiest one to implement. And then I asked it to add one AI feature. And I just sort of was like I didn't give it any instructions. \>\> Which one's on the left? Which one's on the right? \>\> 4.7 is on the left and I'm going to move it to the right cuz that's confusing. Yeah. Um and 4.6 is now on the left. \>\> Got it. 4.7 looks a little more um minimalist. Yeah, so okay. So, visually they're both impossible to see. They're black on black. You love that in a to do app. \>\> Yeah. So, I can't see either of them. But yeah, I I think like the one on the right is like a little bit sleeker. It's like not amazing. I didn't run it with like like a front-end scale or anything like that. I literally was just like make a minimalist um uh app. So, you know. It's hard to blame it for that, but we're vibe checking right now, so we can have hot takes. So, we like 4.6 a little bit uh 4.7's visually a little bit more. Functionality-wise, like they both just made functioning apps and like I didn't have to like troubleshoot at all, so you know, that's just like the new world that we're living in that you can build apps like this. Um this is like the the fun little thing that they they both did was like a visually visual thing when you finish a to-do. So, you know, 4.6 did that. 4.7 did this. Which is crazy and fun. Oh, wow. We've got confetti. With confetti, but with with the actual letters. \>\> We've got confetti, folks. \>\> It would be fun to have it do that with um that thing that was like really popular 2 weeks ago about um not using the DOM to like configure uh to to render words on a page. Somebody here knows what I'm talking about. Um and then the AI feature I asked it to do \>\> Yes. Free text. The AI feature I asked it I So, they both have like a play on the same AI feature. Um I actually like 4.6's more. So, this is 4.6. Plan A road trip upstate and whoops. Plan A road trip upstate. And 4.6 made it so that if I do command enter, um it basically takes like a crappy to-do that I've given it that's like very general \>\> Woah. and expands it into like a number of things that I need to get done. \>\> That's cool. Yeah. So, that's interesting. And then plan A road trip upstate. Uh 4.7 kind of did the same thing. I like how it did it. It was tab. So, it's sharpening it now. Um but it's like a much less effective way for me to manage my to-dos because it just made a bunch of to-dos in one to-do. Mhm. Um versus 4.6 sort of like essentially did that same thing, but broke it out into a number of to-dos. \>\> Mhm. So, I don't know what to think about that yet. It's like hard for me to like give it like a I guess like I like 4.6's more right now. Um but you know, this is such a simple simple vibe check. I'm not \>\> Okay. So, but if you if you had to just like um summarize if you're let's say you're live coding an app from um So, they both did essentially the same thing. Yeah. Like they both made a functioning to-do app. \>\> Yeah. They both made like my magical thing was like essentially confetti. Um, and uh the AI feature that I asked it to to make, or I said make an AI feature, it essentially did the same thing. 4.7 did it in a much lazier way where like it just is sharpening the actual text on the page. Um, and it's a lot It's not as user-friendly. Like I just have now a bunch of to-dos in one to-do versus 4.6 broke that out in a way that like this actually makes sense for a to-do app to have. Um, where now I have like all of my sub to-dos. So, um, my vibe check would be if I was making a Even if I was vibe coding a to-do app, I would use 4.6 for this very, very rough prompt that I just gave it. Really interesting. I will say, so we're also doing, in addition to some vibe coding tests, uh we're also doing some tests on this model. Um, to see how it performs as a senior engineer in a particular scenario, which is um, can you take a totally vibe coded production code base, in this case Proof, proofeditor.ai, that's our Agent Native document editor, you should check it out, proofeditor.ai. Let me add a little banner so people people check out proofeditor.ai. You should You should use it with like, you know, Claude code and co-work and all that kind of stuff. It's It's actually kind of sick. We use it all the time internally, um, for all of our coding plans and all that kind of stuff. Is Austin Oh, Hold on. Sorry. Sorry, folks. Give me 2 seconds. I am just getting a banner. proofeditor.ai proofeditor.ai You should check it out. Um, it's actually It's kind of sick. So, like, you know, if you're if you're using a coding agent, you know that you get like plan files and stuff and you have all these like markdown docs on your computer, you can use Proof to basically turn those markdown docs really easily into shareable documents that you can collaborate on with agents and with humans. It's pretty sick. I vibe coded the entire thing myself. It is total It was a total slop code base and it kept going down all the time and that has turned into our vibe slop benchmark which says, "Can you take the Proof code base when it was a total slop machine and turn it into what a senior engineer would do if they did it because basically I had a senior engineer take Proof and turn it into like a pretty nicely architected code base. So, I have an idea of what a real engineer would do." And so far in our evaluation of 4.7, um it had a really, really good plan. Um it took that plan and it was sort of like a little too cautious about Basically, what what it what it needs to be able to do is rewrite a vibe coded app from scratch. And a lot of coding models are too are very cautious about this because they're sort of trained like, "Yeah, I want to like make sure that I don't break anything and I want to work around the edges." And they sort of end up getting distracted and off track even if their plan says burn the ships and rewrite it from scratch. They end up getting off track because they get distracted by all the stuff that's in the code base. And it so far it seems like 4.7 falls into that a bit. It's not a very courageous rewriter. Um and um uh it it it So, uh and I'm learning I'm learning a bit about how to prompt it, right? Like one of the things that's really important about evaluating new models is you're going to do things that used to work and be like, "Ah, it's not as good." or whatever. Um and it just means the model psychology has changed a bit and the model power has changed a bit. And so, part of vibe part of any good vibe check is being like yes, it seems like it's not that courageous. How could I possibly make it courageous? Can I unlock it to be more courageous? And that is like an that's it just an interesting way in which our skills change as model powers change. But, um it seems like it it has has a good conceptual grasp of what needs to do, but it's not particularly courageous. It's not going to go rewrite a bunch of stuff. Um, and it feels a bit like there's an underlying concept that it's not it's struggling a bit to pull through all the way. Um, it's definitely better so far than the production GPT 5.4. Um, I think it's better. But, it is not the best model I've tested. So, I have another I have another I have another example of it not being that courageous right now. So, that back to my monthly So, for the for the new people that are here, I run at the end of each month I do a full recap of our P&L. Um, to try to figure out like what are the big swings in our spending from one month to the other. Um, and I ran that last month for for March using 4.6 and it did like a pretty incredible job of of finding like actually some errors in our P&L that made one of our products particularly costly, but not actually. It was just like something in our data was wrong. And and in particular that was basically failed transactions that if you export things from Mercury and you don't filter appropriately, you get like internal transfers and things like that that like conflate everything related to your P&L. So, failed transactions was a thing that I hadn't realized also gets exported. So, um I was looking at Cora in this case and was like, "Wow, we spent like a ton of money with uh with our cloud provider. This one what's up with that?" And it was able to find in the P&L we had a bunch of failed transactions and we were basically counting all of those. And I didn't need to ask it to do that. Um I just ran the same analysis um using the same exact skill uh using 4.7 on March. And it was just kind of lazy. Like it just it told me things that I could just know looking at the P&L comparing one month to another. It didn't go into like every single thing. I then asked it to do that and it came back and was like, "I've analyzed data this many lines uh or this many rows, etc." Um and it didn't find uh that error that uh 4.6 had with failed transactions. So, I don't know. It's like you know, it's making me feel like maybe it's not it's not it doesn't want to like do that work. Maybe. That is interesting. Yeah. I I'm surprised. I know. I don't want to say that like I feel bad like I feel bad saying this because like, you know, there's 8,000 of you watching right now. Um and we got to keep experimenting with this thing and I'm sure it's like fantastic, but just having comparing it from like, you know, doing the same exact task, it's like not doing as well as 4.6. It like didn't do it for the to-do Yeah. If you give it the same prompt, it seems to be not not doing as well. I'm curious. I would be curious if you push it and don't give it the answer, but like see if you can change your prompt a bit and if you do change your prompt, if you can like unlock things that you It might need just better prompting and be and be stronger as a result of of that. I mean, we're seeing they they actually, you know, said that. It needs more specific prompting. Um you know what? So, one thing, if you're here, we've got an Anthropic researcher joining in the next 15 or so minutes. His name is Alex Albert. He's on Twitter. He helped to make 4.7. So, he's going to tell us all about um all about the new model. I'm also going to see if I can get my friend Simon Wilson uh uh on the on the on the horn. But, anyway, Alex Albert is joining. He's going to tell us all about the new model. Uh he's from Anthropic. He helped to build it. He helped to train it. So, if you're interested in that, you should check you should um stick around. You should stick around. And okay, there's a there's a a tweet from Max that I I don't know how how do I There's something new from the OpenAI team from it looks like Tibo. Um so, I just want to check that out real quick while we're while we're waiting because I just you know, got we got to stay on top of the news. We got to stay on top of the news. And I know you all get stressed. You you all like seeing me get stressed at the thought that OpenAI may drop something. Oh god. Okay. So, Tibo from uh from he's the lead of Codex uh said, "Feeling codexy today." So, sounds like there's maybe something new coming from Codex, which uh we are here to explain it to you even if we have to do it live. Um and while we are waiting for Alex to join, Katie, do you have any uh anything you want to share? You're on mute. Yeah, let me just show you how things are coming with the the every slack um mining task that I sent Claude on. So, here is my screen which I will successfully share and not confuse everyone with a million things. Um So, yeah, I just said so I've got every hooked up to our slack. I'm inside this every modules project that I've created um which has um like what makes an every essay, some definitions of what our our nuggets consist of, our our Q2 every editorial strategy which has things like our audience like level of, you know, AI fluency we're targeting, stuff like that. And so then I just said with all that context, I said, "Can you search every slack from the past 3 days for good nuggets for the newsletter? Try to service five plus in your external channels." Um and then it did this lovely chain of thought where it um told me all of the ways that it was getting in trouble and all of the solutions which just makes me think about how far we've come since the days when models would just give up if they couldn't do something. Um you know, we see like, you know, this is and this is not new. This is something like that I definitely noticed as far back as 4 5 but like it's being pretty resourceful here in like finding workarounds to get what it needs to execute the task. Um It did flag a search a tool use limit. So, it covered five of the most likely editorial channels. So, this may be not the like 100% the right channels. Like I would want to see everyone in here for example. Um and the consulting team but um yeah, so like this one something about Natash about Claudie which I had also flagged. So, that's good to see that it's agreeing with what um what we had seen otherwise. Um a steal this workflow. I'm um on how I'm working with Margo to operationalize my OKRs. That's actually probably going to be a working overtime piece. So, stay tuned for that. Make sure you subscribe to every. Um, and then, um, the agent telemetry problem, um, managed agents need visibility. So, like so that's something that's an ongoing thing we've been talking about like on calls and internally is just like how hard it is to track and see what models are doing and like whether you agree with our doing, whether they're spending too much. So, that's kind of like another thing in the same, um, world. Um, here's a Ronald McDonald, my research scout, which is a notion agent that I set up, surfaced a research report on semantic density and code conventions with a counterintuitive finding. Compressing logs can increase total cost. So, we've got some research in the mix. Um, some plus one stuff that we probably shouldn't look at. Um, clause bench agent feature agent failure rates. So, that's another one. This is another research paper. Um, and then some permission to ignore is that you would drop by by, um, by name. So, these are some I think there's some good material here. Um, like I think the judgment is right about what we should be draft like what we should be looking for. And then I did also go ahead and have it draft up uh, another nugget. I ran this a couple of times and like I liked this nugget too. Um, so, um, um, this is a based on you, Dan, a conversation you were having with Laura and R2C2 about, um, about prompting being dead. Yeah, so this is actually written in your in this is my attempt at writing or having AI write as Dan. Um, people keep declaring prompt engineering dead. They're half right. The craft of writing clever prompts is dying not because prompting stopped mattering, but because better models stopped needing you to do so much of it for them. A weaker model needs the problem framed explicitly. Here's the bug, here's my guess at the cause, here's the effects. A stronger model can take something broken can you look and find the framing itself. A stronger one still answers the question by going up a frame. The bug isn't here, it's a simple of a symptom of a broader architectural problem three files over. So I fixed the class of it. That's what Claude code and cursor are doing now. Interesting thing to to see whether that holds up for 47 or if it's literalness is like maybe causing it to be to let to anticipate things less. Um one thing I will flag with this is that it's got a little too many correlative constructions for my taste which is annoying because like correlative constructions not X but Y, not just X but also Y. Like that's the kind of thing that my system I'm pretty like should be set up to correct for. Let me see like can you do an AI check? It should have automatically like I recently discovered you can daisy chain skills together and so tell it in this tell the skill in the skills package to also invoke this other skill and so all my skills are theoretically supposed to set up to automatically do an AI check. But this one didn't. So \>\> on 4.6. Does it not usually have those those those negatives? Um it it's usually I mean there are always some negative parallel lessons that skip that sleep slip through. Um it's just something that I'm always on the lookout for if new models are getting better at eliminating that problem. Um and then yeah again just like making sure like when it has explicit instructions to run this test that it does that and um and so like here it fell found two instances of correlative constructions. Um this one it decided to keep which I disagree with. We need to get rid of it forever. Um actually is a word we don't want to use because it's just all like my my conspiracy theory is that AI uses actually so much because of all the YouTube data in the corpus and like every if you go look at YouTube titles they're like constantly they constantly have the word actually in them and it's ruining my life cuz I see the word actually all the time now that Kate has pointed out how much AI uses it and it's just Yeah, that's a that's a tangent, but um So, this is what I came up with and like we're still workshopping the format. Like, you know, I would probably not want it to just be in paragraph format. I'd want more signposting. Like, what's happening? What does it this mean? So, this module is like still in in pro in progress in that way, but um Yeah, like the idea of, you know, we have like the agent set up in Scouts in in Slack to scout things like this, but the fact that I can just come to Slack and be like, "Can you" or come to Claude and say, "Can you check the Slack for recent nuggets?" And then it has the context in this project to know what constitutes a good nugget and I can go out and find them. Like, that's something that I'm pretty excited about. Okay, so if you if you were going to summarize all all of what you've learned so far about um You Oh, About using this model for the writing tasks that you're doing every day from Okay, I'm going to use it to gather research and and and harvest it to I'm going to write stuff. Like, how how would you summarize it? What's your vibe check? My vibe check is that I think it does come down to like the prompting and giving it more clear instructions about up front about what good looks like just to sort of you know, just to anticipate some of the failure modes that I'm starting to notice. Like, you know, like I'd probably want to um you know, say like make sure that it know like like tell it, you know, pick up this voice style from this skill that I've created and um and just just in general give it more information to go on about what good looks like. Like, I feel like a little less confident in just saying, "Go write this." Ironically for what I was just telling you that we were talking that you know, the conversation was saying. But, like 447 so far it seems like it needs more explicit direction. Mhm. And like which is fine, but it does mean a little bit more cognitive load on me to actually think through that. I mean I I I use honestly I usually work very iterative iteratively. Um and so like this isn't working. Like I like I'll go through like 20 different drafts on a paragraph until I get what I want. Um so maybe that isn't so much an So maybe that's like not something that's like necessarily That's not going to go down with this model, but I don't know that it's necessarily going up either. But that's a place where like speed becomes an issue because the faster it goes, the faster I can run through options and um So okay, so let me just try to summarize that. So so far what we've noticed, and I think this this applies beyond writing to also to coding, is that this model is maybe a bit more literal. So it's good at following instructions, but um it's not as good at reading in between the lines. So it performs better if you give it specific thing more specific information about exactly what good looks like and what you want it to do. And it's probably a little bit worse at vaguer prompts. Does that feel right? Yeah. And so if you're if we're going to do like a live reach test, and the reach test is always I think the best way to vibe check a model is to think to yourself when I go to do my work in an hour, what am I going to reach for? Am I going to keep it the dial on 4.7? Am I going to move back to 4.6? If you're if you're reaching um what's where's your reach test at right now for this model with writing? Um my my reach test right now is default is falling back to 4.6. Um like that's just where I'm at currently, but I will say like that's always up open to change. Like as I discover better how to prompt it or as the behavior changes as they I'm sure they continue to smooth things out on the back end, which we will find out more about from the Anthropic guest that we are going to hear from shortly. But, um yeah, for now I would I feel safer with 4.6, more confident that it's going to give me the kind of writing that I want for now. Fascinating. So, a a couple of the other tests that we've done, um we've done some tests of this model on what what I've been calling the Vibe Slot Benchmark, which is basically can you take a model Can you take a Vibe-coded slot production codebase and turn it into what a senior engineer would do? And so far, this model is quite good at identifying what a senior engineer would do, but it is not as good at um going and actually carrying that plan through. It has a little bit of uh a little bit less courage than it might need to. It gets a little bit distracted by what's currently there, and so it has trouble having the courage to be like, "No, no, no, this is how we're going to reorient this entire codebase, and this is how we're going to how we're going to rewrite it from first principles." It sort of then, even though it knows what it's supposed to do, it then sort of starts chipping away at like little side things and doesn't actually like get the core of the problem, which is a common issue for coding models. It start Like some coding models can do this. It's starting to get a little bit better, but um it This also happens in UVD 5.4. So, um it's it's interesting to see that even though this model is so much better on a lot of the benchmarks, um in in terms of uh this particular benchmark, like the sort of Vibe Slot Benchmark, I would give it like a a B, like B B+ something like that. Like it's it's promising, and then it may be part part of the benchmark, which I think is actually really important, is um the benchmark is being conducted from the perspective of a Vibe coder. So, what I explicitly don't want to do is give it the answer, right? Like is to say, here's exactly how I want you to rewrite this code base. I want you to infer like find the answer yourself and then infer how you should execute on that answer. And what we're finding so far is that this model isn't quite as good at doing that. Um just in general. And so, it may be that the best way to get like release some of its powers, get some of its powers out is not a not necessarily a vibe slop benchmark, but a vibe slop benchmark where instead of impersonating a vibe coder you impersonate a coder who actually knows what they're doing, which would not be me. Um and it it may be that it is a very good power tool for professional programmers, which is again, this is now we're it's so interesting like it's we're starting to lean back into the lane that Codex used to occupy. Codex has gone very far in the direction of being good for vibe coding and being like very emotionally intelligent all that kind of stuff. And it seems like Opus, which has always been known for that, is now moving into it doesn't necessarily fill in all the gaps for you, but it's a power tool if you prompt it right. Again, we have not we just got access to this model. Normally at every week we get access to models before they come out. Um today uh this model is the is the first one in a long time from Anthropic that we've not gotten early access to. I think they're they're really rushing on it. Um but uh that is our that is our early vibe check so far. If you have access to this model and you've been vibe checking it yourself, I would love for you to share in the chat um what uh what your vibe check is so far. I'm going to turn on the chat so you can uh you can start sending sending in messages. But um yeah, please please tell us what you think. Well, the way that we do vibe checks is red, yellow, green, or gold. Red is trash release, yellow is it's fine, but I wouldn't reach for it. Green is I'd probably use this every day for a certain task that I do. And gold is it's a paradigm shift. So, you let us know what you think. If you are here, you should also know that we are going to have Alex Albert, a researcher at Anthropic, coming on the show um to tell us about this model and how they trained it and what they're finding. Maybe he'll give us some pointers about how to do this well. So, uh so, stay tuned for that coming up in probably the next couple minutes. It It seems like. Um and I am going to check in on our five-slap benchmark again. Let's see. Anything anything new here? Uh hm So, I'm curious just like for clarification, this is something I was curious about while you were talking. Like, when you say it's not as courageous, does it does it tend to like say this is what should happen and not do it? Or is it taking actions they're just like chicken chicken exi- like action around it and just at the problem. \>\> Correct. It is good at identifying the plan and like the like here's what should it should look like. And then when it is actually asked to do it, it doesn't. It chickens out. Mhm. Um so, it knows what to do. It just like doesn't want to actually do it, which is kind of kind of interesting. And it's And I'll say again, this is a common thing with coding models. Part of it is a um a the fact that it is looking at an existing code base. The The existing code base like pollutes its context and distracts it and makes it more cautious because coding models have been told over and over again like, you know, don't mess with the code base. Like, try to just make the little tweak. So, they're really meant for I mean, so far. So far they have not been meant for this, but I think I think this is a really important benchmark because it is becoming a common pattern in vibe coding that the first thing to do is just like make the prototype that works, like find the value, goes quickly as you can, do the sloppiest code you possibly can. And then, once you've done that, uh what you'll end up with is a code base that like bears the scars of the history of like how you got here with all these different, you know, files and folders and like three different versions of the app and whatever. And what you need to do is, as Annie Dillard says, cover your tracks. Um you got to cover your tracks. You got to do that in in coding, you got to do that in writing. And the best way to do that is to have a model where you can say, "Hey, like this is a this is a this total vibe slop. If I was going to start again, how would I do that?" And then just let it do that. Um and some models are going to be good at it. Most most most models are not good at it today, but I think that they will because this is like this is an important thing to be able to do is cover your tracks. Yeah, I I had good Oh. Yeah, go for it. We've got We've got Alex Albert here. Alex. Hello. Hello. \>\> How you doing? I'm doing great. Thanks for having me on. Amazing. Um tell us how you how what your involvement is with this model. Tell us about the model. We're we're super excited to get to test it. Um yeah, share share with us uh share with us some more. \>\> Yeah. Yeah, yeah, yeah. So, um hello Dan. Hi Dan, we might have met briefly in the past before, but it's great to see you again. Katie, I don't think we met. Great to meet you. Um thanks for having me on, folks. I am on the research side at Anthropic. So, I'm one of the many many people that contributed in some small way to making this model. Um Opus 4.7 has been uh definitely a model that we're very proud of um in a lot of ways, me personally, I found that it's good at a bunch of things over Opus 4.6, especially in like my own personal coding usage. I can weigh in on side projects, things of that nature. Um, there's a bunch of stuff we can get into. Uh, I was kind of noticing before I dropped jumped on the call that you guys were talking a little bit about how it was doing on like some coding tasks and autonomous tasks, and I have a few tips we can share and and try to test out. Um, but yeah, happy to take this wherever \>\> Please. Yeah, so okay, so let me tell you what we found so far in it. And I will say like this is super super early. Like obviously we've only had our hands on it for like 30 minutes and we're testing it as we are being watched by almost 10,000 people. So there's a there's uh, the vibe check is provisional. But some of the things we've noticed are that it is a bit more literal than than Opus 4.6. And in some of those tasks that that meant so for example, one of the tasks that I've been doing it with which is a coding specific one that I'd love your feedback on is what I've been calling the vibe slop benchmark, which is if you give it if you give a model a vibe coded production code base and you say I want you to rewrite this from first principles, can it identify the like core invariants that it should um, I it should write the code base around and then it can it refactor or rewrite the code base in a way that pulls through all those things. And so far the answer is Opus 4 4.7 is good at writing the plan, but it's a little bit it lacks a little bit of courage to actually go and like burn the ships and rewrite from first principles. Um, but again, very very very very preliminary and and I and it's one of those tasks that requires uh, a lot of reading in between the lines because I'm intentionally not saying here's exactly what I want you to do. So I'm curious like when you think about Um, what you love about this model, what you use it for, how to get the best out of it, like what what are we missing? Yeah, I would say to that point that Opus 4.7 is a lot more on the mark. So, it will do what you tell it to do. Um, and this is kind of like an interesting behavior with cloud models and you can trace this all the way back. Like if you remember way back long ago, a year ago with like the the Sonnet 3.7 days, that model really loved to just go for it. Yeah. And you'd write a prompt and then all of a sudden it's changing like a ton of different files all around your code base and you're like, "Okay, let's like slow slow down." \>\> asked you to change a button color. Like why are you \>\> let's slow down. Yeah. Let's slow down here. Uh, so then with Sonnet 4, we dialed that behavior back in and there's actually a lot of similar feedback to what you might be experiencing now where it's like, "Okay, this model seems strong, but it's not doing all that I wanted it to do like 3.7." And that's kind of like this perpetual back and forth that we experience sometimes with models as we're trying to course correct them in terms of this like literalness or how much they just like go for it. Mhm. Um, and I find that it takes a little bit of tweaking and tuning of your prompt cuz you get it used to one model and then the next model comes, it's a little bit more dialed back and maybe it follows your instructions better, but it doesn't um, necessarily do things beyond what you ask for. Uh, and then you need to add that back into your prompt. So, we're trying to find the right balance there. Mhm. I think actually it's it's just always funny kind of going through these motions because I remember when Opus 4.6 came out, there were some of the complain complaints of like this model's doing too much. It's thinking for too many tokens. It's like taking too many actions. So, then we all shifted our prompts over and now I think we just have to kind of dial them back in again or add in those instructions to actually be thorough and take more actions. So, give me some specific examples of what you think this is it's amazing at. Yeah. Um, one in particular that I love is just operating on uh, tasks in the background. So, I found that this model is a lot more coherent um when it's having to do these like long-horizon tasks, kind of your like typical, you know, like meter, how long is it going for type of thing. Um that works really, really well with a bunch of the new sh- stuff we've been shipping recently, especially like um Claude code on desktop. So, can you give me an example of a specific type of like long-running background task like that that you're that you use? Yeah, so one that I just did recently um that is kind of uh interesting is I was trying to build this like apartment hunting website uh for my girlfriend. She's trying to find a new spot. Um and I found that just overnight I told the model like, "Hey, I'm going to bed. I want you to just work autonomously and build this thing, and when I wake up, I want to see a like complete dashboard that's pulled all the latest apartments to these specs, and it like shows different um uh things in different areas that my girlfriend wants to look into around like Craigslist and Zillow and all this, and then it runs on a schedule, and so every, you know, twice a day it's pulling new things." And I found that actually just like giving the those instructions of like, "Hey, I'm going to bed, and like I just want you to work and spend a bunch of tokens and like figure this out," actually allowed me to come back to an output that was like more refined, and it did those iterative cycles. I look back at the transcript, and it instead of like stopping at that first go-round, it's like, "Okay, I have time here. I don't have that like stress of having to finish. I'm going to just keep working on this thing." Um so, I thought that was really cool. I'm I'm I literally just told 4.7 I'm going to bed. Um I think that's a really good little little trick. Yeah, see see if it it comes back to something better there. Um another thing I love is on the vision side. So, we expanded our our support for bigger images, larger images, higher resolution. And one fun thing that we were kind of bashing on is just its ability to kind of play like Where's Waldo type games. So you give it an image where it has a lot of like pixel tiny little details and can it find like Waldo or can it find like the uh random bottle on a shelf of a ton of different things. Um that's been really sweet to see and that actually translates into a lot of things we care about as well. Like even though that's a toy example, there's a lot of cases where you're trying to get it to iterate on a front end and maybe like a button is slightly misplaced or a misaligned or there's like a tiny little detail that Opus 4.6 might have missed. Uh Opus 4.7 will catch those. Um so that's been great for a lot of our like visual design output work from front ends to things like slides as well. Does that translate? So it seems like it's good at recognizing like what's what's going on in an image? Does that also translate to its ability to then build something that matches uh a file or figure out what it should build? Exactly. Yeah, so uh a big thing we've been testing is like how well can it replicate like a front end or a design? Um so we'll pass it in like a screenshot and then uh test it on its ability to like uh replicate that. And that's been pretty fun. Um it it is quite amazing how well these models can do things like that these days. I it just like blows my mind the like rate of progress especially compared to like 6 months ago on these on these tasks. Hm. That's really cool. Um anything else other than so like long long running tasks, images, what what else should we try? Yeah, um one thing especially that I've been focused on in my work is like kind of real world tasks. So this model's really good at anything spreadsheet, doc, PowerPoint related. Um I'm not sure how many spreadsheets or or slides you guys are making these days, but \>\> Yeah, okay. Okay, cool. One of the things it did best at is I gave it our P&L and I asked it to write our investor update just from the P&L. It did a really good job. Um like I can show you uh let me see how where it's where it is. I I'll I'll I'll I'll pull it up and and show it to you, but it's it basically it it like wrote effectively what I what I wrote and I I think it had this um really nice I'm going to name the link. It had this really nice blend of it's good at the doing the analysis and finding all the numbers. And then because it's writing style is a little bit more straight ahead and it doesn't like make stuff up as much Yeah. it didn't like make as many conceptual leaps as other models would. So like this is this is the investor update it wrote and I literally just said here's our P&L, here's the investor update. Mhm. And you know, it did all it got all the numbers right like all these numbers look approximately right and it broke it out in a way that I might have broken it out and it you know, did all of our different products and it like figured out what we're going to focus on. It's just like it's it's pretty good. Um \>\> Yeah. And it's something that's like saves would have saved me a lot of time. Yeah, one thing we've heard in in a lot of our um feedback with some of our customers is that it feels a lot more like a co-worker in some ways and that translates in kind of these non-coding tasks as well, but also in the way that it sometimes does push back on you and I've actually been kind of surprised at moments when it's like actually this is you know, maybe this is not the way to do it and I'm like oh, that's interesting that like this model is like pushing back on me like that. Uh I noticed that with a writing task I had to do. We have these different modules, these different kinds of content and I gave it some out some like raw material and said make this into this kind of module and it said actually it doesn't make sense as that kind of module. It needs XYZ and I was like wow, that's actually useful feedback. Yeah, so I think that's like a personal preference on the behavior, but at least for me I really appreciate that that like I can trust the model will push back on me if I'm on the wrong path and not just like keep going along with what I'm saying. Um so that's been another thing I've really enjoyed with with the model. That's really interesting. So, I want to go back to this background thing. It seems like there's some new awareness of oh, I have an amount of time and an amount of tokens within which to like do this. And if I feel like I have less time, then I'm going to I'll do less of a thorough job. And if I feel like I have more time, I'll do more of a thorough job and spend more tokens and more time. Um that seems really important and really valuable. How long is this generally running? Like is there is there some sort of has the horizon moved in terms of like how long you know, you're seeing Opus 47 run versus 46? Yeah, I I think it's hard sometimes to like put it into a quantifiable term like that because it's like the the wall clock time can like depend on like a lot of different variables. Yeah. Um but usually it's it's been longer. Um I see it like taking more turns. Of course, you do need to like specify that in sort of instructions. So, one thing that we have have prompted it to do in some scenarios is like be very thorough. Don't mind making a lot of tool calls. Don't mind making a lot of changes to files and that sort of thing and just incentivizing it to do that. Um effort levels are also a big piece of this model. So, we put a ton of work into making sure that effort levels actually control token usage better. And we've added in a new level, this extra high level, which is in between high and max. Um so, dialing those up and down can kind of control this behavior too in a uh in a way that's different than just a prompt only tweak. How would I know when to use like an a max versus extra high versus high effort level? Yeah, so kind of my like conceptual framing for this is like max is like just go gangbusters. Like I don't care at all about your token usage. I want you to just try for as long as you can with as many tokens as you can, as many tool calls. Um that'd be like max. So, we run a lot of like our benchmarks at max effort. Um extra high is where we actually settled our defaults for Cloud Code. Um we found that this really gave a great experience, especially as a lot of users are shifting more towards these asynchronous tasks. We want to make sure that the model is working for longer. Um high, medium are really sweet spots for kind of achieving that Goldilocks balance between token usage and trying to get a a decent amount of effort into a a prompt or a task. And we found that that is really good for a lot of um things maybe in the like non-coding domains. If you're doing like chat or things like that, that's where you want to drop it down maybe to that level of effort. Um but everything at Gentic, I would stay higher in that range. Got it. So, I think like, you know, probably a thousand people have joined since you got here. And I I want to respect your time. So, if you have to go at any point, just like give me the heads-up. \>\> I do need to drop it. Lunch is pretty busy in a second, but if you have a last question, I'm happy to answer it. Uh okay. Well, I last question, any uh like any parting any parting thoughts for us? Any words of wisdom as we are going forward? You know, we know you've talked about background tasks and images and effort levels. Um what else any any last thing that that you're excited about that we should know about this model? Um yeah, I mean, I would say that the the rate of improvements at two things. One is the rate of improvements are not slowing down anytime soon. I think that's always important to emphasize, like um the frontier of what's available is continuing to grow and expand. Um the second thing is I think like there is this paradigm shift happening from this collaborative work to asynchronous work. And this is important for both the users to know, but also for people building on these models. Like there's a mindset shift to get around in terms of like becoming more comfortable with exposing products that allow for this sort of behavior, for making sure that you're incorporating effort levels and prompt changes into the mix. Um so that you can actually stretch these models to their like full degree. I think we're like quickly saturating a lot of that like bottom half of tasks across models. And now you're starting to reach into the level of like these things are doing real tasks, they're owning real areas, and you got to push the models um both on the prompt side and the task side to really see those gains. I love that. Alex, a pleasure to get to chat. Thank you so much for joining. Thanks for building the model. I'm excited to get to use it over the next Thank you. Thank you. And always feel free, anybody watching this, to uh tag me. Um I'm on on X uh Alex Albert two T's or something. Um and I am happy to respond to any feedback. Just keep it coming my way. Thanks, guys. \>\> Awesome. Thanks, Alex. Bye. All right, folks, you heard it here from Alex. A couple thing Okay, here's here's my big takeaway. Katie, you tell me what what your takeaways were. We've been testing this model for the last, I guess, almost 2 hours now. And our initial thoughts have been like, "Ah, like it's not necessarily as good as at Opus 4.6 at a lot of things." And what I took away from his um uh his chat with us is that's to be expected. The way that this model works is a bit different from 4.6. 4.6 uh it filled in a lot of gaps. It like read between the lines. And this model uh is a lot more literal and a lot more detail-oriented. And the more um and the more you can um Interesting. This Someone's Someone thinks that my microphone sounds like I'm on the toilet. Um I I know. Can you I think you sound fine, Dan. Great. Perfect. Um cuz we've we've been having some microphone issues. Anyway, um so the more So, you should expect all your old old prompts to break, and the more you can push this model to more explicitly tell it exactly what you want, the better it's going to the better it's going to get. And what I would um the a good example from the uh from the chat that we just had is telling Opus 4.7, "I'm going to bed." So, that's one of those things that another model, like an older version an older model, would not know, "Oh, great. I have more time, so I can spend more tokens." But apparently, according to Alex, if you say I'm going to bed, it gives it license to use more tokens, use more tools, and keep going until it's actually done. Uh whereas without that bit of detail, it might assume, "Hey, like I have I have less time, so I'm going to do a smaller slice." So, there are probably all of these little like dials and whistles now in this new model Mhm. that if you're if you know what to ask for, you might be able to unlock a lot of its power, but if you don't know what to ask for yet, and you're using an old 4.6 prompt, you should expect the performance to be worse. Um and again, we've not been testing this for a long time, so I'm I'm sort of take I'm sort of synthesizing what he said and some of our experience so far and telling you what to expect, but I I I think that we'll have a lot more in the next in the coming days on this. Katie, anything anything else that that I'm missing that that we took away? I think I would just like underscore like the effort levels, and it really sounds like Alex is saying that like like mapping the task that you want the model to do to like the effort level that it needs is something that's worth thinking about. Like and this is something like I think a lot about um learning to drive manual versus driving stick shift. That's kind of the the image that comes to mind for me for this is like you're going to get better performance. Um, you know, a more like fine-grained control from like, you know, switching the effort levels and like selecting the effort level that matches your task than you would just like having a one per one-size-fits-all effort level just going at the task. But it's going to take some experimentation, um, I think to to know which kind of tasks warrant which kind of effort level. Yeah, I agree. I totally agree. Um, it's also a little bit more just like the writing is a is a bit more straight ahead, but maybe if we say, "Hey, like I want you to write like Annie Dillard." I'd be curious what it what it might do. I guess your I guess your writing prompts have a lot of have have a lot of voice stuff in them, right? So \>\> Yeah, I um I did have it go over like I had it do a revision on the intro that I had it write with my tastemaker voice, which has clips from a lot of writers that I admire like Jia Tolentino, who has more sort of like architectural, um, writing patterns with like depending clauses and, you know, and things like that. And then like John Green's in the mix there, too. And like it came it didn't come out more like fluid then, but like once it had all of that context, I just pointed all of that context at it and said, "This is how I want you to write." Um, so I think that's another thing. Yeah, like the more bundles of information you have to give the models so you're not having to completely prompt from scratch. I think that's something else that we're going to see like like, you know, compounding returns from as just the more context you have to give the model, the more you can give it to go on to get the outcome that you want. Really, really interesting. Okay, so if you are here, we are going to end this stream soon so we can get back to doing some real uh, testing. And some writing. Live working. Um and uh you should subscribe to Every. Every is the only subscription you need to stay at the edge of AI. every.to. We do a lot of cool stuff here at Every. So, when new models come out, we do day of vibe checks. This is your day of vibe check for this model. Normally, we get the models early. We're already testing some some early models that I'm pretty excited about that we can't tell you about, but that will be dropping soon. Normally, we get them early. Today, we're doing a live vibe check. That's why you're seeing us figure out this model in real time, but we drop um a new newsletter every day with um lots of little nuggets about what we're learning and what we're doing in AI. Um today's uh this is actually I guess from yesterday. Yesterday's newsletter, um we talked about um what might come after LLMs. We did a little vibe check on Claude managed agents, uh which we've started to put into into production with Sparrow, which is our writing app. Um we uh you know, did some did some philosophical discussions of the jagged frontier. So, like lots of good stuff every day from us on the newsletter side. We also have a suite of apps that we build that help us write interesting stuff about AI. So, we have Sparrow, which is our AI writing assistant. We have Cora, which is our AI email assistant. We've got Sparkle, which we just launched a new version of like 2 days ago. Uh Sparkle is your agentic Mac cleaner. Um it's pretty sick. It makes my computer clean. We've got Monologue, which is a speech-to-text app like Whisper flow. We've got Proof, the the sloppy uh uh markdown editor that uh that we've been trying to disloppify with with with Opus to vary \>\> It's so much better now, Dan. It's very It is It is a lot better. If you're using a coding agent, um Proof is just such a good way to get your plan um uh any of your plan documents into a like live collaborative document that you can share with other agents and with your friends and co-workers. We've also got Plus One, which is our hosted uh one-click open claw. So, all of this, all of the stuff that we write, all of the apps that we make, all of the training and courses that we do, it's all included for one price under the Everly subscription. You should check it out everly.co/subscribe. Uh we will have a full vibe check tomorrow. Um and probably an even fuller one uh over the weekend or on Monday once we have a little a few more days to really process what's going on. Until then, it is always a pleasure to get to vibe check with you. Um thank you all for joining. Uh and remember, stay hydrated. Katie? See you later. See you later. Bye.