A product engineer's talk on what it takes to move AI features from prototype to production, based on Granola's experience shipping a chat feature into its meeting notes app. Covers the limits of generic AI features: web search as a single tool call blowing up context, cost, and reliability across provider updates, and why one prompt cannot serve diverse user roles (sales, engineering, HR). The response was building iteration infrastructure: internal tracing UI exposing tool calls, search trails, reasoning traces, and costs; refactoring the Electron frontend into a web app for per-PR preview links; and automated feature verification with Cursor. The central argument is that iterative feedback loops, not any single technique, determine AI product quality. # Channel Presentation Observations — "Feedback Loops are All You Need — Mehedi Hassan, Granola" ## Video Format & Setting - This is a **recorded conference/ meetup talk** with a live audience, evidenced by: - Opening greeting: *"How's it going, guys? I'm Maddie."* - Closing applause segments and a Q&A: *"Any questions?"* followed by an audience question about Electron vs. Tauri. - Single-speaker presentation with slides/demo ("I'm just going to give you a quick demo"). - "Granola" appears to be the company/product (rendered as "Cronulla" in the transcript — likely a transcription artifact of the same product name); the speaker identifies as *"a product engineer at Cronulla [Granola]"*. ## Intended Audience (Observable) - **Product engineers and technical practitioners** building with LLMs: - *"This is not going to go deep into our engineering stuff. So, if you have for me to like go into LLMs and stuff, it's not going to happen."* - Assumes familiarity with concepts like Electron's main/render processes, IPC, OpenTelemetry, Cursor, PRs, CI, Figma. - Audience shares his experience frame: *"experiencing LLMs change engineering and everything else just like many of you."* ## Claimed Scope / Self-Description - **Speaker role claim**: *"I'm a product engineer at Cronulla"* with long tenure: *"coding since jQuery was cool. I've seen React kind of change front-end engineering."* - **Product self-description** (creator claim, delivered as company pitch): - *"Cronulla is an app for getting your work done... we're a meeting notes app where we sit on your doc... real-time transcription... at the end of your meeting, we can give you really awesome notes."* - Quality claim: *"We have the best-in-class meeting notes no matter what role you're in and it doesn't get in your way."* - Stated philosophy: *"that's been like our product philosophy since day one."* - **Talk scope claim**: Practical product engineering process, not deep AI research — *"we're going to talk about some product engineering stuff that we've been doing."* ## Demonstrated Presentation Patterns (vs. Claims) - **Show-then-explain structure**: live product demo before the talk content ("I was recording the previous talk right here and you can see picked up literally everything the presenter said"). - **Candid, informal tone**: colloquial delivery ("Obviously," "like," "the cool thing"), self-deprecating framing ("This is basically a long talk to tell you that the answer isn't to one-shot better"). - **Concrete internal case studies over theory**: custom tracing tools, web-shell conversion of their Electron app, Cursor testing PRs — presented as things they actually built: *"we started building our own tracing tools."* - **Willingness to disclose limitations honestly** (observable pattern): - On external dependency risk: *"overnight, they shipped an update and for some reason web search degraded and it was completely out of our control."* - On tooling alternatives: *"you can use OpenTelemetry or like other providers. But we essentially just like save things to a DB."* - In Q&A, admits mixed results with Tauri: *"we didn't really see massive performance gains."* ## Takeaway Framing - The talk's central message is process-oriented: building **feedback loops** with LLM tooling — *"it's about figuring out how you can make that feedback loop where it kind of feels like playing a tennis game with LLM"* — rather than one-shotting features. ## Metadata Note - No curated metadata fields were present beyond the title; the title attributes the talk to **Mehedi Hassan** (transcript self-identifies as "Maddie," a plausible nickname) of **Granola**. No further identity context was supplied. # Delivery Observations — "Feedback Loops are All You Need" (Mehedi Hassan, Granola) ## Overall Tone & Register - **Casual, conversational conference-talk style**: opens with > "Cool. How's it going, guys? I'm Maddie." - **Conversational hedge words used frequently**: "you know," "like," "kind of," "basically," "essentially" appear throughout, e.g. > "So, essentially, what we did is we took our Electron app..." - **Upfront expectation-setting with humor**: > "This is not going to go deep into our engineering stuff... I'm warning you right now. So, you know what's coming." ## Recurring Vocabulary & Phrases - **"One-shot" / "one-shotting"** — repeated as both a technique and a thematic pivot: - > "So, let's say I built, you know, like a one-shot this chat system." - > "you can actually one-shot these things" - > "the answer isn't to one-shot better" - **"Black box"** — used as a recurring problem framing: - > "LLM behavior is largely seen as like a black box" - > "figuring out this black box" - > "rather than just like a black box and hoping" - **Filler intensifiers**: "literally" (> "picked up literally everything", > "our founder literally goes into like the details"), "obviously" (many occurrences, e.g. > "And obviously now experiencing LLMs change engineering"). - **"Super" as an intensifier**: > "can be super hard", > "the cooler part of this", > "feels super good", > "has been super nice". ## Teaching Habits & Structure - **Problem → solution pattern**: repeatedly poses a pain point then shows Granola's fix, e.g. web search cost problems → building custom tracing tools; testing friction → web shell for the Electron app. - **Live demo as opening teaching device**: shows the product transcribing the *previous talk* — > "I was recording the previous um talk right here and you can see picked up literally everything the presenter said." - **Concrete numeric examples to ground abstractions**: > "each chat could be costing you like 10 pence"; > "one feature looking very different in four different users." - **Closing metaphor**: the talk's title concept lands via analogy — > "it kind of feels like playing a tennis game with LLM." - **Self-aware deprecation of the talk's length**: > "essentially, this this is basically a long talk to tell you that the answer isn't to one-shot better." ## Q&A Delivery - Brief, direct answers; concedes prior experimentation without hedging excessively: > "We've tried Tauri before as well and we didn't really see massive performance gains." ## Uncertainty Notes - Pauses, pace, and gesture cannot be assessed from transcript alone; tone claims rest on word choice only. - The transcript phonetically renders the company as "Cronulla" — pronunciation-vs-branding cannot be verified from text. # Observation Document: "Feedback Loops are All You Need" — Mehedi Hassan (Granola) ## Speaker - **Mehedi Hassan ("Maddie")**, product engineer at Granola (transcript renders it "Cronulla"), speaking at a product engineering event. He codes "since jQuery was cool" and has observed React and now LLMs change engineering. ## Scope Disclaimers (stated) - The talk would **not** go deep into engineering/LLM internals: "if you have for me to like go into LLMs and stuff, it's not going to happen." ## Product Context: Granola - Meeting notes app that "sits on your doc" during meetings. - Accesses **system audio, microphone audio, system transcription** → real-time transcription and end-of-meeting notes. - Users can write their own notes on top of the transcription, so the final result is closer to what one would "actually write on a notepad." - Demonstrated live: generated notes emphasized a handwritten point ("20% overlap") more than the raw transcript did. - Claim: **"best-in-class meeting notes"** for any role; product philosophy: it "doesn't get in your way." ## Claims and Reasoning ### 1. Generic chatbots fail in production - A one-shot chat feature (Q&A over meetings) in his fake Granola app produced common failures: web search too slow ("give me a list of cities"), wrong-tone follow-up emails, and a coaching request answered with football-coach content. - Framed as "very very common problems" with generic chatbots. ### 2. Web search is harder than providers claim - Providers present web search as "a line of code"; Hassan argues there are complications: - **Token usage/cost** can "bubble up," ~10 pence per chat — infeasible at millions of users. - **Provider dependency/lack of control**: a model provider shipped an overnight update that degraded web search "completely out of our control"; they only understood it via switching providers. - Existence of "billion-dollar companies who do web search" is cited as evidence it's "much more than just adding a web search tool." ### 3. One prompt can't serve all users - Different roles expect different outputs: sales → deal focus; engineering → action items, blockers, Linear tickets; HR → something else. - LLMs are "stubborn"; behavior is "largely seen as a black box," but Granola wants to understand it deeply. ### 4. Building internal tracing tooling - Granola built **their own tracing tools** (one-shot with LLMs) giving complete visibility over tool calls, reasoning, and cost, structured exactly as they want. - UI serves non-engineers too (product, data, CX) — no need for complex CloudWatch queries. - Implementation (basic version): save to a DB, wrap around AI SDK, front-end is the most important part; OpenTelemetry or other providers possible alternatives. - The founder personally follows agent loops front-to-back to diagnose failures. - Value: moves from "this output feels off" to "exactly what failed," enabling targeted iteration. - Previously infeasible without LLMs: "you simply wouldn't have the time" — now you can build tooling that serves your needs. ### 5. Web shell for parallel variant testing - Problem: desktop (Electron) app runs one instance; friction testing features and variants in parallel, unlike web apps; coworkers had to run Electron locally with dependencies. - Solution: turned the Electron front-end into a **web shell** deployed online; PRs now produce preview links via CI. - Technical approach: abstracted **IPC APIs** to fall back to web standards in web environments; moved React APIs (routers, sessions, query layer) to web standards; render process became **Electron-agnostic**. - Additional speedup: since LLMs "can now self-verify their work," **Cursor tests PRs and uploads screenshots** into the PR. - Claimed outcome: dramatically faster development; variants can be felt "in practice rather than just seeing it in Figma," giving conviction in what ships. ### 6. Core thesis - > "The answer isn't to one-shot better. It's about figuring out how you can make that feedback loop where it kind of feels like playing a tennis game with LLM." - Goal: the end product feels like "magic rather than just like a black box," with conviction that shipped features connect with users. ## Q&A - **Question (audience):** Thoughts on re-platforming from Electron to Tauri? - **Hassan:** Considered multiple times; Electron currently serves them well ("super nice"); tried Tauri but saw "no massive performance gains," which is what they care about most. Discussed and prototyped, but not shipped. ## Qualifications & Tensions Preserved - Hassan qualifies claims repeatedly ("obviously," "this is a very basic example," "you might think that this is a lot of work, but it's actually quite simple"). - Tracing example is explicitly basic; alternatives (OpenTelemetry, SaaS providers) acknowledged. - Costs and provider instability described as unresolved external constraints, not solved by Granola. # Speaker Attribution Record ## Video **Title:** Feedback Loops are All You Need — Mehedi Hassan, Granola **Video ID:** `01a15ca9-73fe-741b-87a1-2fe10c2971e4` ## Speakers and Roles ### Primary Speaker - **Self-identified as:** "Maddie" (likely **Mehedi Hassan**, per title metadata — the transcript itself only says "Maddie"; the name linkage to the title is a metadata-based inference, not stated in the transcript) - **Role:** **Presenter / product engineer at Granola** (spoken as "Cronulla" in the automated transcript — apparent transcription variant of "Granola") - **Role type:** **Creator** (of the talk content) and **company representative** (speaking on behalf of Granola) - **Self-description:** Coder "since jQuery was cool," has witnessed React's impact on front-end engineering ### Other Speakers - **Audience Q&A participant** (unnamed): asks about re-platforming from Electron to Tauri - **Audience:** applause; one voice saying "Yay!" at the end ## Promotional Segments (Flagged) - **Product demo of Granola:** early portion demonstrates the meeting-notes app live, including: - Real-time transcription of system audio, microphone audio - Generating meeting summaries - Claim: **"best-in-class meeting notes no matter what role you're in"** — this is a self-promotional product claim by the presenter, not independently verified - Framing statement: "This is Granola. We have the best-in-class meeting notes..." — **explicit promotional language** ## Referenced Third-Party Products / Sources - **jQuery, React** — mentioned as historical engineering context - **LLM providers** (unnamed) — web search tooling, token cost concerns, and an incident where an overnight provider update degraded web search - **Cursor** — cited as automatically testing PRs and uploading screenshots - **Electron** — Granola's desktop app framework; main/render process architecture discussed - **Tauri** — raised in audience Q&A; presenter states the team tried it but saw no significant performance gains and has not shipped it - **OpenTelemetry** — mentioned as an alternative for tracing tooling - **CloudWatch** — referenced as the previous/alternative approach for querying failures - **Figma** — mentioned as where designs were previously viewed before variant testing - **Web search / SaaS providers** — general references; "billion-dollar companies who do web search" ## Third-Party Claims of Note - Claim that LLM labs present web search as "a line of code" while real integration is more complex — presenter's characterization of unnamed providers - Claim that "LLMs can now self-verify their work" — presenter's claim, unattributed ## Comment Availability - **No information provided** in the source material regarding comment availability. ## Uncertainty Notes - The speaker's name is given only as "Maddie" in the transcript; identification as Mehedi Hassan rests on the video title metadata. - "Cronulla" appears throughout the transcript as an apparent mis-transcription of "Granola" (company name in the title). - No curated metadata beyond title and transcript was supplied. \>\> Cool. How's it going, guys? I'm Maddie. We're going to talk about some product engineering stuff that we've been doing at Cronulla. This is not going to go deep into our engineering stuff. So, if you have for me to like go into LLMs and stuff, it's not going to happen. I'm warning you right now. So, you know what's coming. Cool. So, I'm a product engineer at Cronulla. I've been, you know, coding since jQuery was cool. I've seen React kind of change front-end engineering. And obviously now experiencing LLMs change engineering and everything else just like many of you. For those of you who don't know, Cronulla is an app for getting your work done. Essentially, we're a meeting notes app where we sit on your doc like it is doing right now. And it has access to your system transcription system audio as well as your microphone audio, which means we have real-time transcription. And then at the end of your meeting, we can give you really awesome notes. So, I'm just going to give you a quick demo. So, I was recording the previous um talk right here and you can see picked up literally everything the presenter said. And the cool thing about Cronulla is that you can also write your own notes on top of what the transcription is saying. So, the final result is more aligned to like what you'd normally actually write on a notepad. So, I'll go ahead and generate the notes here. And you'll see that this will go ahead and write a really good summary. And as you can see like I wrote down this 20% overlap thing and it focused more on the output, right? So, this is Cronulla. We have the best-in-class meeting notes no matter what role you're in and it doesn't get in your way. And that's been like our product philosophy since day one. So, we ship a lot of AI features in Cronulla and our product is known to be again not to get in your way. So, let's see what happens when you put a simple AI feature into prod. I'm going to kind of give you an example with this chat feature that we have. This is a feature that already exists in Cronulla. You can ask questions about a meeting that you just had and across a bunch of different meetings or like shared context as well. And Cronulla will try answer it to the best of your ability. So, let's say I built, you know, like a one-shot this chat system. It's very easy to do. And I put it into production in my fake Cronulla app. And then as soon as users hear, you know, it's like a call me in a minute. So, give me a list of cities. Web search is too slow. It's not writing follow-up emails how I normally write my emails. I asked it to coach me about my meetings and it's telling me meetings about my football coach. Obviously, these are very very common problems that you're going to run into when you make a generic chatbot. So, how do we get around this, right? So, what we've seen is like molding the LLM to work to your specific use case can be super hard. And one of the examples is is web search. So, web search for most LLM providers looks like a line of code. You simply add the web search tool and you expect it to just work. That's what the labs want you to believe, but once you get into it, there's lots of other complications. So, for example, the token usage and token cost can bubble up quite a lot, especially for complex queries. It's going to blow up your context. And each chat could be costing you like 10 pence. Obviously, at scale when you have millions of users, this is not really feasible. And then, you know, like the web search providers are also like completely up to the labs as well. So, for example, in in our development, what we see was like was we were using a model for a good amount of time. And then overnight, they shipped an update and for some reason web search degraded and it was completely out of our control. And we generally had no idea like what was going on apart from just like switching providers. But we want to have more control over that because it affects our user user experience. And you know, like there's literally billion-dollar companies who do web search. So, that kind of tells you that it's what's much more than just adding a web search tool to your LLM pipeline. The other thing that's super important for apps like Cronulla is the output. So, the summary that you saw was pretty good for what I would expect, but someone in sales might expect more of a deal focus. Someone in engineering might expect like action items, blockers, or like linear tickets. HR might want something completely different. And the thing is that one prompt can't generally serve everyone. And you know, LLMs are stubborn and we need to figure out how to get inside them and make them work how we want it to work. And yeah, as you as you know, like LLM behavior is largely seen as like a black box, but we want to kind of go very deep into the details and figure out exactly what's going on. So, what we did recently at Cronulla is we started building our own tracing tools. And obviously, thanks to LLMs, you can actually one-shot these things. And this is where one-shotting is kind of nice. And so, we built our tooling tracing tools here where we basically have complete visibility on the tool calls straight from the beginning to the end. So, we have full visibility over the individual tool calls, why it's making those tool calls, the search tools, the reasoning tools, the cost structured exactly how we want it. And the most useful part of this is that we structured the data exactly how we want it. And the UI is built to like serve our our employees internally, not just like engineers, but also product, data, and like CX, and everyone. So, you don't have to like, you know, go into CloudWatch and do like very complex queries to figure out why something failed. And that's been like the key for us in like figuring out this black box. And previously, obviously, building this kind of tools would be like up to using a SaaS provider. And it simply wouldn't you simply wouldn't have the time. But now you actually can spend time building this tracing tool that actually serves what you need. And this is obviously a very basic example, but obviously you can use OpenTelemetry or like other providers. But we essentially just like save things to a DB, wrap around like AI SDK, and then the front-end is like kind of like the most important part cuz that's what people are going to use to figure out what breaks and what doesn't. And we literally have like our founder literally goes into like the details like following the agent loop completely front to back to figure out exactly what went wrong. So, then at the end of this, you can actually figure out like, you know, this output feels off to like exactly what failed. And then when you iterate, you can improve on those things. But as I said earlier, this is going to be more more than just like basic LLM stuff. And LLM behavior is obviously part of the picture. The how users interact and experiences your product is also very important. So, with LLMs, you can one-shot more things and you can have more variants, which we like cuz we can experiment with different features. We can experiment with like one feature looking very different in four different users. But the problem for us specifically at Cronulla was that we are a desktop app, which means you can only run one instance of the app at a time. And there was a lot of friction when it came to like testing new features, different variants, and actually testing those in parallel. \>\> \[clears throat\] \>\> So, before, you know, before you'd have to like run the Electron app locally, install the dependencies, and test things. If you wanted a coworker to to test those changes, you'd have to get them to do those things as well. We We didn't have the same luxuries as like web apps do. So, essentially, what we did is we took our Electron app and we turned the front-end of the Electron app into a web shell. And this was deployed online. So, now our CI, whenever we open a PR, we get a preview link and we can go and test those things. And this generally sped up our development time so much more. And like the cooler part of this is that because LLMs can now self-verify their work, these guys are now like once we open a PR, Cursor goes and tests it, uploads a screenshot into our PRs, which speeds up the testing so much more. And again, this is like you might think that this is a lot of work, but it's actually quite simple. So, what we did is for those of you are not familiar with Electron, there's obviously a main process and a render process. The main process works with the system APIs and the render process is basically your front-end. And essentially, we abstracted our IPC APIs, which is the system APIs, to fall back to web standards when we're in the web environment. And similarly with React APIs as well, like routers, sessions, and query layer, we move those to the web standards. And essentially, this just made the render agnostic of of Electron. And we can just simply run it as a web app. So, this has helped us on top of the LLM improvements was like we were able to just like change and like test like one feature in like multiple different variants. So, like whatever the end product is actually feels super good cuz we know that we've tried so many different variants. And we actually felt those products in in like in practice rather than just like seeing it in Figma. So, essentially, this this is basically a long talk to tell you that the answer isn't to one-shot better. It's about figuring out how you can make that feedback loop where it kind of feels like playing a tennis game with LLM. So, the end product feels more like magic rather than just like a black box and hoping that the feature that you're going to release works well with customers and having that conviction that what you're shipping is actually going to connect to the users. Thank you. Any questions? \>\> \[applause\] \>\> What do you think about re-platforming from Electron to Tauri? We've thought about moving to Tauri a couple times. Um I think the way Electron serves us right now has been super nice. Like the APIs changing quite a quite a lot. We've tried Tauri before as well and we didn't really see massive performance gains, which we which is what we care about the most. So, yeah, it's been discussed before. We've played around with it, but haven't shipped it. Cool. Thank you, guys. \>\> \[applause\] \>\> Yay! \>\> \[music\]