Charles Dickens, research scientist at Snorkel AI, describes how Snorkel AI and UC Berkeley's Sky Computing Lab trained a small language model to outperform a much larger one on financial question answering. The team built FinQA, a financial QA dataset with expert-validated answers derived from SEC 10-K filings, and observed that even frontier models hallucinated table schemas, flooded their own context, and repeated failed strategies. Using the open-source rLLM framework and reinforcement learning with a simple pass/fail reward, they trained a Qwen3 4B model for under $500 that reached about 60% accuracy versus 51% for a 235B parameter model. Skills transferred to harder multi-table questions without degrading general tool use. A key finding: simpler training data and simple binary rewards worked best, since the bottleneck was disciplined tool use rather than reasoning depth. Topics include building and verifying FinQA, model failure modes, RL training with rLLM, and why simple setups outperformed complex ones. # Channel Presentation Observations ## Video Overview - **Talk format**: Conference-style technical presentation by Charles Dickens, introduced as a **research scientist at Snorkel AI**, on training a 4B-parameter model to outperform a 235B variant on financial tasks. - **Topic scope**: RL fine-tuning, benchmark/data construction (FinQA), ablations, cost breakdown, and enterprise agent evaluation — with a closing segment advertising grants and hiring. ## Explicit Self-Description (Creator Claims) - **Organizational identity**: > "We're a frontier AI data lab" — team with founders from labs at University of Washington, Wisconsin, Stanford, plus internal researchers and FTEs. - **Claimed scope of work** (three pillars): 1. **Open research and benchmarks** — "constantly contributing to open-source benchmarking"; this project falls under this pillar. 2. **Frontier live data and environments**. 3. **Enterprise deployments**. - **Enterprise grounding claim**: > "that's at the intersection of academia and industry research, but also grounded in enterprise deployments." - **Grant program claim**: > "our open benchmarks grants, which funded this project. And this is a multi-million dollar commitment to producing open benchmarks." - **Hiring**: "we're hiring in professional research... engineers." - **Cross-domain replication claim**: > "we've seen this and repeated this result in other areas including health care, law, and insurance" — a channel-level claim extending beyond this talk's evidence. ## Observable Presentation Patterns (Demonstrated) - **Open-source orientation**: Training scripts, synthetic data, and artifacts released on GitHub and Hugging Face: > "this is all open-source and available to you." - **Collaboration emphasis**: Credits UC Berkeley Sky Computing Lab co-authors by name ("Manon Rupta, Shujian Tang... Bhavishya and Chris Glaze"). - **Rigorous methodology display**: Multi-stage data pipeline with three-layer verification (programmatic checks, independent agent review, expert manual review); held-out benchmark with no company overlap across splits. - **Cost transparency**: Full training budget disclosed — "under $500... roughly 420 and $40 for a judge... on eight H100s." - **Candid negative results**: Reports that simpler approaches (binary reward, single-table-only training) beat more sophisticated variants ("simpler was better"). ## Intended Audience - **ML/AI researchers and practitioners**: Assumes familiarity with GRPO, RL fine-tuning, benchmarks (BFCL), scaling laws, H100 clusters. - **Enterprise AI decision-makers**: Frames motivation around financial institutions' needs — legacy APIs, auditability, deployment economics. - **Academic collaborators**: Grant opportunities, co-authoring, and open-artifact publication are pitched explicitly. - **Job seekers**: Hiring call-out near the end. ## Claimed vs. Demonstrated Distinction - **Claimed**: Channel-wide reliability of the "small specialized beats large generalist" result in healthcare, law, insurance — only insurance underwriting evidence is referenced (prior CAIS conference work). - **Demonstrated**: Single case study (FinQA 4B model) with specific metrics (4B ≈ 60% pass@1 vs. 235B at 51% on 290 held-out samples). # Delivery Observation: Charles Dickens — "Small Models, Big Results" ## Overall Delivery Profile Conversational, presentation-style technical talk with frequent **filler particles** ("uh," "um," "so"), heavy signposting, and consistent credit-giving. The speaker relies on verbal scaffolding ("So," "And finally") to move between sections. ## Tone & Register - **Casual-professional and modest**: "before I get to it, I'm going to take this opportunity to introduce ourselves a little bit" - **Self-deprecating humor tied to his name**: "This is uh the one and only slide with the Charles Dickens reference, I promise, but this is a tale of two models, I'd say." - Uses **playful asides**: "You wouldn't call Terence Tao for a tax audit, um, if you're lucky enough to have him in your contacts." ## Vocabulary & Framing Habits - **Quantified, precise claims** as a default: "almost a 60% pass at one," "roughly 1,000 concurrent environments," "6,900 of those," "under $500." - **Repeated core vocabulary**: "specialized/specialization," "reliability," "tool use," "generalist," "parameter," "frontier." - Recurring rhetorical question structure to pivot slides: > "So, a natural concern is whether this actually transfers to harder problems." > "The next natural question is whether more informative reward signals... could actually accelerate learning." > "The central question, can a 4 billion model... actually compete with its larger variant?" - **Anticipation of audience objections** is a signature move — nearly every result section opens by naming a concern before answering it. ## Recurring Phrases & Patterns - **"And this is / And we"** chains to connect ideas mid-flow, sometimes losing grammatical track: "And these failure modes are not actually just seen in the financial environment." - **Repeated formula "actually + verb"** for emphasis: "actually outweigh raw parameter scale," "actually doubled," "actually erode," "simpler was better... actually again saw." - **"Call out / calling out"** used repeatedly: "I want to call out is," "this is a work that I'm calling out," "I also want to call out that we're hiring." - **"I'll talk about / we'll investigate" forward references**: "I'll talk about opportunities here at the end of this talk, so um get ready for that." ## Teaching Habits - **Thesis stated up front and restated at the end**: opens with "reliability and specialization can actually outweigh raw parameter scale," closes with "small models can successfully be trained with RL when you have quality data." - **Enumerated structures**: "the way we do this is threefold," "three-layer verification process," "the first is... the next is... finally." - **Named failure modes as teaching devices**: "schema hallucination," "context flooding," "poor recovery" — each defined then exemplified ("just calling select star"). - **Generous attribution**: "I got to give credit where credit's due, Manon Rupta, Shujian Tang... as well as Bhavishya and Chris Glaze." ## Communication Patterns - **Spoiler framing to build interest**: "the 4 billion parameter model, spoiler alert, achieve almost a 60% pass at one." - **Conversational closers and resets**: "Yeah, so oops, let me go back"; "All right, now for our training environment"; "Yeah, thank you all." - **Resource-sharing as a closing habit**: repeatedly points to GitHub, Hugging Face, blog, QR code, and grant opportunities. ## Uncertainty Notes - Filler density ("uh"/"um") is high throughout, but per-section comparisons are not reliable from a transcript alone. - The "natural concern" pivot appears at least three times — likely stable, but a single talk limits confidence in whether this is habitual across other presentations. - Live-demo or slide-dependent behaviors cannot be assessed; only verbal delivery is observable here. # Observation Record: Small Models, Big Results — Charles Dickens, Snorkel AI ## Speaker - **Charles Dickens**, Research Scientist, Snorkel AI (sole identified speaker, per title and self-introduction). ## Central Claim - A **4-billion parameter Qwen model** (RLLM FinQA 4B) was trained to **outperform its 235-billion parameter variant** on financial tasks in a simulated financial environment. - Stated results: ~**60% pass@1** for the 4B model vs. **51%** for the 235B model (on the 290-sample expert-curated held-out benchmark). - Headline takeaway (speaker's framing): for enterprise AI workloads, **reliability and specialization can outweigh raw parameter scale**, given high-quality domain/task-representative data, yielding frontier-parity at a fraction of the cost. ## Collaboration and Credits - Partnership with **UC Berkeley Sky Computing Lab**; collaborators named: **Manon Rupta, Shujian Tang** (Sky Computing Lab), **Bhavishya, Chris Glaze**, and the speaker (Snorkel). - All assets — training scripts, synthetic data — claimed **open-source on GitHub/Hugging Face**. ## Snorkel AI Background (speaker's description) - Self-described "frontier AI data lab"; founders from UW, Wisconsin, Stanford labs. - Three workstreams: **open research/benchmarks**, frontier live data/environments, enterprise deployments. ## FinQA Data Construction - **Source**: 10-K annual reports (SEC filings) pulled from the Edgar system. - **Pipeline**: schema/table extraction via **Qwen 3 30B** (~6,900 SQL tables) → one question-answer pair per table, guided by an expert-developed question taxonomy → **three-layer verification**: programmatic consistency checks (metadata/lineage), automated review by independent agents, and expert manual review. - **Splits**: ~4,000 train / 500 val / 290 held-out benchmark; **no company overlap across splits**. - Speaker noted this is a **"first pass"** (qualification). ## Observed Failure Modes of Frontier Models - **Schema hallucination** (assuming tables/columns exist, possibly a pre-training artifact). - **Context flooding** (e.g., `SELECT *` queries overwhelming context). - **Poor error recovery** (repeating failed strategies, e.g., the 235B model). - Speaker claims these recur beyond finance, citing insurance underwriting work presented at the **CAIS conference**. ## Training Setup - **RLLM framework** from Sky Computing Lab collaborators; works with any agent framework, decorator pattern, CLI-first; built-in GRPO, ReinformRL(?) — speaker listed "GRPO, reinforce RL, OOO" as RL algorithms. - Environment: **ReAct loop** agent with tools over ~7,000 tables; **binary reward** via **GPT 5 nano as LLM judge**, reference-based evaluation. - **Base model**: Qwen 3 4B; ~1,000 concurrent environments; total cost **under $500** (~$420 compute + ~$40 judge API) on **8 H100s**. ## Results and Ablations - Base 4B model accuracy **more than doubled** post-training; outperformed 235B variant. - **Multi-table transfer**: tool-use discipline generalized from single-table training to a harder FinQA reasoning set (2–5 tables, sequential planning; ~1,000 train / 120 val / 80 benchmark) **without explicit multi-table training**. - **General capability (BFCL)**: overall accuracy slightly improved; minor gains in multi-turn and memory — specialization "didn't erode" broader tool-use competence. - **Data-mixture ablation**: training on the **simpler single-table set alone gave the most lift**. Speaker's interpretation: "the bottleneck was never reasoning depth, it was just tool use" reliability. - **Reward ablation**: despite expert-designed rubric rewards (weighted intermediate-step components), **simple binary reward performed best** — "simpler was better." ## Speaker's Blueprint for Agent Evaluation Three axes proposed: - **Environment complexity** (realism, dynamism) — "one of the most important axes." - **Autonomy horizon** (testing at real usage lengths; safe autonomous decisions; improving judgment as co-pilot). - **Output complexity** — flagged as **under-explored**; benchmarks overly focused on text artifacts (speaker sees this as an opportunity). ## Broader Claims and Promotion - Speaker asserts the small-model RL result has been **repeated in healthcare, law, and insurance**. - Project funded by Snorkel's **open benchmarks grants** (multi-million dollar commitment); cited successes: agent, "last exam," judgment bench. - Snorkel is **hiring**; website link/QR code for blog and artifacts. ## Qualifications and Tensions Preserved - Speaker's own hedge: the headline result is **specific to a simulated financial environment**, with "general takeaways" asserted as opinion ("I think," "I'd say"). - The "tale of two models" analogy (large generalist vs. small specialist) is presented as the speaker's argument, not settled fact. - The outperformance claim is scoped to this benchmark; speaker proactively raised and tested transfer concerns rather than claiming universal superiority. # Speaker Attribution Record ## Video Metadata - **Title:** Small Models, Big Results: Training a Finance Agent for Under $500 — Charles Dickens, Snorkel AI - **Video ID:** `01a15ca6-b14b-7e53-9112-344a216b47ad` ## Speakers | Speaker | Role | Evidence | |---|---|---| | **Charles** (full name given in title as "Charles Dickens") | **Creator / Presenter** — introduces himself as a research scientist at Snorkel AI | *"I'm Charles and I'm a research scientist at Snorkel AI."* | - **No guests, narrators, or other speakers** appear in the transcript; it is a single-presenter talk. ## Referenced Third Parties (named, non-speaking) - **Manon Rupta, Shujian Tang** — UC Berkeley Sky Computing Lab collaborators - **Bhavishya, Chris Glaze** — Snorkel collaborators - Terence Tao — mentioned as an analogy, no affiliation claim ## Referenced Sources & Artifacts - **Partnership:** UC Berkeley Sky Computing Lab (built the RLLM framework used for training) - **Data sources:** SEC 10-K reports via the **Edgar system**; Qwen 3 30B used for table generation - **Benchmarks:** Snorkel FinQA (expert-validated), FinQA reasoning variant, **BFCL** (general tool calling) - **Models referenced:** Qwen 3 4B (base), Qwen 235B variant, GPT 5 nano (LLM-as-judge) - **Cited prior work:** insurance underwriting results presented at the **CAIS conference** - **Open artifacts:** training scripts and synthetic data on **GitHub**, **Hugging Face** (open-source, per speaker) ## Promotional Segments (flagged) - **Grant program promotion:** speaker "advertises" Snorkel's **open benchmarks grants** ("a multi-million dollar commitment"), directing viewers to the website with application steps. - **Hiring promotion:** *"we're hiring in professional research... engineers."* - **Company/QR promotion:** closing pointer to Snorkel blog and GitHub/Hugging Face artifacts via QR code. ## Availability & Provenance Notes - Comment availability: **not stated** in source material. - No curated metadata supplied; all facts above derive from the transcript only. - Claimed results (e.g., ~60% vs 51% pass@1; <$500 total cost, ~$420 compute + ~$40 judge API on 8 H100s) are **speaker-reported claims**, not independently verified here. \>\> Everyone, thanks for joining. I'm Charles and I'm a research scientist at Snorkel AI. And today I'm going to be talking about how we got a 4 billion parameter model to outsmart its 235 billion parameter variant. And this is a specific result in a simulated financial environment, but I think there are general takeaways that I want to make sure taken home. And the first is for enterprise AI workloads, reliability and specialization uh can actually outweigh raw parameter scale. And this is possible with high-quality data that actually represents the domain and task distribution that you're targeting. And uh this can result in smaller models that can achieve parity with frontier models at just a fraction of the cost. So, before I get to it, I'm going to take this opportunity to introduce ourselves a little bit. This is our team. We're a frontier AI data lab and our team, we have founders from labs at University of Washington, Wisconsin, Stanford, and an internal team of really talented FTEs, researchers, and we're in a fortunate position to play in, by that I mean build and deploy data sets and environments uh for frontier AI. And that's at the intersection of academia and industry research, but also grounded in enterprise deployments. Um and this is all to say that we have a team that's built to focus on building the best data possible to define in advanced AI. And the way we do this is threefold. First is open research and benchmarks, so we're constantly contributing to open-source benchmarking, and the project I'll be talking about today actually falls under this pillar. Um but I also want to acknowledge frontier live data and environments, as well as enterprise uh deployments are the things we work on. And under open research and benchmarks, we support research through grants, and I'll talk about opportunities here at the end of this talk, so um get ready for that. And then co-authoring with academic uh or industry partners um and leading independent research and publishing open artifacts. So, what I'm talking about today is uh a partnership with UC Berkeley Sky Computing Lab, and we built, like I said, a 4 billion parameter Qwen model to outperform its 235 billion model variant on financial tasks. We saw the 4 billion parameter model, spoiler alert, uh achieve almost a 60% pass at one, while the 235 billion model uh achieved 51%. So, exciting results, and this is all uh in collaboration with I got to give credit where credit's due, Manon Rupta, Shujian Tang from the Sky Computing Lab, as well as Bhavishya and Chris Glaze and myself uh from Snorkel. And you can find all the assets, the uh training scripts, synthetic data uh on our GitHub, so check those out, and this is all open-source and available to you. So, I'll cover today what uh we see financial institutions need um from AI, and this is from our enterprise deployments uh learnings, as well as the FinQA open-source benchmark. And this is expert-validated financial question-answering data that we trained our uh and evaluated our model on. And finally, uh the training process for the RLLM FinQA 4B model. So, let's get started. This is uh the one and only slide with the Charles Dickens reference, I promise, but this is a tale of two models, I'd say. First, the large generalist versus small specialist. And the point is here, when financial institutions ask us how well does the model or the agent reason, what they typically mean is can it operate in my stack of complex schemas, legacy APIs, and tech debt? And can we trust it in realistic uh and long workflows without compounding errors? And can we reliably audit the agent? That's important in financial and legal domains, for example. Which brings us to the question is uh how can we evaluate and improve agents to make them reliable in these uh real-world environments? So, um the reflex may be to scale. We've all seen scaling law results, uh but today I'm going to argue that specialized workflows don't actually need generalists. We don't need polymaths. You know, you wouldn't call Terence Tao for a tax audit, um if you're lucky enough to have him in your contacts. You'd call the specialist that knows the forms, the tools, and the rules. So, that's why we built FinQA. This is expert-validated financial question-answering data. And I'll walk through the multi-stage pipeline of how we built this benchmark and training data. So, first is schema and data extraction. This is coming from 10-K reports. Uh these are annual reports required by the SEC for all public companies, and they're used uh by analysts for identifying risks and informing investment decisions. Uh we pull these from the Edgar system and use the Qwen 3 30 billion model to produce tables, SQL tables, approximately 6,900 of those. And we use each table to generate a single question-answer pair. And this is uh produced along with the question taxonomy that was developed with financial experts that work with these documents. And these are used as context uh for the model to generate a question-answer pair along with metadata that we'll see uh what that's used for later. And finally, the third step is verification. And this is a three-layer verification process. First is programmatic consistency checks. That's where we use um the metadata, uh which includes lineage and table names and columns to make sure we're not hallucinating those. The second is automated reviews from independent agents. And finally, expert manual review. So, humans actually going through these and making sure the questions and answers are realistic and verified. And the outcome is Snorkel's FinQA data. And this is our first pass at this. Uh realistic queries requiring planning, tool use, uh tool calling, and reasoning. And um answers are single, verifiable, final answers. And from this, um we get from the uh original set of roughly 7,000 tables, 4,000 in train, 500 in val, and 290 in the held-out benchmark. And the data is split so that no company in the benchmark overlaps um in any split. And on this data, we identified um even with frontier models, uh reoccurring discipline gaps, uh failure modes. And the first is schema hallucination. So, a model will actually assume that tables exist, column names exist, and um this can be an artifact of pre-training data, something it's seen um in its past. The next is context flooding. So, actually flooding its own context um with poorly playing queries. For example, just calling select star, um and this could overwhelm its context limits if successful. And finally, poor recovery um from errors. So, when 3235 would actually repeat the same failed strategy um instead of considering the error messages and adapting. And these failure modes are not actually just seen in the financial environment that I'm talking about today, but also what we saw in insurance underwriting. And this is a work that I'm calling out that was presented at the CAIS conference um few weeks ago. So, now I'll get into actually training the R LLM FinQA 4 billion model. Uh for this we used the R LLM uh framework and this was developed by the R Sky Computing Lab collaborators. And if you haven't used this yet, um this is a really user-friendly tool and a few core features um I want to call out is it really works with any agent framework, requires near zero code changes just using uh a decorator pattern, CLI first workflow, battle-tested results. So, um we've seen R LLM actually improve not just on this finance um case study, but also in math reasoning for example. Um multiple RL algorithms are built in, GRPO, reinforce RL, OOO for example, and you can customize those and then supports multiple uh training backends. All right, now for our training environment, um this is the uh training environment we used for our model. The agent runs in a react loop and has access to tools uh to interact with the tables we generated. The environment includes the roughly 7,000 tables we created and finally reward is binary correctness uh determined by a language model as judge. In this case, GPT 5 nano using reference-based evaluation. Training details are listed here. This is uh I guess would be interesting to folks. We have the Qwen 3 4 billion model is our base model, GRPO and binary reward uh for optimization and the RL and framework. Um we ran roughly 1,000 concurrent environments to generate trajectories and this all was a cost under $500 to get the results that we saw. Um broken down between compute roughly 420 and $40 for a judge uh the judge API. And this was on eight H100s. The first result um is the central question, can a 4 billion model uh armed with the specialized tools and training actually compete with its larger variant? And the answer is yes. So, what we saw was actually the base model's accuracy um more than doubled and it outperforms its larger variant, the 235 parameter model. And this is uh despite being a fraction of the size. And we evaluate all models shown here um on the expert curated held out 290 samples. So, a natural concern is whether this actually transfers to harder problems. So, this is just single question answer pairs on a single table. Um and to investigate this, we uh looked at our FinQA reasoning data. So, this is a development on the FinQA um data I presented earlier. And the difference here is that now we're having a dependencies on multiple tables, two to five tables, and requires sequential um decision-making and planning. And we evaluated on this data set. You can see the breakdown um as well. We have roughly 1,000 in train, 120 in validation, 80 in benchmarks. And what we see is the tool use discipline actually generalized directly from training on the simple data. And no explicit training on multi-table examples was needed actually to see lift in this multi-table variant of the data set. And we'll investigate why with the ablation studies. But the next concern we had was whether this result would generalize to more general use cases. So for this we use the BFCL benchmark and this measures general tool calling capability. And overall accuracy was actually slightly improved and then um we had also minor gains on multi-turn and memory. So specialization didn't actually erode the model's broader tool use competence when we did RL fine-tuning for this model. So to isolate what drove this performance, we ran ablation study on data mixtures, so different ways we can combine the training data we had. The first is just training on FinQA data and the second single and multi-table and finally a curriculum where we start with single and then move to multi. And surprisingly, just training on the simpler set resulted in the most lift. And this partially explains why the generalization result we saw earlier. The bottleneck was never reasoning depth, it was just tool use, meaning the model where it was actually failing and improving the reliability of that. And once you master the fundamentals, it could compose those skills. Yeah, so oops, let me go back. The next natural question is whether more informative reward signals, Um, for example, rewarding intermediate steps, uh, table access, query completeness, things like that could actually accelerate learning. Uh, so this is different from curriculum, for example. And the most sophisticated variant we came up with was, uh, a rubric. And this uses fine-grained scoring with, uh, multiple weighted components. And this is actually coming came up with with experts. Um, and despite this investment in reward engineering, we actually again saw, um, simpler was better. A single binary reward signal, um, was what got us the most lift on this data. Which led us to a blueprint, um, that we've been developing for enterprise agents. Um, and generally, I'd say this study showed us that small models uh, can successfully be trained uh, with RL um, when you have quality data to uh, represent the target um, outcomes and tasks. And this can result in better deployment economics uh, for specialized tasks and domains. And um, we've seen this and repeated this result in other areas including health care, law, and insurance, um, which I pointed to earlier. So, check that out. Um, and our point of view as far as building um, axes for evaluating agents is this is environment complexity is um, one of the most important axes. How complex, how realistic and dynamic are the environments uh, the agents operating in. The next is autonomy horizon. Um, are agents being tested and evaluated at the lengths and the horizon lengths in the way people actually want to use agents? Um, are they making safe and good decisions by themselves? And can they actually improve um, their judgment as a co-pilot over time? And finally, output complexity. We want to capture the wide range of outputs in our day-to-day work. Um, and I would say that this is currently under explored. New benchmarks are coming up in this direction, but uh still I see a big primary focus on um text artifacts and creating more rigorous evaluations and these settings is a big opportunity. Progress is being made here. So, on that note, um I'm going to advertise our open benchmarks grants, which funded this project. And this is a multi-million dollar commitment to producing open benchmarks like this. We've had a lot of success agent, last exam, judgment bench, um others are coming out and this is um really accelerating our ability to measure the frontier and shape the frontier. So, to apply uh check out our website. Um you'll go through a submission selection review process, develop, and then finally publication, launch, um and promotion. And I also want to call out that we're hiring in professional research, um you name it, engineers, and um yeah, check that out. You can check out our website and finally, thank you so much for attending. Um you can use this QR code to see the blog as well as pointers to all the artifacts that released on GitHub, Hugging Face. Yeah, thank you all.