A conference talk by Arek Borucki (Hugging Face) on how the Hugging Face Hub stays responsive at massive scale: 14 million users, millions of models and datasets. Covers full text search built on Apache Lucene via MongoDB Atlas Search ($search operator) with relevance ranking by downloads, likes, and trending; a seven node MongoDB cluster where only the primary takes writes and a hidden analytics node absorbs heavy queries; model artifacts stored in S3 with metadata in Atlas; sharding by key once a single primary is outgrown; and front-end autoscaling from 10 to 500 Kubernetes pods using CastAI plus KEDA-based scaling on event loop utilization. # Channel Presentation Observations ## Speaker Self-Identification - The speaker introduces himself directly: > "My name is Ar Borutki. I work as machine learning platform and database engineer at hugging face." - The video is a **conference talk format** ("Good afternoon everyone... I wish nice day for all of you... [applause]"), presented on behalf of **Hugging Face** rather than a personal channel project. ## Intended Audience - Audience is **assumed to already use Hugging Face**: > "How many of you already use hugging face? Amazing. Almost everyone." - Speaker explicitly frames target listeners as **developers and infrastructure/ML platform engineers**, given deep technical coverage of MongoDB, Kubernetes, sharding, and latency metrics. - The talk addresses people interested in **backend scalability**: "I would like to walk you through how hugging faced scaled infrastructure... share architectural decisions we made, challenges we faced, and lessons we learned." ## Claimed Scope and Scale - Creator claims the channel/platform context involves one of the fastest growing open-source AI communities: > "lessons we learned while scaling one of the fastest growing open-source AI communities in the world." - Specific scale claims (creator-stated figures, not independently verifiable from the material): - "Hugging Face serves more than 14 million users" - "three million public models, 1 million data sets, 50,000 organizations" - "More than 30% of Fortune 500 use hugging face" - "few years ago we had 20,000 models. Today 3 million... around 150x increase" Note: the video **title says 2 million models** while the transcript claims **3 million** — an internal inconsistency in the creator's own presentation materials. ## Demonstrated Presentation Patterns (Observable, Not Claimed) - **Technical depth**: detailed walkthroughs of MongoDB Atlas search, Apache Lucene autocomplete, precomputed tokens, replica sets, hidden analytic nodes, sharding, HPA/KEDA autoscaling, and Cast AI node autoscaling. - **Performance-oriented framing**: emphasis on **P99 over P50** latency: > "P99 is much more important than P50 and we are paying a lots of attention to P99." - **Named-vendor specificity**: references real tools by name (MongoDB Atlas, AWS S3, AWS Redshift, Kubernetes, Cast AI, KEDA, Llama, DeepSeek), signaling an engineering-transparency style. - **User-experience closing message**: > "Keeping the user experience simple no matter how complex it gets under the hood." - **Interactive opening**: polls the audience with show-of-hands questions, consistent with live conference delivery. ## Explicit vs. Demonstrated — Summary | Type | Evidence | |---|---| | **Explicit self-description** | Job title ("machine learning platform and database engineer at hugging face"), community descriptor ("fastest growing open-source AI communities in the world") | | **Demonstrated patterns** | Deep infra engineering content, scale-metrics storytelling, tool-by-tool architecture walkthrough, latency/UX focus | | **Not stated** | No channel-level mission statement, upload cadence, or target-content promises beyond this single talk's scope | # Delivery Observations — Arek Borucki, "Serving 2 Million Models Without Melting" ## Tone & Register - **Friendly, conversational conference tone** — opens with a joke question to the audience rather than formal framing: *"I have a question. How many of you knows hugging face?"* - **Enthusiastic but measured** — audience reactions are acknowledged warmly: *"Nice."*, *"Amazing. Almost everyone."* - Closes warmly and personally: *"Thank you very much. It was pleasure for me to be here today and I wish nice day for all of you."* ## Audience Interaction - Uses **show-of-hands questions** at the very start as an engagement device (two in a row at [13] and [25]). - Transitions the poll result into a light product pitch: *"But I think we still have opportunity to grow our usage."* ## Structural / Teaching Habits - **Explicit signposting** throughout the talk: - *"Before I dive into technical details, let's talk about scale."* - *"Now let's check how search works in details."* - *"Now let's have a look what actually is running on secondaries."* - *"Next the request goes to..."* / *"The next slides will show you how"* - **Numbered enumeration** when listing technical points: *"First, all queries which doesn't require the latest data..."*, *"Second, complex aggregations..."*, *"Third, change streams..."*, *"Fourth, all ad hoc queries..."* — a stable, repeated teaching pattern. - **Contrast-based explanation** — pairs of "was / now" or "small / large" comparisons: - *"At 20,000 models, any query is fast... At 3 million, same approach breaks."* - *"In 2022, we had 10K. In 2024, 100K. Less than a year ago... Today, 1 million."* - **Concrete walkthroughs** of example queries, narrating code step by step: *"For example, someone wants to search the model on the hub and let's say that's llama. So someone type llama into hugging face search bar."* ## Recurring Phrases & Vocabulary - **Scale-related framing repeats heavily**: "scale/scaling" appears across many segments (*"how hugging faced scaled infrastructure"*, *"This grow is exactly why I'm here"*, *"scale horizontally"*, *"we are going to scale everything"*). - **"Let's" constructions** used repeatedly to move the audience forward: *"let's get started"*, *"let's talk about scale"*, *"let's check how search works"*, *"let's have a look"*. - Habitual use of **"This is how it works" / "This is how"** as a pivot into explanations ([800], [1094], [1128] area). - Technical vocabulary is introduced **with brief plain-language glosses**: *"Sharding means scaling your database horizontally."* ## Rhetorical Devices - **Perspective-giving with arithmetic** to make scale tangible: *"With 14 million users, even 1% is a not small number. It is 140,000 of people hitting slow search."* - **Hypothetical user empathy**: *"Imagine what would you do if the hub search would be slow. you would just leave and go somewhere else."* - **Expectation-correction pattern** — flags surprising facts explicitly: *"there is one point that sometimes surprise people. MongoDB does not store the models themselves."* ## Anticipating Audience Questions - Repeatedly **pre-empts objections or curiosity**: - *"What does that mean in practice..."* before explaining metadata contents. - *"However, what happens if horizontal pot autoscaler want to add new ports but kubernetes does not have free nodes anymore."* - *"The key difference between replica set cluster and sharded cluster is..."* ## Language & Delivery Quirks - **Non-native-English features observable throughout**: occasional article/agreement slips (*"How many of you knows hugging face"*, *"they should stay impossible on secondaries"* — likely a slip for "possible" or misplaced word), inconsistent verb forms (*"we started to having problems"*). - Some **filled pauses and restarts** at technical transitions: *"MongoD uh this MongoDBT process is a wrapper..."*, *"when you pop when you push the model"*. - Technical terms are **spelled out verbally** with connective phrasing: *"Atlas search which is using Apache Lucin under the hood"*, *"that's a feature which is using Apache lucine under the hood"* — the phrase **"under the hood" recurs** (at least twice), a stable verbal habit. ## Uncertainty Markers - The speaker flags his own uncertainty when relevant, e.g., on the trending score: *"is as far as I remember from the last seven days"* — a habit of hedging on details not central to the point. - Uses audience-scoping honesty about scope: *"This is not trivial operation but this talk is not about choosing short key."* ## Confidence Assessment - **High-confidence patterns**: audience-interaction opener, "let's" signposting, numbered lists, contrast-based scale framing, "under the hood" repetition — all appear multiple times. - **Lower-confidence observations**: the apology-free handling of language slips suggests informal comfort, but this is a single talk; whether these delivery habits are stable across appearances cannot be determined from one transcript. # Observation Document: Scaling the Hugging Face Hub **Speaker:** Arek Borucki (transcript: "Ar Borutki"), Machine Learning Platform and Database Engineer at Hugging Face **Talk:** "Serving 2 Million Models Without Melting: Scaling the Hugging Face Hub" --- ## 1. Scale and Growth Claims - **Users:** Hugging Face serves **more than 14 million users**, growing quickly. - **Content hosted:** - **3 million public models** (talk title says 2 million; speaker states 3 million — discrepancy preserved as expressed) - **1 million datasets** - **50,000 organizations** - **Enterprise adoption:** **More than 30% of Fortune 500** companies use Hugging Face in AI workflows. - **Growth trajectory (models):** from ~20,000 models a few years ago to 3 million today — roughly a **150x increase**. - **Growth trajectory (datasets):** 10K in 2022 → 100K in 2024 → 500K less than a year ago → **1 million today**. - Big releases like **Llama** and **DeepSeek** each generate thousands of derivative models that the infrastructure must absorb. ## 2. Why Search Became the Hard Problem - Speaker's reasoning: at 20,000 models, "any query is fast" even without an index; at 3 million, that approach breaks. - Slow search would drive users away: users "expect fast instant results." - With 14 million users, even **1% latency problems** affect **140,000 people** — hence the emphasis on **P99 over P50**. - Investments motivated by this: **precomputed tokens, denormalized read-optimized MongoDB collections, Atlas full-text search (Apache Lucene), Kubernetes autoscaling, and (soon) database sharding**. ## 3. High-Level Architecture - Request flow: **front end → hub API → MongoDB Atlas** (metadata source of truth). - Hub runs on **Kubernetes** with a **horizontal pod autoscaler (HPA)** scaling pods automatically on spikes. - **Separation of concerns** (a stated key design decision): - **MongoDB stores metadata**: user repos, models, datasets, buckets, spaces, configuration, billing, access control. - **Model artifacts, tokenizer files, card assets, config files** live in cloud object storage (e.g., **AWS S3**). - Claimed benefit: metadata, binary storage, and compute can each be **scaled and optimized independently**. ## 4. Search Implementation ### Precomputed tokens - Model names are tokenized **at insert time, not query time** — e.g., `meta-llama/Llama-3.1-8B` split into tokens like `llama`, `3.1`, `8b`, stored in an array in the MongoDB document. - Atlas Search (Apache Lucene underneath) then uses **autocomplete** for instant matching. ### Evolution of querying - **Old approach:** MongoDB `find` with `$in`/regex-style operators on the search-token array, sorted by a **trending score** recalculated every five minutes (downloads and likes over roughly the last seven days, per speaker's recollection: "as far as I remember"). - **Limitation stated:** this worked while the dataset was small; regex "doesn't scale well" and latency problems appeared as data grew. - **New approach:** **Atlas Search** — the `mongot` process wrapping **Apache Lucene**; transparent to users via the unified MongoDB query API with `$search` in an aggregation pipeline (index name: `model_search_autocomplete`, path: `search_tokens`), still sorted by trending score. - Claimed result: "much more efficient," has "so far scaled well," latency issues in the search bar resolved. ## 5. MongoDB Cluster Topology - **Seven-node MongoDB cluster** used by hundreds of Hugging Face services handling millions of queries. - **Primary** handles all inserts, deletes, updates; **reads distributed across secondaries** to avoid a single read bottleneck. - **One hidden analytic node:** - Invisible to the application; drivers never route queries to it. - Replicates from primary but serves no production traffic. - Connected to directly for **reporting and heavy queries**. - Secondaries continuously **tail the oplog** from primary. ### Workload placement (stated pattern) 1. Queries not requiring latest data → **secondaries**; only strongly consistent queries stay on primary. 2. **Complex aggregations** (large scans, sorts, groups) → secondaries. 3. **Change streams** (cache validation, syncing to stores like AWS Redshift, event-driven workloads) → secondaries. 4. **Ad hoc / reporting / experimental queries** → hidden replica set member. - Stated principle: "Primary should focus on what only primary can do. Anything else can be pushed to different machines." ## 6. Sharding (Planned Next Step) - Claim: with current growth, a single replica set will soon not suffice. - **Sharding** = horizontal scaling: data split across shards, each shard with its own primary/secondaries (replication preserved, multiplied). - Key distinction stated: replica set holds the **full dataset** per node; sharded cluster holds only **part** of the data per shard. - Adding shards lets the **MongoDB balancer** redistribute data; a **shard key must be chosen** — acknowledged as non-trivial but not covered in the talk. - Claimed benefit: scales CPU, memory, storage, reads, and writes. ## 7. Kubernetes Scaling Layers - **Layer 1 — HPA:** hub deployment scales from **10 to 500 pods** based on CPU/memory thresholds; described as keeping the hub healthy without manual intervention or overprovisioning ("cost effective"). - **Layer 2 — node autoscaling via Cast AI:** when pending pods can't be scheduled due to lack of node capacity, Cast AI adds nodes so the scheduler can place pods. - **Planned migration: HPA → KEDA (Kubernetes Event-Driven Autoscaling)** - Reasoning: HPA reacts only to CPU/memory; KEDA reacts to **real application metrics** like requests per second or **event loop utilization**. - Example given: a pod with low CPU but high request queue — KEDA can see this; HPA cannot. ## 8. Closing Position - Speaker's framing of the goal: users "never have to think about it" — pushing, searching, downloading models "just works." - Stated thesis: **"Keeping the user experience simple no matter how complex it gets under the hood"** is what scaling is really about. --- ## Notes on Unresolved Points - **Title vs. content discrepancy:** talk title references "2 Million Models"; the speaker repeatedly states **3 million** — both preserved without reconciliation. - **Trending score definition** was given from recollection ("as far as I remember from the last seven days"), i.e., tentative, not presented as verified. - Sharding is described as **future work** ("soon"), not current production state; likewise the HPA→KEDA migration. # Speaker Attribution & Provenance Record ## Video **Title:** Serving 2 Million Models Without Melting: Scaling the Hugging Face Hub — Arek Borucki, Hugging Face ## Speakers ### Primary (and sole) speaker - **Speaker:** "Ar Borutki" (self-introduction, as transcribed) — identified in the video title as **Arek Borucki** - **Role:** Creator / presenter; self-identified as **machine learning platform and database engineer at Hugging Face** - **Speaking mode:** First-person conference talk (live audience present: interactive show of hands, applause at end) > "My name is Ar Borutki. I work as machine learning platform and database engineer at hugging face." ### Other voices - **Audience:** Responds to show-of-hands questions; applause noted in transcript — **unidentified, no speaking role** - **No narrator, guests, or additional identified speakers** appear in the transcript. ## Referenced Sources & Technologies (as claimed by the speaker) - **MongoDB / MongoDB Atlas** — metadata source of truth; Atlas Search - **Apache Lucene** — underlying full-text search engine (via Atlas Search / mongot process) - **Amazon S3 (AWS)** — object storage for model artifacts - **AWS Redshift** — mentioned as sync target via change streams - **Kubernetes** — horizontal pod autoscaler; planned migration to **KEDA** (event-driven autoscaling) - **Cast AI** — Kubernetes node autoscaling (third-party service) - **Llama and DeepSeek releases** — cited as traffic/model-growth drivers (third-party model releases, no external sources cited) ## Statistics & Claims Provenance All scale figures (14M users, 3M public models, 1M datasets, 50K organizations, 30%+ of Fortune 500, 150x growth, 10–500 pod scaling) are **first-party claims by the Hugging Face employee-speaker** with no external citation. Note: talk title says "2 Million Models" while speaker repeatedly states 3 million. ## Promotional Segments - **None flagged.** No sponsor mentions, ads, or promotional calls-to-action. Presentation is employer-affiliated (Hugging Face employee describing Hugging Face infrastructure) — inherently promotional of the employer, but no explicit promotional segment present. ## Availability Limits - **Comments:** No comment availability stated (no curated metadata supplied). - **Q&A:** Not present; talk ends on applause. - **Uncertainty:** Speaker name given only as transcription "Ar Borutki"; title metadata indicates "Arek Borucki." Good afternoon everyone. I have a question. How many of you knows hugging face? Nice. How many of you already use hugging face? Amazing. Almost everyone. But I think we still have opportunity to grow our usage. My name is Ar Borutki. I work as machine learning platform and database engineer at hugging face. Today I would like to walk you through how hugging faced scaled infrastructure and how we ended up serving 3 million models to developers around the world. I would like to share architectural decisions we made, challenges we faced, and lessons we learned while scaling one of the fastest growing open-source AI communities in the world. I hope you will enjoy it and let's get started. Before I dive into technical details, let's talk about scale. Today, Hugging Face serves more than 14 million users and this number is growing very fast, especially in the last couple of months. We host three million public models, 1 million data sets, 50,000 organizations, and not only hobbyists or scientists. More than 30% of Fortune 500 use hugging face as a part of AI workflows. Just to give you some perspective, few years ago we had 20,000 models. Today 3 million. It is around 150x increase in just last couple of years. And this grow is exactly why I'm here today talking about infrastructural decisions that keep the hub healthy at scale. This is how fast the number of public models is growing on the hub. Every big release like llama or deepseek generated thousands of new models on top and our infrastructure needs to handle that and it is not only models also data sets. In 2022, we had 10K. In 2024, 100K. Less than a year ago, we had 500K. Today, 1 million. All this data must be stored, indexed, and also must be searchable. And that's the hardest part. And this is also the reason why we had to rethink our search. At 20,000 models, any query is fast. Even without an index, trust me, no one would notice. At 3 million, same approach breaks. Imagine what would you do if the hub search would be slow. you would just leave and go somewhere else. And this is also what user are doing. They expect fast instant results. With 14 million users, even 1% is a not small number. It is 140,000 of people hitting slow search at scale. P99 is much more important than P50 and we are paying a lots of attention to P99 and that's the reason why we invest in premputee tokens denormalize optimize for read collection in MongoDB full text search based on Apache lucine Kubernetes autoscaling and soon in database sharding. The next slides will show you how high level architecture when user interact with the hugging face hub his request flows from the front end to the hub API. The hub is running on Kubernetes. Currently we are using horizontal pot autoscaler. During spikes new ports scale up automatically to handle the load and scale back down when traffic drops. This help us to keep the hub healthy without manual intervention. Next the request goes to MongoDB Atlas which is source of true for our metadata. And there is one point that sometimes surprise people. MongoDB does not store the models themselves. It stores everything about the models. What doesn't mean in practice in MongoDB we hold all the metadata users repositories models data sets buckets spaces information configuration data billing data access control and more. The actual models artifacts, tokenizer files, cart assets and configuration files are stored separately in cloud object storage such as AWSS3. This separation of concern let us scale metadata independently from binary storage and compute independently from both. We can optimize each component individually for specific workload. Now let's check how search works in details. For example, someone wants to search the model on the hub and let's say that's llama. So someone type llama into hugging face search bar. His request flows through the hub to an optimized read collection on MongoDB. And this is not our main repo collection when we keep all the data. It's a separate denormalized copy only for reads and listings. The key information is on the left. We tokenize model names on insert time, not at query time. For example, someone wants to publish model meta minus llama/ llama 3.18b. We split the long model name into small tokens like meta lama 3.18b and we store them in an array in MongoDB document. Next, Atlas search which is using Apache Lucin under the hood use autocomplete to find matching models instantly. This is example of single document from our model collection. In this example, I'm using find one method. I want to find model ID meta minus llama/lama 3.1. So that's the model from the previous slide. And I'm projecting only search token array. And we see that all those premputee tokens are part of this array. So we have metal lama 3.1 metal lama etc. Next there must be a query. In the past we were using classical MongoDB find method on models collection with reax operator and this reax operator we're searching in search tokens arrays models which are equal to llama and then we were s sorting results by trending score which is calculated every five minutes. This is number of downloads and number of likes is as far as I remember from the last seven days. This solution was working well as long as data set was small. Reax doesn't scale well. So when our data set started to grow very quickly, we started to having problems with latency. So we decided to switch to Atlas search that's a feature which is using Apache lucine under the hood. So MongoDB doesn't provide in core MongoDB server full text search there is additional process MongoD uh this MongoDBT process is a wrapper around Apache lucine for end user users this is transparent you are just using Unifi MongoDB query API and when You use aggregation pip pipeline together with dollar search operator. MongoDB will know that you would like to search Apache Lucin index. Obviously you need to put the name of this index which is in this scenario model search autocomplete model equal to llama path search tokens and we still sort results by trending score and this solution is much more efficient and is so far scale Well, so we don't have any more latency issues in our search bar. First two results returned by previous query. First metal lama has trending score 33. Second one 14. But hugging face hub is not only search. We have hundreds of different services in hugging face which are utilizing which are using MongoDB to handle million of queries. We use seven nodes MongoDB clusters cluster with multiple machines. We can distribute queries across multiple nodes. So no single node become read bottleneck. This is how it works. Application talk to the MongoDB cluster. All inserts, deletions or updates goes to single primary because only primary can handle them. However, we are distributing reads across multiple machines. We also have one analytic hidden node. What does it mean? This mean this node is invisible from application. MongoDB driver is not routing any queries to this hidden node. This node is still replicating data from primary but is not interacting with interacting with production traffic. We are connecting directly to this node and we use him for any kind of reporting traffic on or any kind of really heavy queries. All secondaries continuously tail the oplog from primary keeping cluster in sync. Now let's have a look what actually is running on secondaries. First all queries which doesn't require the latest data goes to secondaries. Only queries that must have strong consistency stay on primary and we are paying a lots of attention to this. We are paying lots of attention to the queries which must run on primary. Second, complex aggregations. Aggregations pipelines that scan large amount of data, sort, group or transform the data should not go on primary. They are heavy. Secondaries are better placed for them. Third, change streams. We react to data changes in real time for several reasons. For example, caching validation, sync to different data store technologies like for example AWS, Red Shift or for eventdriven workloads. Those kind of operations are also not very light and they should stay impossible on secondaries. Fourth, all ad hoc queries, reporting queries, maybe some experimental queries go to hidden MongoDB replicas set member which is isolated from production traffic. The pattern is simple. Primary should focus on what only primary can do. Anything else can be pushed to different machines. However, with 14 million users, 3 million models and our grow soon single MongoDB replica set will not be enough. The next step is sharding. Sharding means scaling your database horizontally. Instead of putting full data set on one replic on one replica set cluster, we are going to cut data into pieces and put each piece on separate chart. Each shard will have his own replication primary and secondary. So we will keep replication just multiplied. The key difference between replica set cluster and sharded cluster is replica set keep full data set on each node. Sharded cluster keep only part of the data on each shard. And then if you want to scale horizontally more you are just adding more shards and then MongoDB balancer will balance data across all those shards. There is also shard key which must be selected. This is not trivial operation but this talk is not about choosing short key. This way we are going to scale everything CPU memory storage reads and writes. Now let's have a look what is going on the hub level. DHA is running on Kubernetes. Currently we are using horizontal pot autoscaler. When CPU or memory threshold go above target, Kubernetes adds new pot automatically to handle this spike and scale them back down when traffic drops. Our deployment hub deployment can scale from 10 to 500 bots depends of on the traffic. This is how we keep the hub healthy without manual interventions and without infrastructure overprovisioning. So this is also cost effective solution. However, what happens if horizontal pot autoscaler want to add new ports but kubernetes does not have free nodes anymore. This is where second layer comes in. We are using cast AI for Kubernetes note autoscaling when pods are pending because there is no capacity and Kubernetes scheduler is not able to schedule them. Cast AAI is adding new nodes and then scheduler is able to schedule those spots. So we have two layers of scaling. First one is at deployment level. Second one is at infrastructure level via castai. But we are going to migrate horizontal port autoscaler to keta kubernetes eventdriven autoscaling. The difference HPA scale only based on CPU and memory. KDA scale on real application metrics like request per second or event loop utilization. It means scaling is driven by actual workload not by resource utilization only. For example, pot can have low CPU but high request Q. KDA can see it. HPA not the best part of this architecture. You never have to think about it. When you pop when you push the model, search the hub or download the model, it just works. This is what scaling medium models is really about. Keeping the user experience simple no matter how complex it gets under the hood. Thank you very much. It was pleasure for me to be here today and I wish nice day for all of you. Thank \[applause\] you.