Workshop demo showing how an Obsidian vault connects to a coding agent through an MCP integration, with small open models on team-controlled infrastructure handling OCR, redaction, and other document processing before the agent uses the output. Covers the host, MCP server, and GPU worker pool architecture, the inference layer (model loading, batching, shared clusters), and related topics including embeddings, reranking, and task-specific LoRAs. The talk closes by questioning whether instructions alone enforce privacy if the agent can read raw files before calling the tool. # Channel Presentation Observations **Source:** Video `01a15ca6-ab3c-754b-bd95-704e60f5ca6d` **Title:** *Turning My Obsidian Vault Into a Local AI Engineer — Filip Makraduli, Superlinked* **Inputs available:** Title and transcript only. No curated metadata was supplied, so channel-level identity context is absent. --- ## 1. Explicit Self-Description ### 1.1 Speaker and Affiliation - The **title** names the speaker as "Filip Makraduli, Superlinked." The transcript never states the speaker's name. - The speaker presents as affiliated with Superlinked through first-person plural references: > "I found that the solution was, um in Superlinked, which is a model inference, um engine that's open-source." > "So, it's an integration… which is basically a package that we did" > "we host the hardware, but you can totally do this yourself." ### 1.2 Self-Framing of the Talk - The speaker frames the talk as a personal project narrative: > "it all started for me wanting to build a use case that's around privacy and token costs, basically." - The speaker says the demo uses synthetic data: > "this is all synthetic data but uh relevant data. I'm not using the real data that inspired this project" ### 1.3 Self-Assessment and Candor - The speaker acknowledges limitations and unfinished work: > "there are some failure modes that are already there… there are so many more things to do." - Routing and council-of-models ideas are explicitly marked as roadmap: > "this is currently not built in, but um the idea is to also pick the models based on the task." --- ## 2. Intended Audience (Demonstrated) The transcript addresses these groups, based on what it discusses and asks of listeners: - **Developers using AI coding agents** - References to "Claude Code," "co-work," and "agents," and building an MCP server for them. - **Privacy-constrained organizations and practitioners** - > "for some institutions, for some people, this was, um, a requirement that… needed to be met." - **Open-source ML and inference practitioners** - Discussion of LoRAs, quantization-adjacent concerns, model runtimes, GPU worker pools, and batching. - **Hackathon and contributor communities** - > "I'd be also happy for you, so you get more credits if you um kind of build a project" - Invitations to submit PRs and examples. --- ## 3. Claimed Scope ### 3.1 Functional Scope of the Demonstrated Project - **Token-cost reduction** through offloading OCR and document preprocessing. - **PII redaction** of documents, shown with an NDA example. - **Self-hosted or VPC-hosted inference** to keep sensitive data private. - **Tool-based MCP architecture** exposing tools such as docs-to-markdown, entity extraction, captioning, summarization, and question answering. ### 3.2 Broader Claims Made - Open-source models are competitive in quality: > "you're not compromising in quality by offloading some of the work to open source um, OCR models." - Benchmark claim (unverified within this material): > "you get kind of um 10x lower price roughly um with very similar quality… 99.8% to uh kind of the frontier one… using MTEB tasks." - The speaker says more benchmarks will follow. None are shown here. --- ## 4. Promotional and Commercial Elements The transcript repeatedly directs viewers to Superlinked resources: - **Access offers:** "apply for in inference grant," "free um kind of cluster infrastructure credits." - **Physical promotion:** QR codes described as "memorabilia stuff." - **Contact and distribution:** LinkedIn, an upcoming article, and a GitHub repository ("Superlinked Sci"). - **Incentives for contribution:** Credits tied to building or contributing tools. These are presented as calls to action rather than neutral information. --- ## 5. Demonstrated Presentation Patterns - **Live demo with narration:** The speaker runs Claude Code on synthetic files, redacts an NDA, and processes a scanned contract. - **Hedged on demo reliability:** > "hopefully the demo gods are good." - **Repeated reframing of questions:** The speaker restates audience questions "because of the recording." - **Conversational, informal delivery:** Frequent filler and self-correction throughout. --- ## 6. Limits of This Observation - **No channel-level evidence:** Only one video was supplied. Nothing here supports conclusions about the channel's overall focus, cadence, or history. - **No metadata:** Description, channel name, upload history, and tags were not provided. - **Transcript quality:** The transcript appears automatically generated. Some terms are likely misrecognized (for example, "Fable" and "Laura's" for LoRAs), so exact wording should be checked against the source before quoting. - **Unverified claims:** Benchmark figures and cost reductions are the speaker's assertions and are not independently demonstrated in the material. # Delivery Observation — Filip Makraduli, "Turning My Obsidian Vault Into a Local AI Engineer" ## Overall Impression A conversational, demo-driven technical talk delivered live to an audience, with heavy reliance on softeners, verbal hedging, and frequent self-corrections. The speaker mixes product advocacy with hands-on walkthrough and Q&A. ## Recurring Vocabulary and Phrases - **"Kind of" is the dominant hedge**, appearing dozens of times across the talk: > "I wanted to find a way to kind of do this" > "the heavy lifting is done by this part" > "how I managed to kind of merge these concepts together" - **"Basically" as an explanatory pivot**, used to restart or simplify an idea: > "Basically, you uh, kind of have the tools which give you different um, abilities" > "So, basically, I didn't want to disrupt my normal usage day-to-day." - **"Um" / "uh" fillers are extremely frequent**, appearing in nearly every sentence boundary throughout. - **"So" as a sentence opener and transition**, used constantly to move between slides and ideas: > "So, that's the general idea, especially for OCR this is apparent." - **"And um" as a segment joiner**, often launching new topics: > "And um here we have the repo and also the link..." ## Teaching Habits - **Roadmap-then-deliver structure**: states goals upfront ("So, the first main idea was I wanted to make Claude Code a bit cheaper and save on tokens"), then returns to the same three pillars (privacy, token cost, adaptability) repeatedly. - **Anticipatory framing of objections**: > "it's not that oh, I'm using open source, I'm sacrificing something. That's not the case at all" - **Q&A restatement protocol**, explicitly announced for the recording: > "you'll just kind of speak and I'll restate and reiterate the question because of the recording." - **Concrete tool-name repetition**: "Claude Code", "MCP", "VPC", "OCR", "LoRAs" recur as anchors throughout. - **Audience-directive calls to action**: repeated invitations to grab credits, clone the repo, and contribute: > "I encourage you to take this um free um kind of cluster infrastructure credits" > "you can also do a PR and add a different tool there right now" - **Humor about demo risk**: > "hopefully the demo gods are good." ## Communication Patterns - **Self-corrections mid-sentence** are common: > "Uh, an integra an MCP integration which is basically a package that we did" > "So, Oops, cursor." - **Contrastive explanations** ("not X, but Y") to clarify design choices: > "So, why MCP and not just an API or a plugin?" > "this is more like a future roadmap idea, so this is currently not built in" - **Live-verbalization of demos**, narrating screen actions step by step ("So, I'm going to kind of cloud code. I am in the relevant packages directory.") - **Warm, inviting closing tone** toward contribution and follow-up: > "you can add me on LinkedIn and all that and I'm happy to chat always." ## Isolated Moments (not established patterns) - The "demo gods" joke and the QR-code "memorabilia" mention ([410]) appear once each; no broader humor pattern is confirmable from this single transcript. - The "council of models" forward-looking aside ([2106]) appears once and is explicitly labeled future roadmap. ## Uncertainty Notes - Repetition counts are approximate; "kind of"/"um" frequency is clearly high but not precisely quantified here. - Whether the hedging style persists in other talks or written work cannot be assessed from this single video. # Observation Record: Local, Privacy-Preserving Document Processing via Superlinked and MCP ## Source and Attribution - **Video:** "Turning My Obsidian Vault Into a Local AI Engineer" (ID 01a15ca6-ab3c-754b-bd95-704e60f5ca6d) - **Speaker:** Presented as **Filip Makraduli, Superlinked**, per the title metadata. The transcript itself does not restate the name. - **Audience questions:** Unattributed in the transcript. They are recorded here only as the speaker's restatements or responses. - **Transcript caveats:** Several terms appear to be mis-transcribed. "Sci" appears to refer to Superlinked's product, "Laura's" to LoRAs, and "Fable" to an unspecified model release. These are reproduced as written or interpreted with that caveat. --- ## 1. Stated Motivation The speaker describes the project's origin as a use case centered on **privacy and token costs**. - **Privacy requirement:** Using cloud agents (Claude Code, Co-work) can expose files. The speaker says some institutions treat this as a **requirement** to be met. - **Token cost concern:** Sensitive document processing, especially OCR on scans and images, can "ramp up" token costs. - **Context quality:** The speaker argues that smaller, preprocessed artifacts give better context and limit context loss. - **Adaptability:** The speaker wanted to test multiple models and host task-specific **LoRAs**, noting that sensitive niche use cases usually involve fine-tuning or adaptation. > "What if you could kind of self-host and have your own VPC or your own cloud to preprocess your sensitive documents and then return something back." --- ## 2. Core Positions ### 2.1 Privacy - Sensitive content should be preprocessed inside a self-hosted environment (the speaker's VPC or cloud). - The main agent (e.g., Claude Code) receives only the **redacted or processed output**, not the original file. - **Qualification:** The speaker acknowledges the agent's behavior is not fully guaranteed. Stated verbatim: "this can be kind of evaluated like whether this happens like whether Claude does this kind of off off-rails situation. Definitely, that can be kind of tested more and more." - **Evidence basis:** In the speaker's own tests, the setup "seemed to have worked without kind of the data leaking." The speaker suggests additional guardrails could be added. ### 2.2 Token Economics - Offloading OCR and similar work to open-source models reduces token spend in the main agent. - The speaker states that token savings apply "even beyond OCR," but gives no quantified examples for non-OCR tasks. - **Qualification:** A status-line counter for tokens saved exists, but the speaker says it "need[s] to also update this a bit to kind of do it properly." ### 2.3 Open-Source Quality - The speaker states: "you're not compromising in quality by offloading some of the work to open source OCR models." - **Evidence status:** No OCR benchmark is cited. The only quantitative comparison presented concerns embedding models (see Section 5). ### 2.4 Architecture Choice: MCP over Skill or API The speaker chose an **MCP server** over a plugin or API for these reasons: - A dedicated cluster allows separation of responsibilities and a backbone for later features. - The user keeps control over worker pools, GPUs, and model placement. - MCP provides one interface, which the speaker expects to matter as model routing grows. - The speaker describes MCP's use of JSON-RPC handshakes and notes that the server here exposes **only tools**, not sampling or elicitation, "just because this was more of a privacy setup." --- ## 3. Infrastructure and Inference Claims The speaker frames **inference and infrastructure as the primary bottleneck** and differentiator. ### 3.1 Stated Advantages | Claim | Speaker's Basis | Evidence Strength | |---|---|---| | Per-worker batching outperforms a shared queue | "in our testing and in general performance was lower" for shared-queue setup | Asserted from internal testing; no figures given | | End-to-end latency and performance improved with different models | Architecture-level observation | Asserted; no figures given | | Hot-swapping of models on GPUs is beneficial | Especially for small NER, generation, and embedding models | Described as a capability; no measurements given | | Warm models respond faster | Stated as "powerful" for routing decisions | Asserted | | Auto-scaling reduces idle cost | Stated as a feature | Asserted | ### 3.2 Hardware Flexibility - Cheaper models could run on "a L4 spot instance," avoiding expensive hardware. - The speaker says the user can select worker pools and GPUs. ### 3.3 Self-Hosting - The managed cluster used in the demo is Superlinked's, shared for demos and hackathons. - The speaker states the setup is "totally your own" if self-hosted via the repo's Terraform and Helm/Kubernetes configuration. - **Qualification:** The speaker says self-hosting is "not strict," and that Superlinked's cluster is used "so that things can go quicker." --- ## 4. Demo Observations The speaker describes a live demo on **synthetic data** (explicitly not the real data that inspired the project). - **Redaction:** An NDA document containing a phone number and social security number was redacted. The speaker reports the output masked "only email and person" entities, with "10 spans" redacted. Whether the SSN and phone number were masked is not explicitly stated in the transcript. - **OCR-to-markdown:** A scanned contract was converted to markdown and then redacted. - **Performance caveat:** The speaker says "in certain situations you even get better performances with the OCR stuff," without elaboration. - **Demo risk:** The speaker says "hopefully the demo gods are good," and some steps relied on natural-language prompting for tool calling. --- ## 5. Benchmark Claims The speaker reports one quantitative result, with explicit caveats: - **Task:** Embedding models - **Basis:** MTEB tasks - **Claimed result:** The best model per task reached **99.8%** of a frontier model's score. - **Claimed cost:** Roughly **10x lower price**, described with "roughly." - **Qualification:** "We'll have more benchmarks and details around this soon as well." - **Scope note:** The speaker says generation-model benchmarks are not yet available. "The tests that we have the benchmarks for for now are around embedding models." --- ## 6. LoRA and Domain Adaptation - The speaker reports that colleagues produced **German legal language LoRAs** that were "fairly cheap" to fine-tune. - The speaker states that deploying multiple LoRAs quickly "seemed to be quite useful and improved performance" in this niche. - **Evidence status:** No measurements are provided in the transcript. --- ## 7. Roadmap (Explicitly Not Yet Built) The speaker repeatedly distinguishes current functionality from future plans. The following are **future directions**, not present capabilities: - **Model routing:** Selecting models per task based on evals and cost, so the tool call "will not be tied just to the tool itself." The speaker says this is "more like a future roadmap idea, so this is currently not built in." - **Warm-aware routing:** Preferring a hot, more expensive model over reloading a cheaper, cold one. Described as possible but not implemented. - **Multi-model judgment:** "Judges talking between each other" or a "council of models." Described as "more of a future road map." - **Non-text modalities:** Transcripts, graph data, and ontology generation, described as "opportunities." ### Stated Risks of Multi-Model Designs The speaker names several concerns without resolving them: - Models spending GPU time "reasoning forever" - Biased models acting as judges - Optimization that "becomes" the target, leading models to "hack" the metric so the evaluation is no longer useful The speaker says the team is "very strong on evals" and tests models before such designs are used, but does not present results. --- ## 8. Identified Failure Modes and Tensions - **Thinking tokens:** One summarization tool initially produced very short output or none, because a model's thinking tokens consumed the output. The speaker says the model choice and thinking-token toggle must be considered. - **Reasoning trade-off:** The speaker says reasoning "can be beneficial" but that it is useful to toggle it from the infrastructure side. - **Existing failure modes:** The speaker says "there are some failure modes that are already there" but does not enumerate them. - **Unfinished components:** The token-savings counter, documentation, and model catalog are described as in progress or being revamped. --- ## 9. Audience Q&A (Speaker Responses) - **Privacy with file-based workflows:** The speaker responds that Claude would receive only the redacted version. Question wording is unclear in the transcript. - **Self-hosting:** Confirmed as possible through the repo's Terraform configuration. The speaker notes that free compute credits are available at sign-up for trial use. - **Harness independence:** The speaker suggests the pattern could work with other harnesses, including an open-source one, "if you implement the tool calling well." Presented as likely but untested in the transcript. - **Contributions:** The speaker invites contributions of tools (e.g., content guarding, policy checking) and examples, with extra credits offered for projects. --- ## 10. Summary of Speaker's Central Thesis The speaker's main takeaway is that **"infrastructure and inference is far more important"** than the user-facing workflow, even for API-friendly designs like Claude Code. The speaker presents the MCP server as a privacy and cost layer whose performance depends on the underlying inference system. **Confidence markers in the transcript:** Most architectural and performance claims are presented as experience-based ("in our testing," "seemed to have worked"). The strongest quantitative claim (99.8% of frontier on embeddings, roughly 10x cheaper) is limited to embedding models and is described as preliminary. # Attribution Observation: Obsidian Vault to Local AI Engineer Talk **Video ID:** `01a15ca6-ab3c-754b-bd95-704e60f5ca6d` **Title (as supplied):** Turning My Obsidian Vault Into a Local AI Engineer, Filip Makraduli, Superlinked --- ## 1. Speakers and Roles ### 1.1 Primary Speaker - **Filip Makraduli** - **Role:** Presenter and sole narrator of the talk. - **Affiliation:** Superlinked, per the supplied title. The transcript itself does not state a job title. - **Speaking scope:** Delivers the full presentation, demonstration narration, and most of the Q&A responses. - **Self-described relationship to project:** Built the described use case; refers to "my" vault, "my" workflow, and "my" testing. ### 1.2 Other Voices - **Unnamed audience member(s)** - **Role:** Asked questions during the Q&A segment. - **Identification:** Not identified. The transcript marks question turns with `>>` but gives no name or label. - **Content of questions (paraphrased):** Privacy handling of redacted versus original documents; how privacy is preserved when using a managed cluster. ### 1.3 Referenced but Non-Speaking Parties - **Superlinked colleagues:** The speaker says "some of the colleagues" worked on German-language LoRA fine-tunes. They do not speak in the transcript. - **Hackathon participants:** Described as having tested the system at a London hackathon. Not speakers. --- ## 2. Referenced Sources and Channels ### 2.1 Tools and Products Named - **Obsidian:** Origin of the markdown files used in the project. - **Claude Code:** The agent harness in the demonstration. - **Co-work:** Mentioned as an agent through which files can be exposed. - **"Fable":** Referenced as a model release with API-only pricing. The term is transcribed as stated and may be a transcription artifact; it is not verified here. - **OpenAI API:** Referenced as a possible migration source for workflows. - **Docker container:** Named as a local-run option. - **Helm charts, Kubernetes, Terraform:** Named as deployment tooling in the open-source repository. - **L4 spot instance:** Named as a hardware option. ### 2.2 Protocols and Standards - **Model Context Protocol (MCP):** The integration standard used. The speaker notes it has changed since inception. - **JSON-RPC:** Named as the communication format. - **Streaming HTTP:** Named as the transport chosen over standard I/O. ### 2.3 Benchmarks and Evaluation Sources - **MTEB (Massive Text Embedding Benchmark) tasks:** Named as the evaluation basis for embedding-model comparisons. ### 2.4 Repositories, Sites, and Documents - **Superlinked open-source repository ("Superlinked Sci" as transcribed):** Named as fully open source on GitHub. The MCP package is in a subdirectory. - **Superlinked website:** Hosts an inference grant application. - **Forthcoming article:** The speaker says they will publish a step-by-step article. - **Public docs, model catalog, examples, troubleshooting guides:** Described as available, with no links captured in the transcript. ### 2.5 Events and Partners - **London hackathon with Tech Europe and DeepMind:** Named as a recent event where participants tested the system. ### 2.6 Contact Channel - **LinkedIn:** Named as the speaker's contact method. --- ## 3. Third-Party and Attributed Claims Each claim is attributed to the speaker unless otherwise noted. None are verified in this record. ### 3.1 Cost and Performance Claims - **Token savings:** Preprocessing documents (e.g., OCR on a 10-page scanned PDF) is said to reduce token spend and improve context. - **Embedding-model pricing:** The speaker states roughly **10x lower price** with **very similar quality**. - **Embedding-model quality:** The best per-task embedding model is said to reach **99.8%** of a frontier model's performance, evaluated on MTEB tasks. - **End-to-end latency:** Per-worker batching is said to outperform a shared queue in "our testing" and in general. - **Open-source OCR quality:** The speaker asserts that open-source OCR models do not compromise quality. ### 3.2 Privacy and Data-Handling Claims - **Redaction behavior:** The speaker states that Claude Code receives only the redacted version of documents, not the originals. - **Demo data origin:** Stated as **synthetic**. The speaker says the real data that inspired the project was not used. - **Self-hosting:** Stated that the cluster can be fully self-hosted via Terraform; the managed cluster in the demo is operated by Superlinked. ### 3.3 Domain Claim - **German legal LoRAs:** Colleagues are said to have produced inexpensive, fine-tuned German-language legal LoRAs that improved performance in that niche. No metrics are given in the transcript. ### 3.4 Roadmap Claims (Not Built) The speaker explicitly flags these as **future roadmap, not currently implemented**: - Model routing based on task, cost, and warm/hot GPU state. - Council or judge-based multi-model setups. - Intent-based routing and mixture-of-agents approaches. --- ## 4. Promotional Segments The following segments are **flagged as promotional** because they direct viewers toward the speaker's company products, free credits, or contribution opportunities: - **Inference grant and API key:** Viewers are directed to the Superlinked website to apply for a free inference grant. - **Free cluster credits:** Repeatedly encouraged ("take this free cluster infrastructure credits"). The speaker also says contributors "get more credits." - **Hackathon promotion:** The London hackathon is cited as evidence of popularity ("very popular at the hackathon"). - **QR code memorabilia:** Described as a cluster-access offering after the talk. - **Product comparison framing:** The Superlinked inference engine is positioned as the reason the system works, including comparisons to API-based approaches. - **Contribution solicitation:** Viewers are invited to submit PRs, tools (e.g., content guarding, policy checking), and examples. - **Contact solicitation:** Viewers are invited to connect on LinkedIn. --- ## 5. Availability Limits and Evidence Gaps ### 5.1 Metadata Gaps - **Comment availability:** Not supplied. Cannot be determined from the record. - **Speaker bio or affiliation detail:** Only the title's "Superlinked" attribution is available. - **Event name, date, or venue:** Not supplied (beyond the hackathon mention). - **Recording date:** Not supplied. ### 5.2 Visual Content Not Captured - **Slides:** The speaker notes the slides differ slightly from the title. Slide content is not in the transcript. - **Demo screens:** The speaker says at one point, "I don't know if you can see now." Certain demo outputs, file contents, and status-line token counts are referenced but not readable in the transcript. - **QR codes and URLs:** Referenced but not transcribed. The exact website URL and repository path are not captured. ### 5.3 Transcript Artifacts - Speaker name references and product names are transcribed inconsistently (e.g., "Sci" for Superlinked, "Laura's" for LoRAs, "Fable" possibly a misrecognition). - Non-speech markers (`\[snorts\]`, `>>`, `\>`) appear in the transcript. - The speaker interrupts themselves mid-sentence ("Sorry, not here"; "Sorry, again?"). - Q&A questions are not fully transcribed; only the speaker's restatements and answers are clearly captured. ### 5.4 Unverified Demonstration Claims - The demonstration outcome (e.g., "successfully loaded skill," redaction counts, OCR success) is narrated by the speaker. The transcript does not provide independent verification. - The speaker notes that prompting behavior ("whether Claude does this kind of off-rails situation") has not been fully tested. --- ## 6. Summary of Provenance Status | Category | Status | |---|---| | Presenter identity | Stated in title; speaker self-identifies within talk context | | Q&A questioner identity | Unidentified | | Source of benchmark claims | MTEB named; underlying results not provided | | Demo data | Stated as synthetic by speaker | | Promotional content | Present throughout; flagged in section 4 | | Roadmap items | Explicitly marked as unbuilt | | Comment availability | Not supplied | | Visual evidence | Partially unavailable | Hello, hello. Um, I guess we're starting. I know some of you are still joining in, but I'll go on and you can jump in at any time. Uh, so you might have seen the title of this talk. Um, and what I have on the slides here is slightly different, but I promise it's the same direction and the same general idea. So, it all started for me wanting to build a use case that's around privacy and token costs, basically. So, um, I started with Obsidian. I had some markdown files there. Um, and as you know, it's like you have this nice, um, graphs and it's all good. Looks very nice. However, um, I had certain issues beyond just organizing the files. So, a lot of people do this knowledge graphs and so on and like memory for agents and that's quite good. However, I needed to approach this differently because there was a concern around privacy. So, when using agents, when using cloud code, when using co-work, uh, you can expose your files and for some institutions, for some people, this was, um, a requirement that, um, needed to be met. So, the goal was to keep things private and self-hosted, but also make them quite usable and friendly so that people could actually do these things. And, um, there was kind of this openness to fine-tune, to be very adaptable, to use open-source models, and, um I found that the solution was, um in Superlinked, which is a model inference, um engine that's open-source. So, it has both a cluster and a model inference solution. And, I will be talking about that today, and how I managed to kind of merge these concepts together, and, um create something that leans on these three things. So, the first main idea was I wanted to make Claude Code a bit cheaper and save on tokens. So, we know how with the release of Fable, there initially there was this, um API-only pricing, and that kind of, uh maybe disappointed a few people, or the token economics got a bit more attention now. So, that was one thing. Especially for document processing, sensitive document processing, where you need to do OCR, there are a lot of scans, images, and stuff like this, the token cost can ramp up. So, I wanted to find a way to kind of do this, but also having smaller artifacts means that you get better context, so you kind of limit context loss in a way, and you don't like fill out, um the context with data that can be processed in some way else. Um, and I wanted to make this private. So, yes, for certain situations, it's fine to paste everything into Claude Code or have that API working there. But, what if you could kind of self-host and have your own VPC or your own cloud to preprocess your sensitive documents and then return something back. So, this was another target of mine when building this. And also I wanted adaptability, so it wasn't just um privacy and lowering token costs, but also having the chance to test out multiple models, maybe host Laura's that are very task specific because with this very sensitive use cases, usually there is some sort of fine-tuning or adaptation towards that niche use case. So, I had to do that as well. And the setup um involves Superlinked um and Sci the repo, so you can for the purpose of today, you can go to um the repo which is fully open source, so you can find it on GitHub. It's Superlinked Sci. Uh you can Yeah. So, this is it. This is the logo. And I'll be talking about um an integration that's uh Sorry, not here. So, an integra an MCP integration which is basically a package that we did, which is this one here. So, you can also have a look at this subdirectory. So, this is where most of the talk will be about, but I'll also mention why actually the inference was the bottleneck when building something like this. So, it was easy to figure out how to store the data, maybe in Obsidian and so on and have these connections between the data. Um That was the easy part. I could also use an API. However, if I wanted to do open source, if I wanted to do open source inference at scale and testing out multiple models, being able to deploy this in production, things got a bit difficult and that's why the infrastructure advantage that Sci offers, the fact that you're able to kind of deploy both a cluster and the models themselves was a very big advantage in my case. Um here you can see this um Obsidian example. So I'll work with kind of some of these things. Uh but you can also try this yourself. Um however, I will be using a managed cluster. We cannot um I cannot give that to you right now live. However, you can go to our website, which is this one, and here you have this apply for in inference grant. So you can go there and you'll receive your um API key that you can try and you can do things quicker. If not, you can do things locally cuz there is a Docker container and all that. Uh it will just take a bit slower to do these things, but I'd prefer if you kind of listen to me now and then try things after, but also in parallel if you want. And we also have this very cool um kind of QR codes memorabilia stuff where you can get an even better cluster, as I think, and you can find them here uh after the talk. So, um as I said, the idea was to have this privacy element and have a way to offload. So basically, I didn't want to disrupt my normal usage day-to-day. Um so I'm just working in cloud code and then if there's a specific task that needs to be done on my virtual private clouds that needs to be processed, that's sensitive, that costs a lot of tokens, I can just offload it and continue my work. So, the natural direction was to build some sort of um skill or plugin. But, this was even elevated more by building a whole MCP server that's that gets called in cloud code and there's that element of like using skills, but also having actually a dedicated edge that can actually do the heavy lifting. And you can also benefit from Superlinks inference, which I will talk about later on kind of how we've done this and what are the advantages of using um Superlinks inference engine. So, it saves tokens because for example, a number would be to have like maybe 10 pages of a PDF that has images, different details in there. So, if you pre-process that with good open-source OCR models and then return to your main agent harness or cloud code or whatever you're using, you get the context, but also benefit on the lower token spent and better context and so on. So, that's the general idea, especially for OCR this is apparent. However, there are multiple um kind of token saving advantages even beyond OCR. So, why MCP and not just an API or a plugin? Because we have a dedicated cluster, it was much better to kind of separate the responsibilities and have this as like the backbone to then build more details and more features further on, for which I will also talk about later. And everything runs on a server you can control, so through Sai you can pick your uh worker pools, you can pick your uh um GPUs basically, and understand which models are pinned where. And that flexibility from a cloud perspective was an advantage, and um I wanted to keep it that way, and that's why I went with MCP rather than just a skill. Um and it exposes the relevant tools. So, this is roughly how it would work um if it's just the plugin, it's also on your laptop or on device. In this way, you can kind of offload the same experience, but bring it to the cloud, which is your own cloud, and you can control things there. So, it's a much cleaner setup. Um Also, a lot of the examples here are with PII, so personally identifiable information, and there is a tool that gets called for like redacting specific details around documents, which is one of the central use cases of this privacy aspect. Um and the MCP setup, it has three tiers. Um so, there is You can see in the middle the Sci MCP. There is the MCP client um that's um kind of client and host through co-work or um cloud code or whatever you use, and that communicates with the Sci MCP server. And then you have actually Sci them cluster that has the um GPUs and the worker pools that do the work by communicating to the um MCP server, and you have different models that are available, and we also have a good queuing mechanism for this uh GPUs, um and I will go more into that a bit later. And the idea with MCP, as you know in general, is to have like one interface of communicating, especially if we're building more and more tools with this direction of model routing, because the demo and the version that we have now is around tool calling. However, we're building stuff around model routing and understanding which models to pick for different tasks based on um evals and based on costs. And all of these models would be open source and on your GPU. So, um there is this important direction as well. Um so, it's a forward-thinking approach. Um and there is the host, the client, and the server. Um you can do this like with any um um setup that you want, like whether it's cloud desktop, cloud code, and so on. And the communication is kind of done through JSON RPC. So, there are three like response, request, and a notification just to kind of uh initialize this and have this handshake uh protocol done so that you can authenticate the MCP edge and so on. And um an important detail is that the server exposes different stuff. So, in the MCP standard that has changed a bit since its inception. Um but the main thing is that it exposes tools that then um cloud code would use. However, in general, there is in kind of the MCP universe, there is a situation where um the client can also expose stuff like sampling or elicitation, which kind of makes this a two-way communication, so not like a standard client-server setup. However, in our case, just because this was more of a privacy setup, we do not do this. So, um basically, the side MCP edge um, only exposes the tools and they can be used, the ones that kind of were initialized in the handshake and that's it. Um, so it only kind of does this response as it wouldn't um, do this more complex tasks uh, in our case. And this is the idea that the tools are discovered and you just go from there. Basically, you uh, kind of have the tools which give you different um, abilities and um, you can run run that. Another detail is that um, you can do this also locally, so in the MCP universe again, the standard IO is like for local development. However, we wanted to use the streaming HTTP responses. This was done in the past by two end points um, that did uh, the streaming and one did the HTTP handshake, but now there is a standard that kind of does this all at once, so that makes things smoother. And these are the tools that we have, so I'll spend a bit more time on this slide. So currently, the kind of setup and uh, things you can try are around these models. So there is this tool that's like docs to markdown and um, that one uses OCR uh, models. Um, these are the open models that we have currently. So the idea is it saves token costs, it uses also a very good open source model. Um, a detail here is that with OCR models, especially the open source models are very good seri- you're not compromising in quality by offloading some of the work to open source um, OCR models. So, it's not that oh, I'm using open source, I'm sacrificing something. That's not the case at all, especially here and even in general, open source models are catching up and finding a way to bring them out there, bring them in production is um quite useful and quite important. There is extract entities, which would just generate the relevant entities in the docs with different NER models. There is or different tools around captioning and tagging. There is also um a capability around uh structure generation, uh summarization, question answering. And the good thing is also this is open source, so you can also do a PR and add a different tool there right now, and that would be something that kind of we can make make it as a part of this and um still it will utilize the inference, which is the bigger bottleneck in situations like this. Um and that would be the idea. You can kind of clone the repo that I showed before. And um you can bring up the cluster locally, but also uh if you again go on the website and request that, you'll receive an API key at some point. And this would be kind of the setup. You need to get uh the API key. You can also self-host if you're in for the longer term, let's say. Uh so, it doesn't have to be our cluster, it can be your cluster. Um and you just need to set these things up. I'll also be releasing so, an article that will kind of explain this in a bit more detail, and you can go step by step and run this. So, no no rush. Uh you can enjoy the talk and relax and try things out at your own time in many ways. And um here we have the repo and also the link to kind of getting the uh the compute and the cluster. So, you can basically request and you get some form that you fill out and then you'll receive the API key. And this is just the setup that I'll be going through and then trying out. And hopefully the demo gods are good. Someone mentioned this as uh a thing at one of the talks. I like the idea. Um and yeah, this is kind of just the setup um which I have kind of done to an extent. And I will now do the demo. So, I'm going to kind of cloud code. I am in the relevant packages directory. So, I can say, "Okay, can you So, I have basically just a set of files that are in an inbox um folder. So, I'll just show you here. So, for example, this is the graph and it's like inbox and you have different imagine these are different emails. So, this is all synthetic data but uh relevant data. I'm not using the real data that inspired this project but um obviously these are stuff like, "Okay, um there are some emails about an invoice and um some sensitive information all over and you can just go through this. There is um some NDA that would have like phone number and social security number, for example. So, that's how the data looks. And for example, here you can go and uh maybe say, "Can you uh redact the data from the NDA document. Something like that. Uh Cool, we get three credits. So, um the way things look, one example is this. So, this is some some of the stuff I tried before. So, this is the NDA and then you get this kind of like uh redacted uh document that extracts the relevant details, but obviously there is only email and person and the entities are masked. And all of this is done on your VPC. So, Cloud Code doesn't see this stuff or whatever agent you're using. Um and in certain situations you even get better performances with the OCR stuff. Um and So, for example, you can see here successfully loaded skill. I found the NDA document. So, here I've set up the cluster. So, I just input the endpoint and the API key. You would need to do this slightly different, but you can totally prompt even Cloud Code uh and so on. And here um we get Okay, the file contains the first redacted output um redacted 10 spans from this to this. Um and these are kind of the label counts. It gives us some info there. Um and maybe we can do something like Okay, can you pre-process the scans and also redact them to try to see So, this is in natural language. We try to get the OCR models to work and that's tool to be called. So, you can also configure the tool calling and all that. Um And also, there is this uh status line here that counts the tokens saved, but I need to also update this a bit to kind of do it properly, but that's also part of the um kind of advantage that you can actually see your saving on token costs. So, I'll pre-process the thing to markdown, then redact it. Um and it's working. And for example, we can see I'll just show you some of the files just so that I don't like go through all the stuff all the time. There is this um let's say contract before. So, this is how it looks, and this is a scan. So, I don't know if you can see now cuz it's a clean one, but this is you can see. So, it's an image. It's not like text, so it needs um OCR probably. And then, if we do the after one, this is some You can see markdown file here. Um that's extracting this stuff, but also redacting them. So, Oops, cursor. So, um the idea is that you pre-process the document through the OCR tool calling thing. You get um the token saving cost, the better context management, and so on, but you also get the redacted file, so personal information is not leaked. And this is also getting done in the meantime. So, that's the idea from kind of a functional perspective, but a lot of things are happening in the background. So, it's not just like um the privacy aspect and this looks easy, but the heavy lifting is done by this part. So, Superlink um and the inference engine and the cluster is set up in a way that it allows you to have multiple models, have hot swapping uh of models on different GPUs, um have smarter batching so that you can kind of test out multiple models. For example, if you have five Laura's uh fine-tuned to different sub areas of your um work, you can kind of hot swap between them and test out which one is the best even. So, we have a a gateway that um kind of um helps us connect this to different worker pools that each have um this batcher per worker so that the batching is done per worker and that gave us better performance instead of just having all the batching up front. And this is of course the architecture of the MCP uh solution. So, how it works with um Sai, so you can see this part on the right um is in your VPC, so both the kind of the MCP work that gets offloaded to Superlink Sai is in your own private cloud um and this is what gave us the advantage performance-wise, or at least one of the advantages, was the batching technique where um if you kind of do the batching without having one shared queue that then each of the workers pull from, in our testing and in general performance was lower, so we were able to kind of do this um setup and have an end-to-end improvement on latency and performance even with different models. And the idea is that you also want different worker pools because, for example, at some point you might want to try heavier model, but also have another one that can totally run really well on an L4 spot instance, so you mind be like, "Okay, I don't need to waste um kind of this more expensive hardware." And also we have auto scaling, so this um can kind of come down when not used and so on. So, um that's the setup, and there is the hot swapping of models. \[snorts\] So, um especially around smaller models that can fit into one GPU that are doing like named entity recognition or um even smaller generation or embeddings and so on, um you can benefit a lot from having multiple models on one GPU being switched around and swapped. And this goes in the direction of kind of what the MCP does and the tool calling aspect where you could build agents and have this inference part offloaded using Superlinked, and you can get different bundles of tasks set up. So, basically, document parsing like I explained or embedding content or writing code with um some generation models or extracting images or captions from data and so on. Uh or even like guardrails and structure generation stuff for policy checking and so on. So, I'd be also happy for you, so you get more credits if you um kind of build a project and so on. So, if you contribute a tool um that is maybe around content guarding or policy checking or whatever you find useful, um we'd we'd love to kind of uh have that as a contribution. And in some of our tests, we've also uh seen that due to this infrastructure that I explained, um we get very good performance around embedding models. So, we also have generation models, um but the tests that we have the benchmarks for for now are around embedding models. Um so, you get kind of um 10x lower price roughly um with very similar quality. So, you can see there the kind of the best model we had per task for embedding was 99.8% to uh kind of the frontier one. And this we evaluated this using MTEB tasks. Um so, yeah, that's um that's the case. We'll have more benchmarks and details around this soon as well. But, the point of this is the is that the sometimes, especially in production use cases and in with certain loads, you get the advantage um through the end-to-end improvement. So, if your infrastructure and architecture is set up well, you basically get the latency improvement and the better performance um even if your per model inference is not optimized. And we're doing both in our case, but that was a revelation that for production use cases was very helpful to know. And on the example around LoRAs, we've seen that you could kind of um out Laura's fairly quickly and fine-tune them. So, we're exploring work on that area. So, some of the colleagues worked on this German language specific Laura's that were fine-tuned and they were fairly cheap. So, having a way to do multiple Laura's and test them out in production and deploy them quickly seemed to be quite useful and improved performance in this very Latin niche use cases like German legal language, let's say. And this is kind of how \[snorts\] the MCP works. I mentioned a bit around this. And there are some failure modes that are already there, but again, this is working and it's well, but there are so many more things to do. And I'm kind of very happy to have your input and PRs and even just like comments. So, you can add me on LinkedIn and all that and I'm happy to chat always. And that's the idea that the same kind of MCP edge handles every document job, so you can do question answering, pulling facts, reading documents. Um So, yeah, this should also work and there is different again labels extracted and so on. And you can try this like this I think moved the number, but again, you can test this live and test out different tools with kind of your own demo data. And I've explained this to a certain point and what it does now. Um and yeah, there's the token saving aspect and that's very important um and not sending data to external APIs and it all runs in your cloud. Um and there is a lot of stuff to do kind of in the future. Um but the operational capability is still there, so we have the Helm charts and the Kubernetes cluster and you can use also Terraform apply to deploy the cluster, which is also open source, so it's it's not um hidden like kind of uh just the code is open source, but we have basically everything. Um and this can go beyond just documents. So, if we look at this idea of having the inference solved as a little box in your whole workflow, it unlocks a lot of opportunities that can go kind of beyond text with other models looking at transcripts um even graph-based data, generating ontologies and so on by different tool calls, structure generation. So, there are um different opportunities and um one direction that we're exploring as well is model routing within your own private cloud um which is quite powerful and I think the the field is moving a bit in that direction with usually teams exploring um this API-based optimization where you have uh a combination of models working together and the end-to-end performance is better. Um and there is an option to do something similar in this way. However, this is more like a future roadmap idea, so this is currently not built in, but um the idea is to also pick the models based on the task. So, the tool calling will not be tied just to the tool itself, but you could also understand which model to pick for the underlying tool. So, if I've only set up um one OCR model, I could say that, "Okay, for this tool call, I actually want OCR cheap." or some other sort of tool call that on the task adapts to what I'm trying to do. And um that's a powerful direction. And there's also playing around with like intent and there's this mixture of agents approach. Because if you already Another powerful aspect on why Sci is really good for this is that if you already have the models warm in your GPU, you can hit them much quicker and get the responses quicker. And you can do this model routing based on also the the how hot the model is or how kind of what what was evicted last, what wasn't. So, there is a lot of options also playing around this uh card of latency and trying to figure out what are the best models not just from a general perspective, but from an infrastructure perspective. And um doing this is uh quite useful and I I think a lot of um stuff will be done in this in this way um kind of fairly soon. And um yeah, this is the idea that you can kind of um estimate uh per model to find which models are warm. \>\> \[snorts\] \>\> Um and then in some cases it would be cheaper to hit a model that's already hot, but more expensive rather than reloading a model that's cheaper, but kind of not already there in your GPU. So, this logic can be set up as well to optimize costs and usage beyond the tool level. Um and I'm just mentioning this stuff just to kind of showcase how powerful um inferences and having good model inference setup um with all the infrastructure around it can actually unlock a lot of um use cases. There is also um a potential for doing stuff like um judges talking between each other or like the council of models and so on. Again, this is more of a future road map, but um this is a direction that also is an option for things like this where, okay, if you want to include some maybe reasoning tool calls or adjust the response based on like difficulty of tasks, which models are called, for example. An issue I had when building this was that for one of the summarizing tools, I had a um model that was um that had um thinking tokens. So, basically, the summarization was either very short or it didn't happen cuz everything was done on the thinking side. So, you have to also kind of keep in mind what models you use and whether you toggle the thinking tokens or not. Um But, sometimes reasoning can be beneficial and um it's useful to have a way to do that quickly from an infrastructure side. And of course, there are some kind of um downsides to having um a router or a council and so on. However, um um we're also very strong on evals and trying to test models before doing this. So, um there are different ideas here that can be done. Um and yeah, um basically we don't want to also spend GPU on some models reasoning forever and then nothing happens um or also have biased models as judges um or kind of optimizing something that um then becomes kind of by optimizing something you kind of escape from making it as a real target, but it just becomes something that like the models hack and never actually um use it you can never use that as a proper evaluation. Um and you can look at our repo. So, this is the kind of the QR code if you haven't yet. Um and I think the main takeaway for today is that infrastructure and inference is far more important even for this very API friendly or ergonomic designs where you just work with cloud code and you can kind of offset your um sensitive data, your specific tasks in different ways and um still the backbone is uh the inference. And for example, I will go here again. So, this will be published as well and um you can also find kind of what files are supported, where the Laura aspect feel uh fits, um what are the kind of the troubleshooting guides and so on. And also um the most important thing is ask for a kind of your API keys and so on. We also have a full model catalog where kind of also revamping this to be more detailed and useful, but basically you can have different you can select different output types, different architectures because in order to optimize the models and the inference you need to adapt models in different bundles. So you need to kind of all models maybe implement attention differently. So you cannot run them and run the forward pass in the same way. So you need to have different model run times for different model families so that this works and there are a lot of details around this and you can see that models are tagged based on like whether they do encoding or scoring. So you can also use scoring models that do like re-ranking basically if you even in this use case of the tool calls and the advantage there is that you can basically play with your own version of retrieval through this MCP and so on and I would really love if you could kind of contribute to this or give us ideas on how to kind of set this up and build more stuff on top that people find useful. We did a hackathon in London with Tech Europe and DeepMind recently where a lot of people kind of tested this out and were kind of happy to work both on the retrieval and re-ranking side but also generation. So having the the way of doing both but being supported by that inference is is very cool. And again I encourage you to take this um free um kind of cluster infrastructure credits that you can use. And we also have a lot of examples and um full kind of docs that you can have a look at. Um basically a quick start and then different model versions and how you can also look at how you can migrate, for example, if you have some workflow done with like OpenAI API, then you're like, "Okay, cool. I want to test this thing out quickly. What can I do?" You have those um options as well. And uh this is the repo. So, again, this is the package um and here you have kind of what it exposes um and how the credentials are set up. So, the the readme here um can be um checked out. This is the example. Also, if you contribute with examples and have something more here, that would be very useful. And um this is where you apply for the credits that um help you to build build things and try things out. Uh this was very popular at the hackathon, so uh very kind of happy for you to try this out. And I think since we're slowly approaching the end, I can take some questions as well. Um you'll just kind of speak and I'll restate and reiterate the question because of the recording. So, yeah, I'm happy to answer a few things and we can also play around with this more if you have some query that you want to try. Um yeah. \>\> So, uh using the empty group So, it wouldn't do that basically. So, the concern is around uh, the question is around privacy. So, basically you just Claude would just get the redacted version of the document. So, it wouldn't get the full document that you're inputting there. So, that's kind of the the setup that we have currently. So, it would only see that side of things. Yeah. Yeah. Yeah. Yeah, so I mean, this can be kind of evaluated like whether this happens like whether Claude does this kind of off off-rails situation. Definitely, that can be kind of tested more and more, but in the test that I've done like if you host the kind of MCP and everything else is on the virtual private cloud, this seemed to have worked without kind of the data leaking. But, I'm sure there can be more guardrails to kind of reinforce this at that initial stage or in the worst case even you could say, "Okay, I'm concerned. I can build my own harness and use an open-source exactly." Yeah, yeah. Yeah, yeah. And in that way even um, like you could still save on tokens and get better context even if like this is Claude code in my example, but it can be any harness. If you implement the tool calling well, I think this can be done, but happy happy to get a contribution around this as well. Um, Could be very useful to have this like pre-processing layer as well. Um Yeah, another Yeah. Yeah. Yeah. Yeah. Mhm. Yeah. Yeah. So, again the question is kind of like how do we set this up to make sure that the privacy is well preserved, right? So, on on the on the side side of things, so the cluster in this case is like what we have, but that can be your own cluster. So, through the repo you can set up your own cluster in your own Yes, exactly. Yeah, so you can configure the cluster in any way you want. I'm just using a managed cluster that we have for the purpose of the demo and the example and we use that for example for hackathons and so on. So, we host the hardware, but you can totally do this yourself. So, it's self-hosted and in the as I said in the repo you have the ability to do Terraform apply and have all the infrastructure set up and if you have the GPUs and so on, you basically um yeah, have that set up and it's totally your own. Sorry, again? Does the MCP have a file? So it just sends it to the main worker and there the file is processed. So kind of there is that yeah. And yeah, you can again I think at the entrance there is the free compute that you can get to try out. Again, if you want to self-host, you can still self-host. It's not um strict in that sense. We just do this so that things can go quicker. But again, you don't have to do it in a way that you use our cluster. You can self-host. And yeah, we're also you can kind of add me on LinkedIn. Again, we can chat there and um Thank you for attending.