# Knowledge Mining - darinbuilds.com

> Transform YouTube content into a searchable knowledge bank. Dual-mode search, local ML processing, creator personas, and MCP integration for Claude Code.

## The Problem

3-hour videos. Brilliant insights buried at minute 47. No way to search, no way to find them again. So I built a knowledge system that turns every spoken word into searchable, queryable knowledge.

## The Pipeline

From YouTube URL to searchable knowledge:

- **Ingest** - Paste a YouTube URL. Transcript fetched automatically. YouTube Transcript API pulls the full spoken content. Metadata via oEmbed. Background job queue with retry logic.
- **Chunk** - Content split into searchable segments automatically. Sentence-boundary aware splitting preserves meaning. Overlapping segments ensure nothing is lost between sections.
- **Embed** - Each segment converted to a searchable representation - locally, no external APIs. The model runs locally - zero external API calls, zero cost per query. Batch processing with automatic caching on cold start.
- **Search** - Search that understands meaning and catches exact words. Two search strategies - semantic similarity and exact keyword matching - fused into one ranked result set. Optional temporal decay for recency bias.

## Search System

Two search strategies work in parallel. One understands meaning - finds results about a topic even if those exact words never appear. The other catches exact terms. Results found by both methods get ranked higher.

Here's the actual code. I broke it into the three decisions that made it work.

### search.ts - both searches, in parallel

Vector search alone kept whiffing on exact matches. Search "useEffect" and you want that literal string, not the nearest-meaning thing. Both run in parallel, so it's one extra await, not double the wait.

Source: https://github.com/devobsessed/sluice/blob/main/src/lib/db/search.ts

### mergeAndRank() - rank by position, not by score

You can't average two scores that live on different scales, and cosine similarity and keyword relevance don't. So I rank by position in each list instead. No wrestling two scoring systems onto the same axis.

Source: https://github.com/devobsessed/sluice/blob/main/src/lib/db/search.ts

### search.ts - trim to what you asked for

I pull 2x the limit from each search before merging. That gives the fusion step room to actually reorder things. Ask for exactly the limit and there's nothing spare to boost - you starve the merge.

Source: https://github.com/devobsessed/sluice/blob/main/src/lib/db/search.ts

## Zero External API Calls

The model runs locally. No OpenAI. No Cohere. No API keys. A lightweight ML model processes content directly on the server - no external API calls, no per-query costs.

### local-model.ts - every query costs nothing

Hosted embedding APIs charge you per call, so you start rationing - you don't re-embed a whole library on a whim when the meter's running. This model is tiny, like 23MB, and it runs right on the server. For chunked YouTube transcripts a small model is plenty, and every query being free means I can re-embed everything or run a batch backfill whenever I want.

Source: https://github.com/devobsessed/sluice/blob/main/src/lib/embeddings/service.ts

## Creator Personas

Ask the creator. Get answers grounded in their content.

- Auto-generated at 5+ videos per creator
- Expertise profile built from all ingested content
- "Who's best?" routing matches questions to the right creator
- Ensemble mode: top 3 personas stream in parallel via SSE
- Every answer grounded in actual spoken content - not hallucinated, not generic

## MCP Integration

4 MCP tools expose the entire knowledge bank to Claude Code workflows:

- **search_knowledge** - Dual-mode search with semantic and keyword matching, optional creator filtering (input: topic, creator?, limit?)
- **get_list_of_creators** - List all YouTube channels in knowledge bank with video counts (input: (no input))
- **chat_with_persona** - Query a specific creator persona with responses grounded in their actual content (input: personaName, question)
- **ensemble_query** - Ask multiple personas simultaneously, top 3 parallel responses (input: question)

### mcp.json - meet the AI where it already lives

I could've built a chat UI into the app, but then you're copy-pasting transcripts in and out all day. So I exposed the whole knowledge bank as an MCP server instead. Now Claude searches it, lists creators, and talks to personas right from the terminal - I brought the knowledge to where the AI already lives instead of making it come to me.

Source: https://github.com/devobsessed/sluice/blob/main/docs/mcp-tools.md

## Stats

- **11** database tables
- **100%** local processing
- **4** MCP tools
- **100+** searchable resources

## Tech Stack

Next.js 16, React 19, TypeScript, PostgreSQL, Drizzle ORM, Local ML Model, Claude API, MCP SDK, Tailwind CSS v4, Vitest