---
title: "Letting a small model choose what the big one reads"
description: "On older cards, a question over 250,000 tokens of open-source code took my local model 12 minutes. With smaller models choosing what it reads, it still got 12 of 12 plain-language questions right while reading 48 times fewer tokens. On a private codebase it went from 16 to 37 of 41 correct, with search alone already at 35."
author: Sandeep Shelke
author_url: https://www.linkedin.com/in/sandeepshelke/
publisher: Yantrakar AS
date: 2026-10-28T12:00:00+01:00
url: https://yantrakar.com/model-reads-less.html
image: https://yantrakar.com/assets/blog/model-reads-less/og.png
---

# Letting a small model choose what the big one reads

In my [first article](https://yantrakar.com/running-ai-inside-your-network.html) the open problem was long context. My main model can take a whole codebase, but on an older 8 GB card it reads for minutes before it writes a word. These are my notes from three test runs where smaller models on other cards decide what it reads: two on open-source code, and one on a private codebase.

## In short

- A question over 250,000 tokens of source code took Qwen3.6-35B-A3B about 12 minutes to answer. About 11 of those minutes were spent reading.
- Keyword search and an embedding model now shortlist pieces of the code, and a 9B model picks the 4 that matter. The big model reads about 5,000 tokens instead of 250,000.
- On 12 plain-language questions it got 12 of 12 right, the same as reading everything, with 48 times fewer tokens. About 2.3 minutes per question as run, and I estimate about 52 seconds with a prebuilt index.
- On a private, mostly C++ codebase across 10 repositories, 41 questions: the existing code-search tool led to 16 correct answers, the combined shortlist with the 9B picker to 37. Search alone, reading four times as much, scored 35, so how much of the gain comes from the picker is not settled yet.
- No single search method won both open-source tests. Keywords catch exact names, the embedding model catches different wording, and the picker turns a long shortlist into a short one.

## The problem

The test corpus is open-source code: Redis 7.2.4 and Caddy, about 250,000 tokens. Qwen3.6-35B-A3B writes every answer, on one older 8 GB card with its expert layers in main memory. Reading the whole corpus before each answer takes about 11 minutes, and the answer arrives after about 12.

The other models on the cluster sat idle while that happened. The question was simple: can they cut that time, without losing accuracy, by having the big model read less?

## The pipeline

**Fig. 01 · Small models choose, the big model reads.** Keyword search and an embedding model shortlist chunks of code. A 9B model asks one yes/no question per chunk and keeps the top 4. Only those reach the big model.

```mermaid
flowchart LR
  src["Source code<br/>about 250k tokens"] --> ch["Chunks<br/>about 208 × 1.4k tokens"]
  ch --> kw["Keyword search (BM25)<br/>CPU, 0.1 s"]
  ch --> em["Embedding 0.6B<br/>11 ms/query, indexed"]
  kw --> fu["Shortlist<br/>16 or 24 chunks"]
  em --> fu
  fu --> cl["Clef-Flash 9B<br/>one yes/no per chunk, keeps 4"]
  cl --> qw["Qwen3.6-35B<br/>reads about 5k tokens, answers"]
```

1. The source is split into chunks of about 1,400 tokens, on line boundaries with an 8-line overlap. That gives about 208 chunks.
2. A shortlist comes from keyword search (BM25, on the CPU, 0.1 seconds), from the embedding model, or from both merged by reciprocal-rank fusion.
3. Clef-Flash 9B asks one question per shortlisted chunk: does this code define or return something the question needs? It keeps the 4 chunks with the highest probability. This takes about 17 seconds.
4. Qwen answers from those 4 chunks, about 5,400 tokens, with the same system prompt, judge, seed and output limit as the full-context run.

Clef's probabilities were low and poorly calibrated: the top chunk was often below 0.1. Its ranking was useful, so I keep the top 4 rather than using a threshold.

## Test 1: exact names

The first set had 24 questions about facts I planted in the code. Each planted fact carried a unique tag, so keyword search worked almost like grep.

**Table · Test 1, 24 questions with exact names**

| Pipeline | Correct | Big-model tokens | Per question |
|---|---|---|---|
| Reads all 250k | 21/24 | 250,000 | 12.1 min |
| Keywords → Clef | 24/24 | 5,405 | 54 s |
| Keywords, top 4, no Clef | 21/24 | 5,271 | 37 s |
| Keywords, top 16, no Clef | 24/24 | 21,010 | 59 s |
| Hybrid → Clef | 18/24 | 5,400 | 143 s |
| Hybrid, top 4, no Clef | 12/24 | 5,382 | 123 s |
| Embedding → Clef | 8/24 | 5,398 | 134 s |
| Embedding, top 4, no Clef | 6/24 | 5,300 | 120 s |

- Reading everything missed 3 questions: two answers came back as code-fenced JSON and one never finished. With a short context, those failures went away.
- At the same 4-chunk budget, Clef raised keywords from 21 to 24, hybrid from 12 to 18 and embedding from 6 to 8. It cannot add a chunk the shortlist missed.
- Giving the big model all 16 keyword chunks also scored 24/24. What Clef bought there was 4 times fewer big-model tokens: with 4 chunks the big model's run took 37 seconds per question (the top-4 run without Clef), against 59 with all 16.
- The embedding model found the right chunks in only 6 of 24 cases. For exact identifiers it was the wrong tool.

## Test 2: plain words

The planted tags favoured keyword search, so the second set removed them. It had 12 real facts that already exist in the unmodified Redis and Caddy source, asked in plain words, for example "how many parallel clients does the Redis benchmark tool open by default?" (50) and "after how many MB does Caddy's file logger rotate?" (100). Each answer line appears exactly once in the corpus. A closed-book run, with no code at all, checks that the model is not answering from memory.

**Table · Test 2, 12 plain-language questions**

| Pipeline | Correct | Big-model tokens | Per question |
|---|---|---|---|
| Reads all 250k | 12/12 | 249,922 | 11.6 min |
| Closed book, no code | 0/12 | 132 | 23 s |
| Keywords → Clef | 9/12 | 5,117 | 42 s |
| Embedding → Clef | 10/12 | 5,186 | 2.2 min (47 s indexed) |
| Hybrid, 16 → Clef | 11/12 | 5,280 | 2.1 min (42 s indexed) |
| Hybrid, top 16, no Clef | 11/12 | 19,962 | 2.3 min |
| Hybrid, 24 → Clef → 4 | 12/12 | 5,244 | 2.3 min (52 s indexed) |

- The hybrid shortlist of 24, cut to 4 by Clef, matched reading everything: 12 of 12, with 48 times fewer tokens for the big model.
- Keywords failed when the question used other words than the code: "rotate … MB" against `RollSizeMB`, and "DNS lookups cache" against `maxCacheEntries`. The embedding model caught those.
- Clef lifted the DNS-cache answer from rank 17, which the 16-chunk run without Clef missed.
- Closed book scored 0 of 12 because the model said it did not know every time, so the correct answers most likely came from the code it was given.

**Fig. 02 · Time per question, test 2.** Bars drawn to scale. The two measured runs both got 12 of 12. The third bar is an estimate, not a run: the 85-second corpus embedding is replaced by the measured 11 ms query embedding.

| Run | Time per question |
|---|---|
| Reads all 250k | 11.6 min |
| Hybrid 24 → Clef → 4, as run | 2.3 min |
| Same, with a prebuilt index (estimated) | about 52 s |

## Test 3: private code

Tests 1 and 2 ran on open-source code. The third ran on a private, mostly C++ codebase across 10 repositories: about 178 million characters of indexed text, which I estimate at tens of millions of tokens. That is far beyond the model's 262,000-token context, so there is no read-everything baseline here. This test is about better answers, not speed.

Each of the 41 questions asks for one number that is set in exactly one place in the code, in plain words with no code identifiers. Qwen wrote the questions, and 17 drafts were rejected, 15 of them because they named an identifier. A made-up question of the same kind, not one of the 41: "How long does the service wait for a reply before it gives up?"

The answering model and the shortlist sizes are the same as in test 2. The shortlist merges three sources: the existing code-search tool (the Zoekt and Universal Ctags indexer from my first article), keyword search and the embedding model.

**Table · Test 3, 41 questions on private code**

| Pipeline | Correct | Big-model tokens | Per question |
|---|---|---|---|
| Existing tool, top 4 | 16/41 | 4,600 | 39 s |
| Combined search, top 16, no Clef | 35/41 | 17,400 | 65 s |
| Combined search, 24 → Clef → 4 | 37/41 | 4,400 | 57 s |

- With the existing tool's top 4, the model gave 16 correct answers, declined 23 times because the code it got did not hold the answer, and got 2 wrong.
- The combined shortlist cut to 4 by Clef gave 37 correct, 3 declined and 1 wrong.
- Search alone, top 16, scored 35 while the big model read four times as much. Clef got 2 more right and none fewer, but 2 of 41 with one run per question is close to noise.
- The existing tool failed with a Zoekt error on one question, which was scored as an empty result.
- Building the embedding index once took 244 minutes on one of the older cards.

## What it means for the hardware

- In test 2 the big model's own stage took about 28 seconds per question (27.6 s measured) instead of 11.6 minutes. Most of the rest of the 2.3 minutes is the embedding (85 s without an index) and Clef (about 17 s). Clef and the embedding model run on cards of their own.
- The embedding model costs about 85 seconds per question when it embeds the whole corpus each time. With an index built once, a query embedding takes 11 ms, so that cost disappears.
- The freed time on the big card is throughput: more questions, or other work, on the same hardware.

## What I do not have yet

- One run per question, and small sets: 24, 12 and 41 questions.
- The best pipeline is not the same in both tests. In test 1 the hybrid shortlist used 16 chunks and scored 18/24; the 24-chunk hybrid that won test 2 has not been run on test 1 yet.
- The indexed times are derived from measured stage times, not measured end to end.
- Some answers also appear in doc comments, so a few questions are easier than they look.
- Tests 1 and 2 are open-source code; test 3 is one private codebase.
- Test 3 only asks for single numbers set in one place. Answers spread across files, design questions and call chains are not tested.
- Test 3 has no read-everything baseline, so it says nothing about speed.
- In test 3, Qwen wrote the questions and also answered them, which could favour retrieval.
- In test 3, whether the gain comes from the picker or from better search is not settled: 37 against 35. That test is next.

## Where I would go from here

1. Shortlist with both keywords and an embedding model, because each misses what the other catches.
2. Let a small model pick the few chunks that matter, and use its ranking rather than a threshold.
3. Build the embedding index once, so only the question is embedded at answer time.
4. Keep a closed-book run in every test, to see whether the answer came from the code or from memory.
5. Put the small models on their own cards, so the big one only writes.

This is the work I do at Yantrakar: AI that runs inside a company's own network, sized on real use. To talk about your setup, use the [contact form](https://yantrakar.com/model-reads-less.html#contact) or find me on [LinkedIn](https://www.linkedin.com/in/sandeepshelke/). The setup itself is described in [Running AI inside your own network](https://yantrakar.com/running-ai-inside-your-network.html).