YANTRAKAR

PRIVATE AI

AI that runs inside your network.

For data that can't leave the building. The software is mostly open source. My work is choosing a model per use case and making it run on your hardware.

What teams want AI on

Teams want AI on their business intelligence, source code, meeting notes, finance data, project plans and draft quotations. Stopping AI use is not an option anymore. When the data has to stay in-house, the models can run inside your own network.

  • Business intelligence
  • Source code
  • Meeting notes
  • Finance data
  • Project plans
  • Draft quotations

Where your data goes

Each service has its own rules. These are the tools most teams already have.

FIG. 04WHERE YOUR DATA GOES
Where your data goes: what each service keeps, whether it trains on it, and the source
ServiceWhat is keptTrainingSource
GitHub Copilot chat outside the editor (web, CLI, mobile)Prompts and suggestions, typically 28 daysNot used to train the foundation models (per GitHub)Microsoft Learn: GitHub Copilot data
Microsoft Copilot (consumer)Conversations stored 18 months by defaultUsed for training by default in many countries; signed-in users can opt outMicrosoft: Privacy FAQ for Microsoft Copilot
Microsoft 365 Copilot (work)Prompts and responses kept as your admin's retention policy setsSet by your organisation's agreementMicrosoft Learn: Retention for Copilot
Private AI inside your networkWhatever you decide, on your own disksNothing leaves, so nothing is trained elsewhereYour own policy

Defaults change. This table is a starting point for your own check, not legal advice.

What each service keeps by default, and whether it trains on it.

A model chosen for each job

So far I see that the largest frontier models are not always needed. What helps is a harness, a small layer that routes each request, searches your documents and checks access, plus an open-weight model that is good enough for each job. My first attempt in 2024 failed because open-weight models weren't ready. They are now good enough for many everyday jobs, and lower running costs made it practical to use hardware a company already has.

FIG. 05A MODEL PER JOB
One harness and a model per job, inside your network Inside a dashed boundary labelled your network and your hardware: people and tools (code editor, documents, chat) send requests to a harness made of a router, an index of your data and access control. The harness sends each request to one of four open-weight models: a coding model, a general model for docs, slides and research, a vision model for images and video, and an embedding model for search over your data. A label outside the boundary reads nothing leaves. YOUR NETWORK · YOUR HARDWARE PEOPLE AND TOOLS Code editor Documents Chat HARNESS Router Index of your data Access control OPEN-WEIGHT MODELS Coding model code and reviews General model docs, slides, research Vision model images and video Embedding model search over your data NOTHING LEAVES One harness and a model per job, inside your network Inside a dashed boundary labelled your network and your hardware: people and tools (code editor, documents, chat) send requests to a harness made of a router, an index of your data and access control. The harness sends each request to one of four open-weight models: a coding model, a general model for docs, slides and research, a vision model for images and video, and an embedding model for search over your data. A label outside the boundary reads nothing leaves. YOUR NETWORK · YOUR HARDWARE PEOPLE AND TOOLS Code editor Documents Chat HARNESS Router Index of your data Access control OPEN-WEIGHT MODELS Coding model code and reviews General model docs, slides, research Vision model images and video Embedding model search over your data NOTHING LEAVES
Each request goes through the harness to the model that fits it. Everything stays on your hardware.

How a delivery goes

Setting it up

  1. Use-case study. Which work, which data, who uses it, how often.
  2. Model choice. An open-weight model per use case, tested on your own examples.
  3. Deploy on your hardware. Automated with Ansible, so it can be rebuilt and moved.
  4. Index your code and documents. A local indexer, so answers come from your material.

After that, ongoing

Monitor and upgrade. Open-weight models improve quickly. Swapping one in is planned for from the start.

Before you start

Everyday interactive use needs capable hardware, so I would not judge the idea on a laptop test. Some work runs well on modest hardware, such as the two first use cases below.

START HERE

Two good first use cases

Documentation and user guides.

This work can run over longer time slots, so it fits modest hardware and gives useful output early.

Old codebases.

Walking through old code with a local indexer, especially when the code can't be disclosed.

What I can do for you

Private AI plan

An advisory session on use cases, hardware and which models to try first.

Deployment

Open-weight models and tools running on your hardware, automated with Ansible, with Nix where it helps.

Code and document indexing

Local search and questions over codebases and documents that can't be disclosed.

Contact

Tell me which data and which work you have in mind.

Stavanger, Norway

Remote across EU and India time zones, on-site when needed

Your details are used only to reply to this message.