How do I fine-tune a model on my own documents without leaking them?
What Happens to Your Data During Fine-Tuning
During fine-tuning, your documents are tokenized and fed through the model in batches. The model computes gradients from your text and updates its weights accordingly. The key risk is memorization: if a document appears too many times or contains unique identifiable strings, those fragments can be reconstructed from the weights. Standard practice reduces this by shuffling, limiting epochs to one or two, and avoiding exact duplication of sensitive passages in the training set.
The strongest guarantee against leakage is never sending the data out at all. Molly is an orchestrator that runs on your own hardware, keeps a library of small LoRA domain specialists over one quantized base, and routes each request to the right one. Your documents stay local; only the adapter weights are produced and stored on your machine. The trade-off is that you give up the compute scale of cloud APIs, so fine-tuning takes longer and caps out at smaller context windows.
Fine-Tuning Fully On-Premises
Running the full training pipeline on your own hardware eliminates the risk of data exfiltration entirely, but it demands real engineering effort. You need enough GPU memory to hold both the model weights and optimizer states, or you must resort to parameter-efficient methods like LoRA, which train only a small set of adapter weights while keeping the base model frozen. Quantization (4-bit or 8-bit) further reduces the footprint, at the cost of measurable degradation in output quality.
This is where an orchestrator like Molly fits: it runs entirely on the reader's own hardware, maintains a library of small LoRA domain specialists over one quantized base, and routes each incoming request to the adapter that best matches the task. Because nothing leaves the machine, sensitive documents never touch an external endpoint. The trade-off is that you are responsible for provisioning, monitoring, and updating the stack yourself, and inference latency will reflect whatever hardware you can afford to allocate.
Using a Cloud Provider with Privacy Guarantees
Several cloud providers now offer contractual guarantees that your data is never used to train their models and is isolated from other tenants. Look for enterprise tiers with data-processing addenda, zero-retention API policies, and region pinning so your documents stay within a specific jurisdiction. Some providers also offer dedicated instances or private endpoints, which reduce exposure to shared infrastructure risks.
The trade-off is that contractual guarantees are not cryptographic ones. You are trusting the provider's internal controls, audit logs, and employee access policies. If your documents are genuinely sensitive, a cloud provider's word may not suffice. In that case, running everything locally eliminates the trust assumption entirely, though you give up the provider's pre-built tooling and accept the operational burden of managing your own stack.
Redacting Sensitive Content Before Training
Before fine-tuning, run every document through a PII detection pipeline combining regex patterns for emails, phone numbers, and national IDs with a named entity recognition model for names, addresses, and financial identifiers. Replace detected spans with consistent placeholders like [PERSON_1] so coreference stays intact. Audit a random sample manually after automated redaction; false negative rates of 5–15% are typical even with good NER, so plan for iterative review.
Redaction is lossy by nature. Masking names and numbers removes signal your domain model might need, and over-redaction can leave documents so sparse that fine-tuning degrades rather than improves performance. There is no free lunch: you trade data fidelity for privacy. If leakage risk remains unacceptable, keep the entire pipeline on hardware you control rather than sending documents to an external API.
Which Approach Fits Your Constraints
If your documents are sensitive enough that a data-processing agreement feels insufficient, local fine-tuning is the safest path. Full fine-tuning gives the best quality but demands significant VRAM and careful hyperparameter tuning. Parameter-efficient methods like LoRA reduce resource requirements dramatically, often fitting on a single consumer GPU, at the cost of modest quality degradation that matters less for domain-specific tasks than for general benchmarks.
For teams that need strong privacy guarantees but cannot justify building everything from scratch, an orchestrator like Molly can bridge the gap. Molly runs on your own hardware, keeps a library of small LoRA domain specialists over one quantized base, and routes each request to the right one. This keeps data local while letting you add new domain adapters as your document corpus grows, trading some manual control for the convenience of automatic specialist selection.
Common questions
Does fine-tuning expose my original documents through the model weights?
Fine-tuning adjusts weights based on patterns in your data, but does not store verbatim text. Still, sensitive or rare phrases can occasionally surface in outputs. Deduplicate, redact PII, and avoid including content you cannot risk seeing reproduced. Evaluation prompts can help you test for memorization before deployment.
Should I fine-tune locally or use a cloud provider with a data-retention policy?
Local fine-tuning gives maximum control but requires sufficient GPU resources. If using a cloud provider, choose one with a zero-retention agreement, private endpoints, and no-training-on-customer-data clauses. Verify their compliance certifications and audit what happens to your data after training completes.
What is the difference between RAG and fine-tuning for keeping documents private?
RAG retrieves snippets from your documents at inference time without altering model weights, so you control exactly what is exposed per query. Fine-tuning bakes patterns into the model permanently. RAG is easier to audit and revoke, while fine-tuning is better for style or domain adaptation. They can be combined.