We fine-tune open-weight AI models on our own hardware and run them in production. Jesse has also given talks on the process, most recently at Boston AI Week. This post is the condensed version of what we cover: the steps we follow, where the real work is, and what tends to go wrong.
If you want the full first-person walkthrough, Jesse wrote it up in detail here.
First, decide whether you should
Fine-tuning is the heavier, slower, more expensive tool. A lot of the behavior people reach for it to fix can be handled with a better system prompt, or with retrieval over your own documents. We walk through that tradeoff on RAG vs. fine-tuned models. Everything below assumes you have made that call and fine-tuning is the right move.
What “your own model” actually means
Almost nobody trains a model from scratch. Pretraining a foundation model is a game for a handful of very well funded labs. What everyone else does, and what we do, is take an open-weight model one of those labs already trained and specialize it for a specific behavior.
That result is still a real, proprietary artifact. The base model is the foundation someone else poured. The fine-tune is the part that makes it yours.
The process
- Choose a base model. Two things drive it: what the model needs to do, and how much GPU memory you have. Fine-tuning nudges behavior; it does not rescue a model that was never good at the underlying task. Use the base model on real work before you commit to it.
- Define the behavior change. Not “make it better.” What should it do differently, in which situations, and what should the output look like? A fuzzy goal produces fuzzy data, which produces a fuzzy model.
- Establish a baseline. Build an evaluation suite before touching the weights and save the base model’s outputs. The base model should fail the tests that target the new behavior. That failure is what you are about to fix.
- Generate synthetic training data. We generate it on our own GPUs, so there is no per-token cost and no data leaving the building. Then inspect it carefully. Bad examples teach bad behavior just as efficiently as good ones teach good behavior.
- Set up a small LoRA adapter. Instead of updating billions of parameters, you train a small set of extra weights alongside the frozen base. Keep it as small and narrowly aimed as the change requires. A bigger adapter has more capacity to learn, and more capacity to forget.
- Smoke test and back up. Run a thirty-second training job to catch broken data paths and config mistakes, and keep a pristine copy of the base model.
- Train. With QLoRA, the base is compressed to 4-bit in memory and frozen while the adapter learns, which is what lets a large model train on a single GPU. This part is anticlimactic: the loss comes down, and a run can finish in about an hour.
- Evaluate. Run the same suite again. The tests that failed at baseline should now pass. Compare the base and the fine-tune at the same precision with the same settings, so the adapter is the only variable.
- Merge and re-quantize. Bake the adapter into the full-precision base, then compress the merged model to 4-bit so it fits on the serving hardware.
- Evaluate again, quantized. Compression can subtly shift behavior. A model that tested perfectly at full precision can quietly degrade once it is shrunk.
- Serve it. We serve with vLLM, on hardware the client controls.
Three of those eleven steps are measurement. That is the point.
Check for forgetting, not just the new trick
A model can learn its new behavior so aggressively that it gets worse at everything else. Our suites include general-capability tests that have nothing to do with the target behavior, plus checks on the specialized skills the product depends on, such as structured tool calling. Those scores should not move.
What catches people off guard
- Merging is often not optional. Some serving stacks can apply an adapter at runtime, but not every architecture supports that on a quantized model. Merging sidesteps the problem entirely.
- Quantization needs calibration data. It is not blind rounding. Real examples run through the model to decide how to compress each layer with the least damage. We use a sample of the training data so the calibration matches real use.
- The quantize step can run out of memory. Compressed layers pile up in system RAM over a long run. Watch memory headroom on quantization, not just training.
- Mixture-of-Experts models need selective quantization. These models have an internal router that decides which expert handles each token. Compress the router along with everything else and quality degrades in ways that are hard to diagnose. It has to stay at full precision.
Be honest about the result
“Passes the suite” and “flawless on every conceivable prompt” are different claims. A good fine-tune holds up under the conditions it actually runs in. Push it into adversarial corners it was never trained for and you can still find seams. Know where those are, and decide whether they matter for the real workload.
Want help with this?
We run fine-tuning engagements end to end on dedicated on-premise hardware, so your data never leaves your building, during training or in production. If you have years of data a generic model knows nothing about, talk to us.