Fine-Tune a Local LLM with Unsloth and Serve It with Ollama

Your model knows the right facts but keeps answering in the wrong tone. Prices never come out wrapped the way the business wants, and the output format your ops team expects never sticks. You sharpen the system prompt, and within a few turns it drifts back.

That is a behavior problem, not a knowledge problem. And RAG does not fix behavior. RAG adds facts, not habits. When the issue is how a model writes, formats, or follows your domain conventions, fine-tuning is the right tool.

The good news: this does not need a GPU cluster. With QLoRA you can adapt a 7B model on a single GPU with 8-16GB of VRAM. A free Google Colab T4, which has 16GB, is enough. This guide walks the whole pipeline: a small clean dataset, Unsloth for training, an export to GGUF, and a local server through Ollama.

Prerequisites

• Python 3.10+ with pip
• A GPU with at least 8GB VRAM. A free Colab T4 (16GB) works
• A training dataset in JSONL, which you build in step 2
• Ollama installed on your machine for the final step

Step 1: Decide whether you actually need fine-tuning

Fine-tuning changes behavior and output format. It does not add new facts to the model. If your problem is that the model does not know recent information, use RAG or a search tool instead and skip the training run. If your problem is that the model will not follow your tone, your schema, or your domain writing conventions, fine-tune.

Here is the mental model that keeps me from wasting runs:

• Prompt engineering: the behavior is close but sags at the edges. Try this first, it costs nothing.
• RAG: the model lacks up-to-date or private facts. Add retrieval.
• Fine-tuning: the model consistently refuses to write the way your domain demands, no matter how you prompt it. Train.

Step 2: Build a small dataset

SFT wants prompt-completion pairs. Do not start with a hundred thousand scraped lines. A few hundred examples that you have actually checked beat a few thousand noisy ones in most small-domain cases.

Create a dataset.jsonl with one JSON object per line:

{"instruction": "Turn this ID into a tracking link: ORD-1042", "output": "Your package is on its way. Track it here: https://portal.example.com/ORD-1042"}
{"instruction": "Turn this ID into a tracking link: ORD-1199", "output": "Your package is on its way. Track it here: https://portal.example.com/ORD-1199"}

Make 5-30 of those examples the hard cases the model currently gets wrong, not only the easy ones. The model learns proportionally to what is in the file, so the trickiest behaviors need the most examples.

Step 3: Install Unsloth

pip install unsloth

Unsloth is a wrapper around Hugging Face Transformers and PEFT. It adds custom kernels that reduce VRAM and speed up LoRA and QLoRA training, which is what lets a 7B model train on consumer-grade GPUs.

Step 4: Load a base model and attach LoRA

Load the base model in 4-bit and attach a LoRA adapter. Setting load_in_4bit=True is what makes this QLoRA:

import torch
from unsloth import FastLanguageModel, is_bfloat16_supported

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen2.5-7B",
    max_seq_length=2048,
    dtype=None,  # auto-detects fp16 on T4, bf16 on Ampere+
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)

Use unsloth/Qwen2.5-1.5B or a 3B variant if the 7B does not fit your card. r=16 is a sensible default. Raising it does not automatically give better results, it just trains more parameters and eats more memory.

Step 5: Train with TRL's SFTTrainer

Convert your JSONL into the conversational format, then hand it to SFTTrainer from TRL. It applies the model's chat template automatically:

from datasets import load_dataset

raw = load_dataset("json", data_files="dataset.jsonl")["train"]

def to_messages(example):
    return {
        "messages": [
            {"role": "user", "content": example["instruction"]},
            {"role": "assistant", "content": example["output"]},
        ]
    }

ds = raw.map(to_messages, remove_columns=raw.column_names)

from trl import SFTTrainer, SFTConfig

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=ds,
    args=SFTConfig(
        max_seq_length=2048,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        num_train_epochs=3,
        learning_rate=2e-4,
        warmup_ratio=0.1,
        optim="adamw_8bit",
        logging_steps=10,
        output_dir="./outputs",
    ),
)

trainer.train()

The learning rate 2e-4 is about 10x the full fine-tuning rate, which is what LoRA needs since it only updates a small set of parameters. On a Colab T4, a few hundred examples finish in roughly an hour. Watch the training loss; if it stops dropping after a few hundred steps, your dataset is too noisy or too repetitive.

Step 6: Save the adapter and test before you commit

model.save_pretrained("lora_model")

Test the trained adapter on the exact prompts that used to fail:

from unsloth import FastLanguageModel

ft_model, ft_tok = FastLanguageModel.from_pretrained("lora_model")
FastLanguageModel.for_inference(ft_model)

messages = [{"role": "user", "content": "Turn this ID into a tracking link: ORD-1042"}]
inputs = ft_tok.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

outputs = ft_model.generate(**inputs, max_new_tokens=512)
print(ft_tok.decode(outputs[0]))

If the output still ignores your format, add more examples of the failing cases and retrain. Iterating on a few hundred examples is fast and cheap, so do it now instead of after you have built a wrapper around a model that is still wrong.

Step 7: Export to GGUF and serve with Ollama

Export the adapter as a GGUF file and let Ollama run it. GGUF is the format llama.cpp and Ollama understand, so this works on a laptop CPU after training.

model.save_pretrained_gguf("lora_gguf", tokenizer, quantization_method="q8_0")

Unsloth writes a Modelfile into that folder automatically. Then register and run it:

ollama create qwen2.5-finetuned --model lora_gguf/Modelfile
ollama run qwen2.5-finetuned

From here your fine-tuned model is just another model in Ollama. Any app that already talks to your local Ollama endpoint can use it without code changes.

When to fine-tune vs the alternatives

You want to change	Reach for	Why
Tone, schema, domain writing conventions	Fine-tune	Prompts sag; training makes the new behavior the default
Model lacks new or private facts	RAG	Cheaper, updateable, no training run
Minor phrasing tweaks	Prompt engineering	Zero cost, iterate instantly

Next steps

• Start with one narrow domain and a few hundred checked examples before expanding
• Try DPO next for preference pairs if you have human feedback, it steers output without needing exact gold answers
• Benchmark your fine-tuned model against the base model on your own held-out examples, not just on vibe

References

• Unsloth installation documentation
• Hugging Face TRL: SFTTrainer
• Ollama: Importing a Model