Getting Started (BentoML): Fine-tuning is one of the most effective ways to adapt an LLM for a specific use case. It continues the training process on a pre-trained model using new, task-specific data. This can involve updating the entire model or just specific layers.
A key driver behind fine-tuning is efficiency. Instead of training a model from scratch (which is extremely resource-intensive), it's far easier and more cost-effective to build on top of a base model that has already learned general language patterns from massive datasets. Fine-tuning sharpens those broad capabilities for your particular task.
For example, fine-tuning can significantly improve a model’s:
Setting Up Your Environment Let's get the tools installed. Start with the package and its training extras, which pull in everything the fine-tuning commands need. pip install "mlx-lm[train]" Confirm the install works with a quick generation test against a small model. mlx_lm.generate --model mlx-community/Mistral-7B-Instruct-v0.3-4bit --prompt "Explain LoRA in two sentences." --max-tokens 120 The first run downloads a 4-bit quantized Mistral model from the MLX Community organization on Hugging Face, caches it locally, then streams a response. The mlx-community org hosts thousands of pre-converted models, so you rarely need to convert weights yourself. One constraint worth noting early: MLX fine-tuning requires models in Hugging Face safetensors format. GGUF files, common in other local tools, work for inference but not for training here. Supported architectures include Llama, Mistral, Qwen2, Phi, Gemma, and Mixtral, among others, so most popular open models are available out of the box.
Preparing Your Dataset Now that the environment is ready, the next step is getting your data into a shape the trainer can use. MLX LM reads training data from a folder containing three files: train.jsonl, valid.jsonl, and an optional test.jsonl. Each line holds one JSON example. The training file is required, the validation file lets the trainer report validation loss as it runs, and the test file scores the model after training finishes. Three formats are supported: chat, completions, and text. The chat format is the most robust default. It stores role-tagged messages per line and lets MLX LM apply the model's own chat template, so your data matches how the model was trained to handle conversations. {"messages": [{"role": "user", "content": "What is LoRA?"}, {"role": "assistant", "content": "An efficient way to fine-tune a model."}]} For plain input and output pairs, the completions format is simpler and works well for instruction-style tasks. {"prompt": "Summarize: The market rose sharply today.", "completion": "Markets gained."} {"prompt": "Translate to French: good morning", "completion": "bonjour"} By default, the trainer computes loss over the entire example, meaning the model spends effort learning to reproduce the prompt as well as the answer. Passing --mask-prompt tells it to compute loss on the completion alone, so training focuses on the response you actually care about. This usually produces a model that follows instructions more reliably, and it works with the chat and completions formats. For chat data, the final message in the list is treated as the completion. Keep each example on a single line with no internal line breaks, since the reader treats every line as a separate record. Split your data so that roughly 80 percent lands in train.jsonl and 10 to 20 percent in valid.jsonl. Around 200 to 500 examples is a sensible minimum for changing a model's behavior (far fewer tend to overfit and memorize rather than generalize).
Training Your First LoRA Adapter With your data in place, here's where things get interesting. Rather than updating every weight in the model, Low-Rank Adaptation (LoRA) freezes the original weights and trains small adapter matrices alongside them. This drops memory and storage needs to a fraction of full fine-tuning while keeping most of the quality. The method comes from the LoRA paper by Hu and colleagues. Launch a training run with one command, pointing it at a model and your data folder: mlx_lm.lora --model mlx-community/Mistral-7B-Instruct-v0.3-4bit --train --data ./data --iters 600 --batch-size 1 As it runs, MLX LM prints training loss, validation loss, tokens processed, and iterations per second. Adapter weights save to an adapters folder by default. Key flags worth knowing: --fine-tune-type accepts lora (the default), dora, or full; --num-layers sets how many transformer layers receive adapters (default: 16); and --iters controls training length. The example sets --batch-size 1 on purpose to keep memory use as low as possible. This prevents crashes on 16 GB machines. If you have 64 GB or more, raising it to 2 or 4 shortens total training time. When memory is tight but you want the smoothing effect of a larger batch, --grad-accumulation-steps raises the effective batch size without raising memory use. If you prefer live graphs over terminal output, add --report-to wandb to log metrics to Weights & Biases. If you hit memory pressure, lower --num-layers to 8 or 4, or add --grad-checkpoint to trade computation for lower memory. These two flags are usually enough to fit a job that would otherwise run out of room.
Choosing a Base Model and Adapter Settings Building on the training mechanics above, two early decisions shape the rest of your run: which model to start from, and how much of it to adapt. For a first project, an 8B parameter model in 4-bit form is the sweet spot. Once the workflow feels comfortable, you can move up to 13B or 14B models, which need 14 to 18 GB of working memory and sit comfortably on a 32 GB machine. The number of trained layers and the adapter rank together control capacity. More layers and a higher rank give the adapter more room to learn, at the cost of memory and time. A common starting point uses 16 layers with a moderate rank, then adjusts based on whether validation loss is still falling. If training loss drops while validation loss climbs, the adapter is memorizing your examples. Learning rate matters too. Values in the range of 1e-5 to 5e-5 work for most LoRA runs. Too high and training becomes unstable; too low and the model barely moves. Change one setting at a time so you can attribute any improvement to a specific choice.
Reducing Memory Use with Quantization Notice that the base model above already ends in 4bit. Training a LoRA adapter on top of a quantized model is what people call QLoRA, described in the QLoRA paper. Because quantization is built into MLX, the same mlx_lm.lora command trains adapters directly on quantized weights with no extra setup. The payoff is concrete. A 4-bit 7B model cuts weight memory by roughly 3.5 times compared with full precision, bringing a 7B fine-tune comfortably into 8 GB of working memory. On a 16 GB MacBook, that leaves ample headroom for the operating system and your training batch. If you prefer to quantize a full precision model yourself before training, the convert command handles it. mlx_lm.convert --hf-path mistralai/Mistral-7B-Instruct-v0.3 --mlx-path ./mistral-4bit -q This writes a 4-bit version to a local folder that you then pass to --model.
Testing and Generating with Your Adapter With training complete, it's time to see how well the adapter learned. Score it against your held-out test set to get a number you can track across experiments. mlx_lm.lora --model mlx-community/Mistral-7B-Instruct-v0.3-4bit --adapter-path ./adapters --data ./data --test To see the model respond, pass the same adapter path to the generate command. MLX LM loads the base model and applies your adapter on top of it. mlx_lm.generate --model mlx-community/Mistral-7B-Instruct-v0.3-4bit --adapter-path ./adapters --prompt "Summarize: Our quarterly revenue grew twelve percent." Run the same prompt without the adapter to compare. If your dataset matched the target task well, the adapted responses should track your training examples more closely than the base model does.
Fusing and Serving the Model Adapters are convenient during experimentation, but for deployment you often want a single, self-contained model. The fuse command merges the adapter back into the base weights. mlx_lm.fuse --model mlx-community/Mistral-7B-Instruct-v0.3-4bit --adapter-path ./adapters --save-path ./fused-model The fused folder behaves like any other MLX model. You can serve it through an OpenAI-compatible endpoint, which lets existing client code talk to your local model after only a base URL change. mlx_lm.server --model ./fused-model --port 8080 For a graphical alternative, LM Studio runs MLX models with a one-click local server and a chat interface, particularly useful when you want to compare your fine-tuned model against others side by side.
Fine-tuning LLMs doesn’t have to mean building everything from the ground up. Several open-source frameworks are designed to streamline the process. These tools provide out-of-the-box support for training open-weight models on custom datasets. They make it easier to apply modern optimization techniques without having to write complex training code yourself. Many of these frameworks are also built with efficiency in mind, helping users reduce memory usage and speed up training, even on limited hardware.
Last modified 11 September 2026