A folder of resolved tickets looks like a promising starting point for a custom AI model. It also contains rushed replies, outdated policies, missing context, and answers that only made sense after a phone call nobody recorded. Training on the whole export would preserve those problems alongside the useful work.
We recommend preparing a small, reviewable collection of examples before committing to a larger fine-tuning effort. The aim is to establish what the model should learn, which records teach it, and how the team will tell whether it improved. Our fine-tuning process overview covers the training stages. This post focuses on the work that should happen before them.
Choose one behavior worth teaching
Start with a task narrow enough that two reviewers can agree on a good answer. Turning an incoming support message into a category and a short internal summary is a clearer target than making an assistant understand the business.
Write down the allowed inputs, required output, and conditions that should stop the task. For a hypothetical equipment supplier, the output might be a service-request category, a summary, and a missing-information field. The model should not invent a serial number or decide that a warranty applies when the request does not establish either.
If the main difficulty is finding the current manual or policy, improve access to that information first. Our RAG and fine-tuning comparison explains that distinction. A changing reference library and a stable response pattern need different treatment.
Build examples from what the model will actually see
Supervised fine-tuning can use prompt-and-answer pairs or conversations. Hugging Face’s training documentation also distinguishes training on assistant responses from training on every message. That makes the boundary between input and desired answer an explicit preparation decision.
For each selected record, reconstruct the information available when the task began. Put the desired result on the answer side. Leave later discoveries out of the input if the deployed system would not have them.
A closed service ticket may include a technician’s final diagnosis. A model categorizing the first customer message cannot depend on that diagnosis. Testing with the completed ticket would make the job artificially easy.
We recommend a simple review sheet with a source identifier, task input, approved answer, reviewer, and reason for inclusion. Keep administrative notes separate from model-visible text. Otherwise a note such as “this is an escalation example” can accidentally give away the intended answer.
Resolve disagreements before adding volume
Have people who understand the work independently review a sample. Compare their answers and record why they differ. One reviewer may treat a missing order number as a reason to ask for details, while another routes the request immediately.
Settle that distinction in the task instructions. Where both responses are acceptable, describe the acceptable range rather than pretending there is one perfect sentence. For classification, give each category a definition and examples of nearby categories it should not absorb.
Keep obsolete responses and unresolved disputes out of the approved training collection. Retain their identifiers and exclusion reasons separately so the next export does not quietly put them back. A disagreement register is useful business documentation even if the project never reaches training.
Split by the underlying case
Separate training, validation, and final test data. Training examples teach the behavior; validation examples help choose among changes; the final test measures the selected version. Duplicate records across those sets can make apparent progress misleading.
For business conversations, we recommend grouping examples by the underlying case before splitting. The first message and a follow-up from the same ticket should stay together. Apply the same rule to rewritten or synthetic variants derived from that case.
Group-aware splitting tools support this approach. The important decision is what constitutes a group in your workflow. A ticket may be enough for one task; another may need an entire project held together because its documents repeat the same distinctive facts.
Save a record of the split. If a later data export cannot be matched back to those groups, it is easy to contaminate the evaluation without noticing.
Keep the final test out of the editing loop
Use validation failures to improve examples and instructions. Reserve the final test for checking the version selected through that process. Repeatedly choosing changes because they improve the same final test gradually turns it into another development set.
We recommend naming a person responsible for that boundary. They should know which examples informed revisions and which remained untouched. If the team studies a final-test failure and writes training examples around it, record that decision and refresh the independent test before making another broad performance claim.
Also keep a visible list of gaps. A test with no unusually long messages cannot establish how the model handles them. A high overall score should not hide an important task type the sample barely covers.
Make the handoff inspectable
Before training, package the approved examples with their review rules, source identifiers, split assignments, and exclusions. Record the dataset version and the intended model task. A colleague should be able to understand what changed between two versions without reopening every original ticket.
Restrict access to the collection and remove details the task does not need. Confirm that the proposed use of the source records is approved within the business before copying them into another system. These are preparation checks, not a claim that removing names solves every privacy concern.
There is no useful universal record count for this decision. Start by checking whether the collection covers the behavior, whether its answers are defensible, and whether separate evaluation is possible. If those foundations are missing, another export will not fix them.
Decide whether training is the next step
Compare a well-instructed base model with the proposed task before spending effort on a fine-tune. If it already produces acceptable results, a simpler integration may be enough. If the errors are consistent and the approved examples clearly teach the missing behavior, a controlled training experiment has a stronger starting point.
Our AI fine-tuning service covers custom models, and the first useful conversation can be about the records rather than the hardware. Tell us what the model should do, what examples are available, and where people still disagree. That is enough to start assessing the fit without assuming a larger training run is the answer.