Holon.liveHOLON OF HOLONS

Pretraining, fine-tuning and SFT: what differs?

English · Compare Chinese

Pretraining learns representations and patterns from broader material. Fine-tuning adjusts an existing model's parameters; supervised fine-tuning uses examples with desired outputs. These differ from supplying a document at use time.

Learning from examples

Examples can demonstrate format, expression or handling. Performance on new cases still needs assessment; similar examples do not establish reliable domain competence.

Where LoRA fits

LoRA is a public parameter-efficient adaptation method using fewer trainable parameters. It is not all fine-tuning and is not universally better.

Fine-tuning and RAG

Changing answer behavior and querying frequently updated facts may be different needs. Fine-tuning changes parameters; RAG supplies external material. Systems may combine them while explaining their separate roles.

Does uploading material train the model? Not necessarily. Determine whether it enters context, a knowledge base or a parameter update.

Continue with What are AI models and large language models? to compare related ideas.

For a related project perspective, explore LingoAI’s AI work.

Sources and further reading