10 Public Datasets for Fine-Tuning Instruction-Following Models

Most instruction-tuned models fail in dull, fixable ways. They ramble when they should answer, ignore a requested format, or bury the useful sentence in paragraph four. A few thousand well-chosen examples usually fixes more of that than another round of compute. The hard part is choosing the data.

What follows are ten public instruction datasets worth knowing: what is inside them, how they are licensed, what format they ship in, and the job each one does best.

Check the license before you check the data

Two datasets with identical contents can be completely different decisions depending on their terms. Run this checklist before you download anything.

• Provenance. Human-written, template-expanded, or generated by another model. Mixed is fine; knowing which is which is not optional, because synthetic sets carry the quirks of whatever produced them.
• License. Permissive (Apache 2.0, MIT) versus non-commercial or research-only (CC BY-NC, CC BY-NC-SA). Non-commercial sets are fine for experiments and papers, not for shipped products.
• Format. Single-turn triplets (instruction, input, output) or chat transcripts with role labels. Converting between them is a short script; losing multi-turn context or system prompts is not recoverable.
• Language coverage. Most of the popular sets are English-only or English-dominant.
• Overlap with your evals. Several of these draw on benchmarks you may also be testing on.
• Duplicates. Near-duplicate prompts teach repetition more than reasoning.

Broad mixtures that teach format following

FLAN Collection

A large aggregation of task templates: on the order of 1,800 tasks and tens of millions of examples, spanning classification, question answering, summarization and simple reasoning. The collection's repository is Apache 2.0, but the underlying sources keep their own licenses, so audit subset by subset. Use it when you want broad instruction-following behavior and generalization across task types.

Super-NaturalInstructions

More than 1,600 tasks, each with a written definition, positive examples and negative examples, adding up to several million instances. Apache 2.0. The task definitions are the interesting part: they give you a ready-made instruction schema, which makes this set strong for zero-shot generalization.

P3 (Public Pool of Prompts)

Dozens of source datasets rewritten into thousands of prompt templates, often with several phrasings per example. Apache 2.0. If your model handles one phrasing well and falls apart on another, this is the set that teaches prompt robustness.

Human-written sets that punch above their size

Dolly 15k

Roughly 15,000 records, English, written by people, released under CC BY-SA 3.0. Entries are short and cover categories such as summarization, classification, closed-book QA and brainstorming. The brevity is a feature: it is easy to inspect, easy to dedupe, and a solid basis for small supervised fine-tuning runs or a hand-checked evaluation slice.

OpenAssistant Conversations (OASST1)

About 10,000 conversation trees in roughly 35 languages, written and ranked by volunteers. Apache 2.0. Because the trees carry rankings, you get multi-turn dialogue and preference pairs from one download. Best used for conversational behavior and response ranking.

No Robots

10,000 English instructions written by paid annotators, with a deliberately open-ended flavor: advice, explanations, creative prompts. Licensed CC BY-NC-SA, so non-commercial. Reach for it when you want tone and helpfulness rather than task coverage.

LIMA

Around 1,000 carefully curated examples, assembled to test how far a very small set can steer behavior. The release is aimed at research use, so read the terms before anything commercial. Even if you never train on it, it is a useful reference for what "high quality" looks like at the paragraph level.

Large generated corpora

Alpaca 52k

52,000 single-turn English instructions, generated and lightly filtered, under CC BY-NC 4.0. Cheap, broad and dated — expect factual errors and repetitive phrasing. Fine as filler in a mixture, poor as your only source.

UltraChat

Roughly 1.5 million multi-turn dialogues, plus a widely used 200k subset under MIT. This is the set that teaches a model to survive vague user turns, ask nothing and still produce something useful. Watch for canned assistant openings; filter them out.

OpenHermes 2.5

Around a million examples assembled from many sources, mostly generated, deduplicated and cleaned. License terms have shifted between releases, so read the current dataset card. It is a practical catch-all when you need volume and diversity in one file.

Domain sets that shift behavior quickly

General instruction data teaches compliance; domain data teaches competence. A few hundred thousand worked examples in one narrow area often changes outputs more than another million generic ones.

• Math: MetaMathQA and MathInstruct pair problems with step-by-step solutions.
• Code: CodeAlpaca and the Evol-Instruct-Code style sets cover short programming tasks and edits.
• Long-form writing: LongForm-style collections target outputs longer than a few hundred words, where generic sets fall short.
• Preference pairs: Anthropic HH-RLHF provides chosen and rejected responses, useful if you are doing preference optimization rather than plain fine-tuning.

A mixing recipe you can run this week

Start with a target: a model that follows formats and stays on task. Then work through this order.

1. Pick a small core. Between 5,000 and 20,000 curated examples. Below roughly 50,000, quality dominates quantity; do not start with a million-row file because it is available.
2. Blend roughly 60/20/10/10. Broad mixture for format following, domain data for skill, multi-turn dialogues for conversation, and a slice of refusals, safety and strict-format examples.
3. Normalize the schema early. Convert everything to one shape — either instruction/input/output or a messages list with roles — and keep the conversion script in version control.
4. Carve out two holdouts. Two hundred to five hundred examples you never train on, plus a small set drawn from benchmarks you care about, to catch contamination.
5. Dedupe. Exact matches first, then near-duplicates by normalized text or embedding similarity.
6. Train two to three epochs, checkpoint every epoch, and evaluate each checkpoint. Instruction models degrade quickly with overtraining; the best checkpoint is often not the last one.

One non-commercial subset can restrict an entire blend. Keep a spreadsheet listing every source, its license and its date, and re-check the cards before you ship.

Licensing terms for public datasets change, and

Photo: panumas nikhomkhai / Pexels