Skip to content

Self-Instruct: Aligning Language Models with Self-Generated Instructions

Authors: Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi (University of Washington, Tehran Polytechnic, Arizona State University, Johns Hopkins University, Allen Institute for AI) Published: December 2022 (ACL 2023) Paper: arxiv.org/abs/2212.10560

Why This Matters

Self-Instruct is the paper that made instruction tuning free. Before it, turning a base model into a helpful assistant required tens of thousands of human-written instruction/response pairs - the expensive, proprietary asset behind InstructGPT. Self-Instruct showed a model can generate that dataset itself, starting from 175 hand-written seed tasks.

  • 52,000 instructions from 175 seeds, generated by the model being improved.
  • About 33 points of absolute improvement over the base GPT-3 on SuperNaturalInstructions, closing most of the gap to InstructGPT-001.
  • Directly enabled Alpaca, which fine-tuned LLaMA 7B on 52K self-instruct-style examples for roughly 500 dollars in API costs and set off the open instruction-tuned model explosion of 2023.
  • Synthetic data is now the default, not the exception. Nearly every open model's post-training mix is mostly model-generated.

The insight: a large pretrained model already contains the knowledge needed to write good instructions and good responses. It just has not been told to behave like an assistant. Bootstrap the behavior out of the model with a small seed set, filter aggressively for quality and diversity, and fine-tune the model on its own output.

The Problem: Instruction Data Was the Moat

By late 2022, the recipe for a useful assistant was clear: pretrain, then fine-tune on instruction/response pairs, then optionally do RLHF. The first and third steps were reproducible in public. The second was not.

Human instruction data has three hard properties: - Expensive. Skilled annotators writing diverse, high-quality tasks and responses, at scale. - Limited in diversity. Annotators converge on the kinds of tasks they find natural. Coverage of long-tail task types is poor. - Proprietary. OpenAI's instruction data was the differentiator, and it was not published.

Existing public collections (FLAN, SuperNaturalInstructions) were built by reformatting existing NLP datasets into instruction format. Broad, but skewed toward classic academic tasks - classification, entailment, extraction - and unlike what real users ask.

The Core Innovation

A four-stage bootstrapping pipeline that runs in a loop:

  Seed pool: 175 human-written tasks (1 instruction + 1 instance each)
        |
        v
  1. INSTRUCTION GENERATION
     Sample 8 instructions from the pool (6 human, 2 model-generated)
     Prompt the model: "come up with a new task instruction"
        |
        v
  2. CLASSIFICATION-TASK IDENTIFICATION
     Ask the model: is this a classification task?
     (determines which instance-generation strategy to use)
        |
        v
  3. INSTANCE GENERATION
     Non-classification -> input-first: generate an input, then the output
     Classification     -> output-first: generate the label FIRST, then an
                           input that fits it (prevents the model from only
                           ever producing the majority class)
        |
        v
  4. FILTERING
     - Discard if ROUGE-L similarity > 0.7 with any existing instruction
     - Discard instructions needing images/audio/real-time data
     - Discard degenerate instances (empty, too long, input == output)
        |
        v
  Add survivors to the pool ---> loop back to step 1

Run this until you have 52,000 instructions covering 82,000 instances. Then fine-tune the base model on them.

Key Components Explained

1. Bootstrapping From Seeds

What it does: Provides format and quality anchors without providing scale. How it works: 175 human tasks are enough to demonstrate what an instruction looks like. Mixing human seeds with previously generated instructions in each prompt keeps output anchored to human quality while allowing drift into new task types.

2. The Diversity Filter

What it does: Prevents mode collapse, which is the failure mode of self-generated data. How it works: A new instruction is only kept if its ROUGE-L overlap with every existing instruction is below 0.7. Without this, the model regenerates variations of the same handful of tasks and the dataset is large but uninformative. This filter is the load-bearing part of the method, and the reason later synthetic-data efforts obsess over diversity metrics.

3. Output-First Generation for Classification

What it does: Fixes label imbalance in generated classification data. How it works: Ask for an input first and the model overwhelmingly writes inputs whose answer is the most typical class. Generating the label first and then an input matching it produces balanced coverage. A small, specific trick with a big effect on data quality - and a reminder that synthetic data generation is a design problem, not a prompting problem.

4. Fine-Tuning on Your Own Output

What it does: Converts latent capability into default behavior. How it works: The base model could already produce assistant-style responses when heavily prompted; fine-tuning on 52K of its own best outputs makes that the unconditioned default. Nothing new is learned about the world - the format and disposition are what change. This is why instruction tuning is cheap relative to pretraining.

Key Results

  • GPT-3 base + Self-Instruct improved by about 33 absolute points on SuperNaturalInstructions over the base model, reaching within a few points of InstructGPT-001.
  • On a human evaluation over 252 user-oriented instructions written for the paper, Self-Instruct-tuned GPT-3 dramatically outperformed the base model and other public instruction-tuned models.
  • Applying the same pipeline to T5-LM and GPT-3 both worked, showing model-independence.
  • Data analysis showed genuine diversity: 52K instructions spanning a wide verb-noun distribution, well beyond the seed set's coverage.

Why This Was Revolutionary

  • Broke the data moat. Within months of publication, anyone with API access could build a competitive instruction dataset. The 2023 open-model boom (Alpaca, Vicuna, WizardLM, Orca, and dozens more) is downstream of this.
  • Legitimized synthetic training data. The intuition that training on model output causes degradation was widespread; this showed that with sufficient filtering, it works well.
  • Introduced distillation-by-API. Alpaca's use of text-davinci-003 outputs to train LLaMA is the template for a practice that is now ubiquitous and legally contested.
  • Reframed instruction tuning as elicitation, not teaching. The capability is in the base model; instruction tuning surfaces it. This framing shaped how the field thinks about post-training.

Real-World Impact

  • Alpaca (Stanford, March 2023) - 52K instructions generated by this recipe, LLaMA 7B fine-tuned for about 500 dollars, behavior qualitatively close to text-davinci-003. It went viral and started the open fine-tuning wave, alongside LoRA making the compute cheap.
  • Evol-Instruct / WizardLM - extends Self-Instruct by iteratively rewriting instructions to be harder (deeper constraints, more reasoning steps), a widely copied technique.
  • Orca and distillation-with-traces - training on teacher reasoning chains, not just final answers.
  • Modern post-training pipelines. The instruction mixes behind Llama 3, Qwen, and most open models are predominantly synthetic, with human data reserved for high-value or safety-critical slices.
  • Constitutional AI applies the same self-generation philosophy to the alignment signal rather than the instruction data, generating AI feedback instead of human preference labels.

Key Takeaways for Practitioners

  1. Diversity filtering matters more than volume. 10K diverse examples beat 100K near-duplicates. Set a similarity threshold and enforce it.
  2. Seed with your actual domain. The seed set determines the distribution. Fifty well-chosen seeds from your real use cases produce far better data than generic ones.
  3. Generate the label first for classification tasks. Otherwise your synthetic dataset will be badly imbalanced.
  4. Filter with everything you have. Similarity, length, format validity, execution (for code), and a judge model. Every filter measurably raises downstream quality.
  5. Check the terms of service. Using a commercial API's outputs to train a competing model is prohibited by most providers' terms. This is a legal question, not a technical one.
  6. Do not expect new knowledge. Self-Instruct changes behavior and format. It cannot teach the model facts it does not have.

Limitations & Future Directions

  • Capped by the teacher. A model cannot bootstrap past its own knowledge; it can only reorganize what it has. Self-generated data amplifies existing biases and errors uniformly.
  • Quality is uneven. The paper's own analysis found a meaningful fraction of generated instances had incorrect or partially correct outputs. It works anyway, which is itself interesting, but it caps the achievable ceiling.
  • Model collapse risk. Training repeatedly on model-generated data across generations degrades diversity - a documented concern for the web-scale data supply as more of the internet becomes model output.
  • Style over substance. Follow-up work ("The False Promise of Imitating Proprietary LLMs," 2023) showed imitation-tuned models match the teacher's style while lagging badly on factuality and reasoning, and that benchmarks emphasizing style overstated the gains. This is an important corrective and does not undo the core result.
  • Superseded in part by RLVR. For domains with verifiable answers, generating data and checking it (RLVR, rejection sampling) beats generating and merely filtering it.

Further Reading

Citation

@inproceedings{wang2023selfinstruct,
  title={Self-Instruct: Aligning Language Models with Self-Generated Instructions},
  author={Wang, Yizhong and Kordi, Yeganeh and Mishra, Swaroop and Liu, Alisa and Smith, Noah A. and Khashabi, Daniel and Hajishirzi, Hannaneh},
  booktitle={Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)},
  year={2023}
}