Skip to content

DALL-E 3: Improving Image Generation with Better Captions

Authors: James Betker, Gabriel Goh, Li Jing, et al. (OpenAI) Published: September 2023 Paper: cdn.openai.com/papers/dall-e-3.pdf

Why This Matters

DALL-E 3 solved the prompt adherence problem that plagued all image generators:

  • 🎯 Actually follows prompts - Generates what you ask for, not what it wants
  • 📝 Renders readable text - First model to consistently put text in images
  • 🔄 Better captions = better models - The key insight was data quality, not architecture
  • 🤖 ChatGPT integration - Natural language prompting, no prompt engineering
  • 🛡️ Safety by design - Built-in content filtering and artist opt-outs

Real-world impact: - Eliminated prompt engineering for image generation - Made AI art accessible to non-technical users via ChatGPT - Set new standard for text-image alignment - Forced competitors (Midjourney, Stable Diffusion) to improve prompt following

The insight: The bottleneck in image generation isn't the model - it's the training data captions. Replace noisy, short alt-text with detailed, accurate descriptions, and image generation dramatically improves.

The Core Problem

Why Previous Models Ignored Your Prompt

The training data problem:

Typical internet image-text pair:
  Image: [Complex photo of a sunset over mountains with a lake]
  Caption: "beautiful view"

What the model learns:
  "beautiful view" → vaguely pretty landscape
  Model never learns to distinguish specific details

Better caption:
  "A vibrant orange and purple sunset over snow-capped mountains,
   reflected in a calm alpine lake in the foreground, with pine
   trees framing both sides"

What the model learns:
  Specific colors, objects, composition, relationships

Previous models (DALL-E 2, Stable Diffusion, Midjourney): 

User: "A red cube on top of a blue sphere"
Model: [Random arrangement of colored shapes]
       Often wrong colors, wrong spatial relationships

Why? Training captions never described spatial relationships precisely

The Solution: Better Captions

Re-Captioning the Training Data

DALL-E 3's key innovation:

Step 1: Take existing image-text dataset
  Image: [Photo of a cat sitting on a bookshelf]
  Original caption: "my cat" (useless)

Step 2: Use a vision model to generate detailed caption
  New caption: "A fluffy orange tabby cat sitting on the
  second shelf of a wooden bookcase, looking directly at
  the camera with green eyes. Behind the cat are several
  hardcover books in various colors. The lighting is warm
  and natural, coming from a window to the left."

Step 3: Train the image generator on the new captions
  Model learns fine-grained visual-text correspondence

The Caption Generation Pipeline

1. Train an image captioner
   - Vision model that produces highly detailed descriptions
   - Describes: objects, attributes, spatial relations, style
   - Much more detailed than BLIP/CLIP captions

2. Re-caption the entire training dataset
   - Millions of images get new, detailed descriptions
   - Short captions → 100+ word descriptions
   - Capture relationships, counts, colors, positions

3. Mix caption styles during training
   - 95% synthetic detailed captions
   - 5% original short captions
   - Prevents model from requiring long prompts

Why This Works So Well

Before (short captions):
  "a dog" → Model generates generic dog
  No info about breed, color, pose, background

After (detailed captions):
  "A golden retriever puppy sitting on a red cushion,
   looking up with tongue out, in a sunlit living room
   with hardwood floors"
  → Model learns each element independently
  → Can compose novel combinations at inference

Text Rendering

The Text Breakthrough

Before DALL-E 3: AI couldn't put readable text in images DALL-E 3: First model to consistently render legible text

User: "A storefront sign that says 'OPEN'"

Previous models: "OEPN" or garbled characters
DALL-E 3: Clean, readable "OPEN" sign

How it works: 

1. Detailed captions include text content
   Caption: "...with a sign reading 'Fresh Coffee $3.99'..."

2. Model learns character-level correspondence
   Associates specific character patterns with visual glyphs

3. Not perfect for all text
   Works well: Short phrases, signs, labels
   Struggles: Long paragraphs, small text, handwriting

ChatGPT Integration

Natural Language Image Generation

The killer feature: No prompt engineering needed

Traditional image generation:
  User must craft specific prompts:
  "masterpiece, best quality, highly detailed, 8k, photorealistic,
   cinematic lighting, a cat sitting on a bookshelf, bokeh background"

DALL-E 3 via ChatGPT:
  User: "Draw me a cat on a bookshelf"
  ChatGPT: Automatically expands into detailed prompt
  DALL-E 3: Generates exactly what you meant

How the pipeline works: 

1. User describes what they want (natural language)
2. ChatGPT interprets intent and generates detailed prompt
3. DALL-E 3 generates image from detailed prompt
4. User can refine: "Make the cat orange" / "Add more books"
5. ChatGPT modifies prompt, regenerates

This eliminated the entire field of "prompt engineering" for images.

Performance

Prompt Adherence

T2I-CompBench (Compositional Generation):

Model Color Shape Texture Spatial Overall
Stable Diffusion 2.1 0.50 0.42 0.49 0.13 0.39
SDXL 0.55 0.47 0.56 0.20 0.45
DALL-E 3 0.81 0.68 0.75 0.42 0.67

Massive improvement across all compositional metrics.

Human Evaluation

Prompt following (human judges):
  DALL-E 2: 42% preferred
  DALL-E 3: 71% preferred (vs DALL-E 2)

Text rendering accuracy:
  DALL-E 2: ~10% correct text
  Stable Diffusion XL: ~15% correct text
  DALL-E 3: ~60-70% correct text

Comparison with Competitors (2023-2024)

Aspect DALL-E 3 Midjourney v5 SDXL
Prompt adherence Best Good Moderate
Text rendering Best Poor Poor
Aesthetic quality Good Best Good
Customization Limited Moderate Best (open)
Speed Moderate Fast Varies
Cost ChatGPT Plus Subscription Free (open)

Safety and Ethics

Built-in Protections

DALL-E 3 safety measures:
1. Content filters (violence, explicit, etc.)
2. No generating real people's likenesses
3. Artist style opt-out (artists can request exclusion)
4. C2PA metadata (marks images as AI-generated)
5. Classifier-based blocking of harmful prompts

Artist Opt-Out

DALL-E 3 introduced opt-out for artists:
- Artists can request their style not be replicated
- Blocks prompts like "in the style of [artist name]"
- First major model to offer this

Controversy:
- Some praised the respect for artists
- Others noted it was insufficient
- Doesn't address broader copyright questions

Technical Details

Architecture

DALL-E 3 uses:
- Diffusion model backbone (likely DiT-based)
- CLIP text encoder (for text conditioning)
- T5-XXL text encoder (for detailed text understanding)
- Dual text encoding for better prompt following

Two text encoders:
  CLIP: Good at visual concepts, trained on image-text pairs
  T5: Good at language understanding, handles complex prompts
  Combined: Best of both worlds

Training

Training data:
- Large-scale image dataset (undisclosed size)
- Re-captioned with detailed synthetic descriptions
- Quality filtered
- Safety filtered

Training approach:
- Progressive resolution training
- Classifier-free guidance
- Caption mixing (detailed + short)

Practical Usage

Via ChatGPT

# Simply describe what you want in ChatGPT
"Create an image of a cozy coffee shop on a rainy day,
 with warm lighting coming through steamy windows,
 a few people reading books, and a chalkboard menu
 in the background"

# ChatGPT generates detailed prompt and produces image
# Refine with follow-up messages:
"Make it look more like watercolor painting style"
"Add a cat sleeping on one of the chairs"

Via API

from openai import OpenAI

client = OpenAI()

response = client.images.generate(
    model="dall-e-3",
    prompt="A watercolor painting of a serene Japanese garden "
           "with a red bridge over a koi pond, cherry blossoms "
           "falling, and Mount Fuji in the background",
    size="1024x1024",
    quality="hd",
    n=1
)

image_url = response.data[0].url
print(image_url)

# Get the revised prompt (what DALL-E 3 actually used)
print(response.data[0].revised_prompt)

Limitations

1. No Fine-Tuning

Can't customize for specific styles or subjects
No LoRA, DreamBooth, or similar
Unlike Stable Diffusion (infinitely customizable)

2. Text Still Imperfect

Long text: Often garbled
Multiple text elements: Inconsistent
Small text: Usually unreadable
Non-Latin scripts: Less reliable

3. Restricted Content

Many valid use cases blocked by safety filters
Medical/educational imagery sometimes filtered
Overly cautious on some topics
Less flexible than open-source alternatives

4. Superseded

Newer models (Flux, Stable Diffusion 3, Midjourney v6)
have caught up on prompt adherence
Some surpass DALL-E 3 on aesthetic quality
The "better captions" insight has been widely adopted

Key Takeaways

  1. Data quality > Model architecture - Better captions were the breakthrough, not a better model
  2. Text rendering - First AI to consistently put readable text in images
  3. Prompt adherence solved - Finally generates what you actually ask for
  4. ChatGPT integration - Killed prompt engineering for images
  5. Insight adopted universally - Every image model now uses better captions

Bottom line: DALL-E 3's most important contribution wasn't a new architecture - it was the insight that image generation is bottlenecked by caption quality. By re-captioning training data with detailed descriptions, they solved prompt adherence and text rendering in one stroke. This insight has since been adopted by every competitive image generation model.

Further Reading

Official Resources

  • Paper: https://cdn.openai.com/papers/dall-e-3.pdf
  • Announcement: https://openai.com/index/dall-e-3/
  • DALL-E 2: https://arxiv.org/abs/2204.06125
  • Stable Diffusion (Latent Diffusion): https://arxiv.org/abs/2112.10752
  • CLIP: https://arxiv.org/abs/2103.00020

Published: September 2023 Impact: 🔥🔥🔥🔥 HIGH - Solved prompt adherence, first readable text in images Adoption: Massive via ChatGPT integration Current Relevance: Superseded by newer models, but the re-captioning insight is universal Legacy: Proved data quality is the bottleneck, not model architecture

Modern Status (March 2026): DALL-E 3 has been superseded by newer models (including OpenAI's own updates), but its core insight - that better training captions dramatically improve image generation - has been adopted universally. Every modern image generator now uses synthetic re-captioning.