Hugging Face's logo

Oysiyl
/
flux-control-lora-brightness-10k

Text-to-Image
Diffusers
flux
flux-diffusers
lora
control
controlnet
brightness
grayscale
template:sd-lora
Model card Files
xet
Community

FLUX.1 Control LoRA - Brightness Control (10k @ 512×512)

A Control LoRA model trained on FLUX.1-dev to control image generation through brightness/grayscale information. This model uses LoRA (Low-Rank Adaptation) for efficient control, providing an ultra-lightweight alternative to ControlNet with excellent pattern preservation and minimal overhead.

Model Description

This Control LoRA enables brightness-based conditioning for FLUX.1 image generation. By providing a grayscale image as input, you can control the brightness distribution and lighting structure while maintaining creative freedom through text prompts.

Key Features:

  • 🎨 Excellent brightness and pattern control across all scales (0.25-1.5)
  • 🚀 188x smaller than SDXL ControlNet: ~25MB vs ~4.7GB
  • ⚡ Ultra-fast loading: LoRA weights load in <1 second
  • 💡 Flexible scale control: Adjustable adapter weights from 0.25 to 1.5+
  • 🔄 Compatible with any FLUX.1-dev setup: Just load the LoRA weights
  • 📦 Minimal storage: All checkpoints + final model = ~125MB total

Intended Uses:

  • Artistic QR code generation (scale 0.7-1.0 recommended)
  • Image recoloring and colorization
  • Lighting control in text-to-image generation
  • Brightness-based pattern integration
  • Watermark and subtle pattern embedding
  • Photo enhancement and stylization

Training Details

Training Data

Trained on 10,000 samples from latentcat/grayscale_image_aesthetic_3M:

  • High-quality aesthetic images
  • Paired with grayscale/brightness versions
  • Resolution: 512×512 (balanced for FLUX.1)

Training Configuration

Parameter Value
Base Model black-forest-labs/FLUX.1-dev
Architecture Control LoRA (~7M trainable parameters)
LoRA Rank 128
Training Resolution 512×512
Training Steps 2,500 (1 epoch)
Batch Size 1 per device
Gradient Accumulation 4 (effective batch: 4)
Learning Rate 1e-4
Empty Prompts 50% (enables better control conditioning)
Mixed Precision BF16
Hardware NVIDIA H100 80GB
Training Time ~1.5 hours
Final Loss 0.435

Model Size Comparison

Model Parameters Size Training Resolution
This Control LoRA ~7M ~25MB 10k @ 512 512×512
ControlNet (SDXL) ~700M 4.7GB 100k @ 512 512×512
T2I Adapter (SDXL) ~77M 302MB 100k @ 1024 1024×1024

Usage

Installation 

pip install diffusers transformers accelerate torch sentencepiece peft

Basic Usage 

from diffusers import FluxControlPipeline
import torch
from PIL import Image

# Load FLUX.1 Control pipeline
pipe = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16
)

# Load Control LoRA weights
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k")
pipe.to("cuda")

# Set adapter weight (scale) - controls conditioning strength
pipe.set_adapters(["default_0"], adapter_weights=[0.75])

# Load grayscale/brightness control image
control_image = Image.open("path/to/grayscale_image.png")
control_image = control_image.resize((512, 512))

# Generate image
prompt = "a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors"

image = pipe(
    prompt=prompt,
    control_image=control_image,
    num_inference_steps=28,
    guidance_scale=3.5,
    height=512,
    width=512,
).images[0]

image.save("output.png")

Adjusting Control Strength

The adapter weight controls how strongly the brightness map influences generation:

# Subtle control (scale 0.25-0.5)
pipe.set_adapters(["default_0"], adapter_weights=[0.25])

# Balanced control (scale 0.7-1.0) - Recommended for artistic QR codes
pipe.set_adapters(["default_0"], adapter_weights=[0.75])

# Strong control (scale 1.0-1.5)
pipe.set_adapters(["default_0"], adapter_weights=[1.25])

Artistic QR Code Generation 

import qrcode
from PIL import Image

# Generate QR code
qr = qrcode.QRCode(
    version=1,
    error_correction=qrcode.constants.ERROR_CORRECT_H,
    box_size=10,
    border=4
)
qr.add_data("https://your-url.com")
qr.make(fit=True)

qr_image = qr.make_image(fill_color="black", back_color="white")
qr_image = qr_image.resize((512, 512), Image.LANCZOS).convert("RGB")

# Load pipeline and LoRA
pipe = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16
)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k")
pipe.to("cuda")

# Set scale for good QR code readability (0.7-1.0 works best)
pipe.set_adapters(["default_0"], adapter_weights=[0.75])

# Generate artistic QR code
image = pipe(
    prompt="a beautiful garden with colorful flowers and butterflies, highly detailed, professional photography",
    control_image=qr_image,
    num_inference_steps=28,
    guidance_scale=3.5,
    height=512,
    width=512,
).images[0]

image.save("artistic_qr.png")

Using Different Checkpoints

The model includes intermediate checkpoints from throughout training:

# Early checkpoint (25% - 2,500 samples)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k", revision="checkpoint-625")

# Mid checkpoint (50% - 5,000 samples)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k", revision="checkpoint-1250")

# Late checkpoint (75% - 7,500 samples)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k", revision="checkpoint-1875")

# Final checkpoint (100% - 10,000 samples)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k", revision="checkpoint-2500")

# Final model (same as checkpoint-2500, on main branch)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k")

Adapter Weight (Scale) Guide

The adapter weight controls how strongly the brightness map influences generation:

Recommended Scale Ranges

Scale Behavior Best For
0.25-0.5 Subtle artistic integration with hints of pattern Natural images, soft lighting hints
0.5-0.7 Light control - visible structure with strong artistic freedom Artistic images, creative reinterpretation
0.7-1.0 🔥 Balanced control Artistic QR codes, watermarks (recommended)
1.0-1.25 Strong control - clear patterns with artistic overlay Geometric patterns, structured designs
1.25-1.5 Maximum control - dominant patterns Strong brightness maps, technical applications

Visual Examples

Checkpoint Progression at Scale 0.75

This image shows how the model improves across checkpoints at scale 0.75 (recommended for QR codes):

Checkpoint Progression at Scale 0.75

From left to right: Original QR input, checkpoint-625 (25%), checkpoint-1250 (50%), checkpoint-1875 (75%), checkpoint-2500 (100%), final model

Final Model - All Scales Comparison

This grid shows the final model's behavior across all scales (0.25 to 1.5):

Final Model - All Scales

Grid showing scales: 0.25, 0.5, 0.7, 0.75, 1.0, 1.25, 1.5 - demonstrating the range from subtle hints to strong pattern preservation

Input QR Code:

QR Code Input

Performance Comparison

vs ControlNet (SDXL)

Metric ControlNet (SDXL) This Control LoRA Advantage
Parameters ~700M ~7M 100x smaller
Model Size 4.7GB 25MB 188x smaller
Load Time ~5-10 seconds <1 second 10x faster loading
Storage (w/ checkpoints) ~18.8GB ~125MB 150x less storage
Pattern Preservation @ Scale 1.0 Excellent Excellent Comparable quality
Flexibility Fixed architecture Adjustable weights More versatile

vs T2I Adapter (SDXL)

Metric T2I Adapter (SDXL) This Control LoRA Advantage
Parameters ~77M ~7M 11x smaller
Model Size 302MB 25MB 12x smaller
Base Model SDXL FLUX.1-dev More advanced base
Training Samples 100k 10k More efficient

Checkpoint Progression Analysis

The model includes checkpoints from throughout training:

  1. checkpoint-625: 25% complete (2,500 samples)
  2. checkpoint-1250: 50% complete (5,000 samples)
  3. checkpoint-1875: 75% complete (7,500 samples)
  4. checkpoint-2500: 100% complete (10,000 samples)
  5. Final model: Same as checkpoint-2500 (main branch)

Key Observations

All checkpoints show consistent, high-quality performance across scales. The progression analysis reveals:

  1. Early Checkpoint (625 steps, 2.5k samples):

    • Good pattern awareness, developing control
    • More artistic interpretation of prompts
    • Recommended scales: 0.5-1.0

  2. Mid Checkpoints (1250-1875 steps, 5k-7.5k samples):

    • Strong balance between control and creativity
    • Stable pattern preservation
    • Recommended scales: 0.7-1.25

  3. Final Model (2500 steps, 10k samples):

    • Maximum control capability
    • Excellent pattern preservation at all scales
    • Recommended scales: 0.7-1.5

No Overfitting Observed

Unlike larger models trained on 100k samples, this 10k Control LoRA shows no signs of overfitting:

  • Consistent improvement throughout training
  • Final checkpoint is recommended for production use
  • 10k samples appears optimal for LoRA-based control training

When to Use This Model

✅ Use This Control LoRA When:

  • Creating artistic QR codes with FLUX.1 quality (scale 0.7-1.0)
  • Need minimal storage overhead (<30MB per checkpoint)
  • Want fast model switching between checkpoints (<1 second)
  • Building production applications requiring small model sizes
  • Working with FLUX.1-dev as base model
  • Require flexible control strength via adapter weights
  • Need multiple checkpoints without massive storage (125MB total vs 18.8GB)

⚠️ Consider Alternatives When:

  • Need higher resolution (>512×512) - consider SDXL T2I Adapter @ 1024×1024
  • Require extremely precise control at very low scales (<0.25)
  • Working with existing SDXL pipelines (use T2I Adapter or ControlNet)

Limitations

Current Limitations

  • Resolution: Trained at 512×512, may not perform optimally at higher resolutions
  • Base model dependency: Requires FLUX.1-dev base model
  • Grayscale conditioning only: Trained specifically for brightness/grayscale control
  • Single control type: Only brightness, not other conditioning types (pose, depth, etc.)

Recommendations

  • For 512×512 generation, use this Control LoRA
  • For 1024×1024+ generation, consider the SDXL T2I Adapter models
  • Experiment with scales 0.7-1.0 for most use cases
  • Use earlier checkpoints (625-1250) if you prefer more artistic freedom

Training Script 

accelerate launch --mixed_precision="bf16" train_control_lora_flux.py \
  --pretrained_model_name_or_path="black-forest-labs/FLUX.1-dev" \
  --dataset_name="<path_to_10k_dataset>" \
  --conditioning_image_column="conditioning_image" \
  --image_column="image" \
  --caption_column="text" \
  --output_dir="./flux-control-lora-brightness-10k" \
  --mixed_precision="bf16" \
  --resolution=512 \
  --learning_rate=1e-4 \
  --proportion_empty_prompts=0.5 \
  --rank=128 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --num_train_epochs=1 \
  --checkpointing_steps=625 \
  --validation_steps=625 \
  --gradient_checkpointing \
  --report_to="wandb"

Available Branches

  • main: Final model (10,000 samples, recommended)
  • checkpoint-625: Early checkpoint (2,500 samples)
  • checkpoint-1250: Mid checkpoint (5,000 samples)
  • checkpoint-1875: Late checkpoint (7,500 samples)
  • checkpoint-2500: Final checkpoint (same as main)

Citation 

@misc{flux-control-lora-brightness-10k,
  author = {Oysiyl},
  title = {FLUX.1 Control LoRA - Brightness Control (10k @ 512×512)},
  year = {2025},
  publisher = {HuggingFace},
  journal = {HuggingFace Model Hub},
  howpublished = {\url{https://huggingface.co/Oysiyl/flux-control-lora-brightness-10k}}
}

Acknowledgments

License

Apache 2.0 License. The base FLUX.1-dev model has separate license terms at black-forest-labs/FLUX.1-dev.

Downloads last month
14
Inference Providers NEW
Text-to-Image
Examples

Model tree for Oysiyl/flux-control-lora-brightness-10k

Base model

black-forest-labs/FLUX.1-dev
Adapter
(36529)
this model