jdopensource
/

JoyAI-Image-Edit

+<h1 align="center">JoyAI-Image<br><sub><sup>Awakening Spatial Intelligence in Unified Multimodal Understanding and Generation</sup></sub></h1>
+<div align="center">
+[![Report PDF](https://img.shields.io/badge/Report-PDF-red)](https://joyai-image.s3.cn-north-1.jdcloud-oss.com/JoyAI-Image.pdf)
+[![Project](https://img.shields.io/badge/Project-JoyAI--Image-333399)](https://github.com/jd-opensource/JoyAI-Image)
+[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Checkpoint-JoyAI--Image--Edit-yellow)](https://huggingface.co/jdopensource/JoyAI-Image-Edit)&#160;
+[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
+Welcome to the official project page for **JoyAI-Image**.
+</div>
+## 🐶 JoyAI-Image
+JoyAI-Image is a **unified multimodal foundation model** for image understanding, text-to-image generation, and instruction-guided image editing. It combines an 8B Multimodal Large Language Model (MLLM) with a 16B Multimodal Diffusion Transformer (MMDiT). A central principle of JoyAI-Image is the **closed-loop collaboration between understanding, generation, and editing**. Stronger spatial understanding improves grounded generation and contrallable editing through better scene parsing, relational grounding, and instruction decomposition, while generative transformations such as viewpoint changes provide complementary evidence for spatial reasoning.
+![JoyAI-Image Architecture](https://joyai-image.s3.cn-north-1.jdcloud-oss.com/assests/architecture.png)
+## 💎 Highlights
+- **Unified multimodal foundation**: one model family for understanding, generation, and editing through a shared MLLM-MMDiT interface.
+- **Practical data and training recipe**: a scalable pipeline with spatial understanding data ([OpenSpatial](https://github.com/VINHYU/OpenSpatial)), long-text rendering data, editing data, and multi-stage optimization strategies.
+- **Awakened spatial intelligence**: stronger spatial understanding, controllable spatial editing, and novel-view-assisted reasoning through a bidirectional loop between understanding and generation.
+- **Advanced visual generation**: strong long-text typography, layout fidelity, multi-view generation, and controllable editing with better preservation of scene structure.
+## 🔍 Visual Overview
+### Capability Profile
+JoyAI-Image demonstrates broad multimodal performance across understanding, synthesis, and editing, with particular strengths in spatial reasoning, long-text rendering, multi-view generation, and controllable editing.
+![JoyAI-Image Capability Radar](https://joyai-image.s3.cn-north-1.jdcloud-oss.com/assests/capability-radar.png)
+### Advanced Text Rendering Showcase
+JoyAI-Image is optimized for challenging text-heavy scenarios, including multi-panel comics, dense multi-line text, multilingual typography, long-form layouts, real-world scene text, and handwritten styles.
+![JoyAI-Image Text Rendering Showcase](https://joyai-image.s3.cn-north-1.jdcloud-oss.com/assests/text-rendering-showcase.png)
+### Multi-view Generation and Spatial Editing Showcase
+JoyAI-Image showcases a spatially grounded generation and editing pipeline that supports multi-view generation, geometry-aware transformations, camera control, object rotation, and precise location-specific object editing. Across these settings, it preserves scene content, structure, and visual consistency while following viewpoint-sensitive instructions more accurately.
+![JoyAI-Image Multi-view Generation and Spatial Editing Showcase Showcase](https://joyai-image.s3.cn-north-1.jdcloud-oss.com/assests/spatial-editing-showcase.png)
+### Spatial Editing for Spatial Reasoning Showcase
+JoyAI-Image poses high-fidelity spatial editing, serving as a powerful catalyst for enhancing spatial reasoning. Compared with Qwen-Image-Edit and Nano Banana Pro, JoyAI-Image-Edit synthesizes the most diagnostic viewpoints by faithfully executing camera motions. These high-fidelity novel views effectively disambiguate complex spatial relations, providing clearer visual evidence for downstream reasoning.
+![JoyAI-Image Spatial Editing for Spatial Reasoning Showcase](https://joyai-image.s3.cn-north-1.jdcloud-oss.com/assests/spatial-reasoning-showcase.png)
+## 🚀 Quick Start
+### 1. Environment Setup
+**Requirements**: Python >= 3.10, CUDA-capable GPU
+Create a virtual environment and install:
+```bash
+conda create -n joyai python=3.10 -y
+conda activate joyai
+pip install -e .
+```
+> **Note on Flash Attention**: `flash-attn >= 2.8.0` is listed as a dependency for best performance.
+#### Core Dependencies
+| Package | Version | Purpose |
+|---------|---------|---------|
+| `torch` | >= 2.8 | PyTorch |
+| `transformers` | >= 4.57.0, < 4.58.0 | Text encoder |
+| `diffusers` | >= 0.34.0 | Pipeline utilities |
+| `flash-attn` | >= 2.8.0  | Fast attention kernel |
+### 2. Inference
+#### Image Understanding
+```bash
+python inference_und.py \
+  --ckpt-root /path/to/ckpts_infer \
+  --image "test_images/test_1.jpg,test_images/test3.png" \
+  --prompt "Compare these two images." \
+  --max-new-tokens 1024
+```
+#### Image Editing
+```bash
+python inference.py \
+  --ckpt-root /path/to/ckpts_infer \
+  --prompt "Turn the plate blue" \
+  --image test_images/test_1.jpg \
+  --output outputs/result.png \
+  --seed 123 \
+  --steps 30 \
+  --guidance-scale 5.0 \
+  --basesize 1024
+```
+---
+### CLI Reference (`inference.py`)
+| Argument | Type | Default | Description |
+|----------|------|---------|-------------|
+| `--ckpt-root` | str | *required* | Checkpoint root |
+| `--prompt` | str | *required* | Edit instruction or T2I prompt |
+| `--image` | str | None | Input image path (required for editing, omit for T2I) |
+| `--output` | str | `example.png` | Output image path |
+| `--steps` | int | 50 | Denoising steps |
+| `--guidance-scale` | float | 5.0 | Classifier-free guidance scale |
+| `--seed` | int | 42 | Random seed for reproducibility |
+| `--neg-prompt` | str | `""` | Negative prompt |
+| `--basesize` | int | 1024 | Bucket base size for input image resizing (256/512/768/1024) |
+| `--config` | str | auto | Config path; defaults to `<ckpt-root>/infer_config.py` |
+| `--rewrite-prompt` | flag | off | Enable LLM-based prompt rewriting |
+| `--rewrite-model` | str | `gpt-5` | Model name for prompt rewriting |
+| `--hsdp-shard-dim` | int | 1 | FSDP shard dimension for multi-GPU (set to GPU count) |
+### CLI Reference (`inference_und.py`)
+| Argument | Type | Default | Description |
+|----------|------|---------|-------------|
+| `--ckpt-root` | str | *required* | Checkpoint root containing `text_encoder/` |
+| `--image` | str | *required* | Input image path, or comma-separated paths for multiple images |
+| `--prompt` | str | `"Describe this image in detail."` | User question or instruction. When omitted, defaults to image captioning |
+| `--max-new-tokens` | int | 2048 | Maximum number of tokens to generate |
+| `--temperature` | float | 0.7 | Sampling temperature. Use `0` for greedy decoding |
+| `--top-p` | float | 0.8 | Top-p (nucleus) sampling threshold |
+| `--top-k` | int | 50 | Top-k sampling threshold |
+| `--output` | str | None | Optional output file to save the response text |
+## License Agreement
+JoyAI-Image is licensed under Apache 2.0.
+## ☎️  We're Hiring!
+We are actively hiring Research Scientists, Engineers, and Interns to join us in building next-generation generative foundation models and bringing them into real-world applications. If you’re interested, please send your resume to: huanghaoyang.ocean@jd.com