keras/blip2_opt_2.7b

Model Overview

Model Summary

BLIP-2 (Bootstrapping Language-Image Pre-training) is a generic and efficient pre-training strategy that bridges the modality gap between frozen image encoders and frozen large language models (LLMs). It was introduced by Salesforce in the paper "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models".

The architecture consists of three main components: a frozen image encoder (EVA-CLIP ViT-g/14), a lightweight querying transformer (Q-Former) that acts as an information bottleneck to extract the most relevant visual features, and a frozen LLM (like OPT or Flan-T5) that handles the text generation. Because the heavy vision and language models are kept frozen during pre-training, BLIP-2 achieves state-of-the-art performance on various vision-language tasks with significantly fewer trainable parameters than existing methods.

Key Features:

• State-of-the-art vision-language pre-training method
• Combines frozen EVA-CLIP vision encoder with frozen LLMs (OPT, Flan-T5)
• Highly efficient pre-training via the lightweight Q-Former
• Capable of visual question answering, image captioning, and conversational image understanding

Model Details

• Model Family: BLIP-2
• Architecture: Vision-Language Model (EVA-CLIP ViT + Q-Former + LLM)
• Developer: Salesforce
• Paper: BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Source: Salesforce BLIP-2 Collection
• Task Type: Image-to-Text / Vision-Language Generation
• Max Sequence Length: Varies by LLM (typically 512+ tokens)

Architecture Details (BLIP-2 General)

• Vision Encoder: EVA-CLIP ViT-g/14 (Frozen)
• Vision Encoder Parameters: ~1 Billion
• Q-Former: Querying Transformer (Trainable)
• Q-Former Parameters: ~188 Million
• Language Model: Varies (OPT-2.7B, OPT-6.7B, Flan-T5-XL, Flan-T5-XXL) (Frozen)
• Total Parameters: ~3B to ~12B (depending on the LLM variant)

• [BLIP2 Quickstart Notebook](coming soon..!)
• [BLIP2 API Documentation](coming soon..!)
• BLIP2 Model Card
• BLIP2 Technical Paper
• KerasHub Beginner Guide
• KerasHub Model Publishing Guide

Installation

Keras and KerasHub can be installed with:

pip install -U -q keras-hub
pip install -U -q keras

Jax, TensorFlow, and Torch come preinstalled in Kaggle Notebooks. For instructions on installing them in another environment see the Keras Getting Started page.

Preset Table

Preset	Architecture	Vision Encoder	Language Model	Description
blip2_opt_2.7b	BLIP-2	EVA-CLIP ViT-g/14	OPT-2.7B	BLIP-2 model using OPT-2.7B as the frozen language model.
blip2_opt_6.7b	BLIP-2	EVA-CLIP ViT-g/14	OPT-6.7B	BLIP-2 model using OPT-6.7B as the frozen language model.
blip2_flan_t5_xl	BLIP-2	EVA-CLIP ViT-g/14	Flan-T5-XL	BLIP-2 model using Flan-T5-XL (~3B) as the frozen language model.
blip2_flan_t5_xxl	BLIP-2	EVA-CLIP ViT-g/14	Flan-T5-XXL	BLIP-2 model using Flan-T5-XXL (~11B) as the frozen language model.

Example Usage

BLIP-2 can be used for various vision-language tasks such as image captioning and visual question answering (VQA). Depending on the preset you choose, you will use either BLIP2CausalLM (for OPT models) or BLIP2Seq2SeqLM (for Flan-T5 models).

The BLIP2Seq2SeqLM class supports BLIP-2 variants that use Flan-T5 as their base language model. This architecture requires passing your text prompt to the encoder using the "encoder_text" key.

import keras
import keras_hub
import numpy as np
from PIL import Image
import requests

model = keras_hub.models.BLIP2Seq2SeqLM.from_preset("blip2_opt_2.7b")

image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")
image_array = np.array(image)

vqa_input = {
    "images": image_array,
    "encoder_text": ["Question: what is in the picture? Answer:"]
}
print(model.generate(vqa_input))

caption_input = {
    "images": image_array,
    "encoder_text": ["A picture of"]
}
print(model.generate(caption_input))

Example Usage with Hugging Face URI

BLIP-2 can be used for various vision-language tasks such as image captioning and visual question answering (VQA). Depending on the preset you choose, you will use either BLIP2CausalLM (for OPT models) or BLIP2Seq2SeqLM (for Flan-T5 models).

The BLIP2Seq2SeqLM class supports BLIP-2 variants that use Flan-T5 as their base language model. This architecture requires passing your text prompt to the encoder using the "encoder_text" key.

import keras
import keras_hub
import numpy as np
from PIL import Image
import requests

model = keras_hub.models.BLIP2Seq2SeqLM.from_preset("hf://keras/blip2_opt_2.7b")

image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")
image_array = np.array(image)

vqa_input = {
    "images": image_array,
    "encoder_text": ["Question: what is in the picture? Answer:"]
}
print(model.generate(vqa_input))

caption_input = {
    "images": image_array,
    "encoder_text": ["A picture of"]
}
print(model.generate(caption_input))