Instructions to use thinkingmachines/Inkling-2-Mini with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use thinkingmachines/Inkling-2-Mini with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="thinkingmachines/Inkling-2-Mini") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("thinkingmachines/Inkling-2-Mini") model = AutoModelForMultimodalLM.from_pretrained("thinkingmachines/Inkling-2-Mini", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Inference
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use thinkingmachines/Inkling-2-Mini with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "thinkingmachines/Inkling-2-Mini" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingmachines/Inkling-2-Mini", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/thinkingmachines/Inkling-2-Mini
- SGLang
How to use thinkingmachines/Inkling-2-Mini with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "thinkingmachines/Inkling-2-Mini" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingmachines/Inkling-2-Mini", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "thinkingmachines/Inkling-2-Mini" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thinkingmachines/Inkling-2-Mini", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use thinkingmachines/Inkling-2-Mini with Docker Model Runner:
docker model run hf.co/thinkingmachines/Inkling-2-Mini
stuck repeating "actually let me act"
Using pi harness with 256k context, single stream, running SGLang it degrades over time (after few compactions) to constant repetition within CoT. In my case it ends up infinitely repeating
Let me actually check the actual ACT β the actual issue β actually the actual issue β actually the ACT β actually ACT
β "ACT" OK actually let me ACT β actually the actual issue β actually the actual ACT β actually the ACT β ACT
βActually the ACTACT ACTACT ACT ACTACT ACTACT ACT the ACTACT ACT ACT ACT ACT ACT ACT ACT ACT ACT ACTACT ACT ACTACT
ACTACT ACTACT ACT ACT ACT ACT"ACT"OK actually the ACT β ACT actually ACTACT ACT ACT ACT ACT ACT ACT ACTACT ACT ACT ACT
ACT ACT
increase temp or rep penalty π€
Actually act ACT ACT ACT ACT β actually π€βοΈ
idk how accurate these settings are but https://unsloth.ai/docs/models/inkling-2-mini#recommended-settings
I've also seen it get stuck looping in the thinking blocks running the original weights, but with much a much longer repeated segment than yours.
Like:
OK β I'll extend the box left to x=0 (it's already at x=0). The 'p' descender β looking at the image: the 'p' β the descender extends to the LEFT of the box β the orange border line at x=0 β the 'p' descender β looking at the image: the 'p' β the descender extends to x=0 β at the very edge of the box!
Hmm β wait β actually looking at the image: the orange box β the left edge β the 'p' descender β looking at the image: the 'p' β the descender extends to the LEFT of the box β the orange border line at x=0 β the 'p' descender β looking at the image: the 'p' β the descender extends to x=0 β at the very edge of the box!
OK β I'll extend the box left to x=0 (it's already at x=0). The 'p' descender β looking at the image: the 'p' β the descender extends to the LEFT of the box β the orange border line at x=0 β the 'p' descender β looking at the image: the 'p' β the descender extends to x=0 β at the very edge of the box!
...
So it clearly has a repetition problem. Guess they really meant it when they talked about a looping transformer. π
What quants do you guys