Dolphin 3.0 Llama 3.2 3B — Core ML
Built with Llama.
This repository contains Core ML conversions of
dphn/Dolphin3.0-Llama3.2-3B,
plus the tokenizer, generation configuration, reproducible export code, and
Python and Swift generation examples needed to use the recommended model.
Recommended artifact
| Artifact | Weights | Core ML state | Context | Prefill chunk | Minimum OS |
|---|---|---|---|---|---|
Dolphin3.0-Llama3.2-3B-stateful-int4.mlpackage |
INT4, per-block linear | keyCache, valueCache |
2,048 tokens | 512 tokens | iOS 18 / macOS 15 |
The stateful package is the supported interactive-generation artifact. It
keeps the transformer key/value cache inside MLState, so decode submits only
the new token instead of recomputing the full prefix.
The upstream model advertises a 131,072-token context. This Core ML conversion deliberately caps state at 2,048 tokens to bound on-device memory. The prompt prefill chunk is capped at 512 tokens. A caller must reject or deliberately truncate input before either limit is exceeded.
Quick start: Python on Apple silicon
Requirements: macOS 15 or newer, Xcode Command Line Tools, and
uv. The Core ML runtime is available only on
Apple platforms; the export script itself can run on Linux.
uvx --from 'huggingface_hub[cli]' hf download ales27pm/Dolphin3.0-CoreML \
--include 'Dolphin3.0-Llama3.2-3B-stateful-int4.mlpackage/**' \
--include 'config.json' \
--include 'generation_config.json' \
--include 'special_tokens_map.json' \
--include 'tokenizer.json' \
--include 'tokenizer_config.json' \
--include 'scripts/generate.py' \
--include 'examples/swift/DolphinCoreMLCLI/**' \
--local-dir Dolphin3.0-CoreML
cd Dolphin3.0-CoreML
uv run scripts/generate.py \
"Explain why the sky is blue in two sentences." \
--model Dolphin3.0-Llama3.2-3B-stateful-int4.mlpackage \
--tokenizer . \
--max-new-tokens 64
scripts/generate.py applies the bundled Dolphin chat template, creates a new
Core ML state for the conversation, performs a causal prefill, decodes one
token at a time, verifies that logits are finite, and stops on token IDs
128256, 128001, 128008, or 128009.
Quick start: Swift
The buildable example uses a revision-pinned copy of Hugging Face
swift-transformers for tokenization and calls Core ML directly for correct
state and causal-mask handling.
cd examples/swift/DolphinCoreMLCLI
swift run dolphin-coreml \
../../../Dolphin3.0-Llama3.2-3B-stateful-int4.mlpackage \
"Write one friendly sentence about Montreal." \
--tokenizer-folder ../../..
The example targets macOS 15 and iOS 18. It uses greedy decoding so its output is deterministic for a fixed model and prompt. The source is suitable as a small reference integration; production apps should add cancellation, memory pressure handling, sampling controls, and their own safety policy.
Model interface
| Name | Kind | Type | Shape |
|---|---|---|---|
inputIds |
Input | Int32 | [1, 1...512] |
causalMask |
Input | Float16 | [1, 1, 1...512, 1...2048] |
keyCache |
State | Float16 | [28, 1, 8, 2048, 128] |
valueCache |
State | Float16 | [28, 1, 8, 2048, 128] |
logits |
Output | Float16 | [1, query_length, 128258] |
The causalMask final dimension is the absolute end position of the submitted
chunk. For a prompt of N tokens, pass [1, 1, N, N]. For the following
single token, pass [1, 1, 1, N + 1] using the same MLState. Create a fresh
state to start a new conversation.
Core ML stateful models and the attention operations used here require iOS 18 or macOS 15. The model is not compatible with the older iOS 15/macOS 12 claims that appeared in the previous card.
Reproducible export
The conversion pins the source model and tokenizer to:
dphn/Dolphin3.0-Llama3.2-3B@392a6f57223e7ccfe6ef4ebdb2ff101a42d57364
Pinned conversion dependencies are in requirements.txt and in the PEP 723
header of scripts/export_stateful_coreml.py:
uv run scripts/export_stateful_coreml.py \
--model-id dphn/Dolphin3.0-Llama3.2-3B \
--revision 392a6f57223e7ccfe6ef4ebdb2ff101a42d57364 \
--max-context-length 2048 \
--max-query-length 512 \
--quantize int4 \
--output Dolphin3.0-Llama3.2-3B-stateful-int4.mlpackage
The exporter first compares cached single-token decode against a full-prefix PyTorch calculation, traces the stateful model, converts it with Core ML Tools 8.0 for iOS 18, applies symmetric per-block INT4 weight quantization, embeds the source/runtime contract in model metadata, and emits a per-file hash report.
Bundled tokenizer/config file hashes:
| File | SHA-256 |
|---|---|
config.json |
e21ff53ea39726f972362beba869807216775d5e308bc2f531784846c06a0249 |
generation_config.json |
e627b5a8b2dc371f90388947ada64fa6e71de0f991c04c835f0c0bc97e305a4f |
special_tokens_map.json |
2df2c4620bb1a9eb877bc7c90c7fa04608bda9fa7c0cf2cdcc0a17b849649683 |
tokenizer.json |
e40b93124a3e29f62d5f4ff41be56cb2af34ecacf9239acd9da53a98860380b5 |
tokenizer_config.json |
51ad9580aba8d00016efda43357185a0d8ff9884584dcc82ab58ca552afd14e1 |
The recommended package totals 1,808,547,525 bytes. Its release inventory is:
| Package path | Bytes | SHA-256 |
|---|---|---|
Data/com.apple.CoreML/model.mlmodel |
809,496 | a34a00a253c98153cf3b231105493edddde532086e224443e40b255b0f10a924 |
Data/com.apple.CoreML/weights/weight.bin |
1,807,737,412 | 6240edc377b1a0158812454c4bb6e3053d8e8a75a7eedb751b9740fffdfd3e15 |
Manifest.json |
617 | 5b8ac347a822f02ba3a6d9ccff60dd723f2649424c8e88570961f12b1c59afb6 |
coreml_artifacts.json is the machine-readable inventory for the recommended
package and all three legacy packages.
Validation
Run the repository, model-schema, hash, Swift interface, and Apple compiler checks with:
uv run scripts/validate_release.py \
Dolphin3.0-Llama3.2-3B-stateful-int4.mlpackage \
--compile
swift build --package-path examples/swift/DolphinCoreMLCLI
validation/ contains the machine-readable export and release reports. The
release report independently matches every package file against the export
report and artifact manifest, checks specification version 9, both state
shapes, dynamic input ranges, FP16 logits, and 197 blockwise-quantized weight
operations, then passes coremlcompiler metadata, Swift generation, and macOS
15 compilation. A compiler pass proves package and deployment-target
compatibility; it does not substitute for generation on the target device.
Before shipping in an app, measure cold/warm load, peak resident memory,
prefill latency, tokens/second, thermal behaviour, cancellation, and
memory-pressure recovery on each supported device class.
validation/tiny-stateful-runtime-smoke.json records an actual Core ML
prefill/decode test of the cache and dynamic-position contract with a synthetic
two-layer model. It is deliberately not presented as 3B model-quality or
target-device performance evidence.
Legacy artifacts
The original FP16, INT8, and INT4-LUT packages remain available for reproducibility, but are not the recommended chat runtime:
| Artifact | Context per call | Core ML state | Status |
|---|---|---|---|
Dolphin3.0-Llama3.2-3B-fp16.mlpackage |
256 | None | Legacy, stateless |
Dolphin3.0-Llama3.2-3B-int8.mlpackage |
256 | None | Legacy, stateless |
Dolphin3.0-Llama3.2-3B-int4-lut.mlpackage |
256 | None | Legacy, stateless |
All three legacy packages require iOS 18/macOS 15 and expose input_ids,
attention_mask, cache_position, and logits. They require full-prefix
recomputation for every generated token. They are retained as conversion
baselines, not advertised as interactive 131K-context models.
Intended use and limitations
The upstream model is a Dolphin instruction-tuned Llama 3.2 3B model. It can hallucinate, generate biased or unsafe content, and produce incorrect factual claims. It should not make unsupervised medical, legal, financial, safety-critical, or high-impact decisions. Dolphin is intentionally less guardrailed than many assistant models, so downstream applications are responsible for appropriate policy enforcement and evaluation.
INT4 compression can change output quality. Compare the stateful INT4 model
against the pinned PyTorch source on representative application prompts before
deployment. This repository does not claim parity beyond the checks recorded
under validation/.
License and attribution
This conversion inherits the Llama 3.2 Community License. See LICENSE,
USE_POLICY.md, and NOTICE. Redistribution and use must comply with those
terms, including the required attribution. Review the upstream Dolphin model
card for any additional dataset or fine-tune considerations.
Relevant primary references:
- Downloads last month
- 480