FluidInference/vela-3.0-coreml

Vela 3.0 (vela3-fast, 0.4B) · Core ML

vllm-sr/vela3-fast (vLLM Semantic Router × KR Labs, Vela 3.0 family) converted to Core ML for Apple silicon: decisions (choice / yes-no over options you name) and text spans (17 types of personal information, words of an answer its context does not support) from one UltraBERT-380M encoder, on device.

Source	vllm-sr/vela3-fast @ 9e338f39
Vela3Decision.mlpackage	decision backbone, fp16, 726 MB: functions L128, L256, L512, L1024 (encoder tokens)
Vela3SpanSets.mlpackage	backbone + the release's span_sets adapter (LoRA merged), fp16, 725 MB: functions L512, L1024
Host side	heads_decision.safetensors, heads_span_sets.safetensors (fp32 readout heads), tokenizer.json, config.json, vela3_config.json (the release's thresholds)
Runtime	vela3_coreml.py (Python: coremltools, tokenizers, torch for the small heads, no PyTorch weights) + upstream/ (the release's own rendering, span decoder and thresholds, unmodified)
Requires	macOS 15+ / iOS 18+

from vela3_coreml import Vela3CoreML
m = Vela3CoreML(".")  # GPU by default
m.decide("My card was charged twice and I need this fixed today.",
         {"intent": {"type": "choice", "instructions": "What does the customer want?",
                     "criteria": {"refund": "Money back for the duplicate charge", "cancel": "Cancel the order"}},
          "urgent": {"type": "noul", "instructions": "The customer needs a fast response."}})
m.pii("Hi, I'm Dana Whitfield (dana@example.com, 312-555-0110).")   # [{label, start, end, text, score}, ...]
m.unsupported(context="Adults may take up to 4 g of paracetamol in 24 hours.",
              answer="Adults may take up to 6 g of paracetamol in 24 hours.")  # -> "6 g"

Fidelity

Against the fp32 PyTorch release on the same inputs (reports/verify.txt):

• Decisions: typed-decisions TEST, all 1,200 choice and yes/no questions: 0 answers differ, max probability difference 0.0026.
• Personal information: 25 varied passages (support tickets, clinic notes, HR reports, bank chats; 183 spans): identical span sets.
• Unsupported claims: identical spans on every context / answer pair tested.

On a 20,000-word synthetic case file with 2,971 planted values, the Core ML model catches 99.5 % (2,956), with 30 other flags.

Speed (MacBook Pro M5 Pro, GPU)

Encoder function	Decision	Span sets
L128	6.7 ms
L256	9.5 ms
L512	17.6 ms	18.7 ms
L1024	37.6 ms	37.5 ms

One encoder pass answers one question. The question (options, or the 17 PII label descriptions, ~250 tokens) counts toward the length, so a PII pass over a paragraph usually lands in L512: about 0.5 ms per word end to end. The Neural Engine is not used: it was slower at every length and drifted further from the fp32 answers (15 / 1,200 decisions flipped).

Scope and limits

• Covered: decisions on the decision backbone, and the two label sets of the span_sets adapter (17 PII types, unsupported). The release's other adapters (open-label NER, relations, evidence, router signals, domain, fact-check, feedback) are not converted; each retrains every layer, so each would be its own encoder package.
• The release's score question type is not part of the decision backbone and is not supported (as upstream).
• Inputs are limited to 1,024 tokens per encoder pass. Longer decision inputs raise Unsupported; longer span inputs are windowed at 1,024 tokens (the release uses 8,192), so long-document spans can differ.

Files and licences

Apache-2.0, as the source release (LICENSE). upstream/ is the release's own code, unmodified. The conversion: a fixed-length export of the encoder (local layers' 128-token sliding window as a band mask, the two RoPE bases precomputed, fp16-safe masks), the adapter's LoRA merged into plain weights, heads kept in fp32 on the host. conversion/ holds the export, packaging and verification scripts.