dataset.name: MMLU-Pro (TIGER-Lab/MMLU-Pro, MIT licence, public and ungated).
sample_count is provisionally 1 pending the real figure.
Both records now pass llm-generative.schema.yaml.
Still outstanding: runtime.version (the ExecuTorch version). Unlike created_at and sample_count, it forms part of the report identity URN
urn:ai-portal:perf:{runtime.name}:{runtime.version}:{dataset.name}:{dataset.version}:{batch_size}:{target.name}:{profile}
so ingesting without it and adding it later would insert a duplicate report rather than update the existing one. Worth supplying before first ingest.
2026-09-07T11:39:36Z - the time the measurements were sent over (12:39:36 BST).
context.dataset (name + sample_count) is still outstanding and remains the only schema validation error. Note dataset.name forms part of the report identity URN, so it should not be filled with a placeholder: a later correction would insert a duplicate report rather than update the existing one. created_at is not part of that URN, so this value can be corrected in place if the true measurement time differs.
Update readme.md to align naming conventions (#1)
- Update readme.md to align naming conventions (b520f5ff7db26ae63321fca674ffb4d3cbff8e41)
Co-authored-by: Will Lord <willlordarm@users.noreply.huggingface.co>
The baseline is BF16, not FP32. google/gemma-4-E2B-it's safetensors metadata reports {'BF16': 5123178051}: 5.123B parameters x 2 bytes = 10,246.6 MB, which matches the published file size exactly. FP32 would be ~20.5 GB.
The model README's line 'Compared to the FP32 baseline - 10246.6 MB -> 3000 MB, 2.6x smaller' is wrong three ways: the baseline is BF16 not FP32; the optimised artefact is 2564.0 MB not 3000 MB (the .pte is 2,563,959,040 bytes, matching metadata.yaml); and the true reduction is 4.00x, not 2.6x.
Add draft benchmark records from Richard's vivo X300 measurements
Baseline (original) and Arm-Optimized runs, 4 threads (1 Ultra + 3 Perf), 128 input / 128 output tokens, 5 warmups, 20 measured runs.
Draft for review — deliberately incomplete rather than filled with placeholders. Absent because they are not yet known:
- created_at (when the measurement was recorded)
- runtime.version (ExecuTorch version)
- context.dataset (name + sample_count; the schema requires these, so these records will not validate until supplied)
target_ref: vivo-x300 is inferred from the '1 Ultra 3 Perf' core description and needs confirming.
Prefill throughput is omitted: 128 / 49.71 = 2575 ms and 128 / 180.54 = 709 ms, identical to the reported TTFT in both runs, so it is the same measurement and the schema has no field for the rate.
Restores the file removed by the 13:49 sync. The AI Portal needs it to ingest this model; without it the model fails on the next re-ingest. Identical to the version at b840f3a - unchanged filename, size and provenance fields. Worth adding to the local copy the sync mirrors from, so it is not removed again.
Applies the provenance fields from ai-portal-model-schema#27, following the convention used across the Marvik catalogue. vendor is the upstream developer of the source baseline (Google), not the organisation that quantised it. metadata.yaml only.
Adds metadata.yaml so the model can be listed in the AI Portal. Fields are derived from the README frontmatter and the uploaded .pte; model_size_mb is the actual artefact size. No other files touched.
