Spaces:
Running
Running
Commit History
-
Upload from GitHub Actions: Merge pull request #32 from datenlabor-bmz/eval-crash-safe-incremental-runs Running
Verified signatureRecover interrupted eval runs; cap CI to incremental work55e7c3b
-
Upload from GitHub Actions: Merge pull request #31 from datenlabor-bmz/ci-fail-red-on-dry-balance
Verified signatureFail the eval run visibly when the OpenRouter balance gate trips9ea166d
-
Upload from GitHub Actions: blocklist: drop the grace for slow failing models, not just egregious ones
Verified signaturePer feedback: the 2-run grace is only worth it when retrying is cheap. main.py now measures per-model seconds/sample during the run and flags rate-limited models (> AUTO_BLOCKLIST_SLOW_SEC_PER_SAMPLE = 0.3s/sample); update_blocklist_ strikes excludes a failing model after ONE run if it's egregious (>=80%) OR slow. The grace now only protects a fast, moderately-failing model — i.e. retry only when the time investment is reasonable.608b646 -
Upload from GitHub Actions: results: refresh model_health.json snapshot
Verified signaturee94f7b1 -
Upload from GitHub Actions: models: drop kimi-k2.6; exclude egregiously-failing models after one run
Verified signature- Remove moonshotai/kimi-k2.6 from important_models: ~58% errors and so heavily rate-limited that a single model ran >50min (it ate a run's 6h budget). kimi-k2 represents Moonshot. - Tiered strikes: a model failing >= AUTO_BLOCKLIST_IMMEDIATE_FAIL_PCT (80%) is excluded after ONE bad run instead of waiting out the 2-run grace — it's broken, not transiently rate-limited. The grace still protects the borderline 50-80% band where a provider outage is plausible.4c601bb
-
Upload from GitHub Actions: eval: check runtime budget per-batch so a slow model can't blow the 6h cap
Verified signatureThe graceful budget was only checked between models, so a single rate-limited model (kimi-k2.6 ran >50min) blew past 5.5h and the run got hard-cancelled at 6h. Now check the budget (a) before starting each model and (b) after every batch — a slow model is interrupted at a batch boundary, its partial work saved to results-detailed (skipped next run) but NOT published (not coverage-complete). Lower the workflow budget to 5h for headroom. Checkpointing already worked: the cancelled run published 42 clean fully-covered models before the cap.594d28a -
Upload from GitHub Actions: Refresh dashboard UI: design tokens, indigo accent, research footer
Verified signatureIntroduce a CSS design-token layer (palette, type, spacing, shadows) in index.css and apply it across the app: - Hero: system-sans headline and flat indigo primary + outline secondary buttons, replacing the gradient/emoji buttons; quiet GitHub link - Cool off-white background with a single restrained indigo accent on links, primary button, active filter states and the active carousel dot - Harmonized table header tints and softened card/chart-panel chrome; data tables, score bars, medals and world-map encodings unchanged - Refreshed About/Contribute dialogs and added a footer with the paper citation (arXiv:2507.08538) and BMZ/GIZ/DFKI credits - Fix chart carousel overflow so the indicator dots no longer overlap the footer - Drop the external web-font dependency (system fonts only)15a7ddf -
Upload from GitHub Actions: eval: fix per-combo resilience (tqdm_asyncio.gather has no return_exceptions)
Verified signatureThe previous resilience commit passed return_exceptions=True to tqdm_asyncio.gather, which it doesn't accept (TqdmKeyError) — crashing the loop immediately. Wrap each task in a _safe() coroutine that returns non-fatal exceptions as values (combo skipped, re-attempted next run) and re-raises FatalAPIError. Verified locally: normal results pass, non-fatal skipped, fatal propagates.18acfdb -
Upload from GitHub Actions: eval: don't let one bad (task,language) combo crash the whole run
Verified signatureA transient HF dataset download (e.g. arc-easy-autotranslated config 'rn' failing to build on the runner) raised uncaught and killed a multi-hour run. Two layers of resilience: - _load_dataset retries the 'Couldn't find cache for config X' ValueError (flaky build) with backoff, bounded so a genuinely-missing config still surfaces. - the eval loop gathers with return_exceptions and SKIPS a combo that still throws (logged, no row) so it's simply re-attempted next run; FatalAPIError still aborts. Transient failures no longer count against a model's strikes.a92221e -
Upload from GitHub Actions: results: update committed snapshots to current 71-model state
Verified signatureThe repo snapshots were stale (40-model b7b017f restore) while HF/the live dashboard are at 71 fully-covered models. Refresh results.json/models.json/ languages.json/model_health.json so GitHub matches what's published. (Actions runs push to HF but don't commit back, so these are updated by hand.)19fbc15 -
Upload from GitHub Actions: models: migrate catalog to /api/v1/models; enforce privacy per-request
Verified signatureOpenRouter removed the undocumented /api/frontend/models endpoint (404), which crashed load_models at import on every fresh run/backend start. Rebuild the catalog from the official /api/v1/models and normalize it to the shape the rest of the module expects (slug/permaslug/created_at/endpoint.pricing/hf_slug from hugging_face_id), with retry/backoff on the fetch. That catalog no longer exposes per-provider data policy, so privacy moves to a STRONGER per-request guarantee: complete() now sends provider.data_collection= 'deny', so OpenRouter refuses to route any prompt to a training/retaining provider (verified: gpt-5->OpenAI, deepseek->DeepInfra, qwen->Parasail, all training=false). The catalog-level dataPolicy check is kept as a no-op for shape compatibility. Also drop 'openrouter' from the discovery provider allowlist (its namespace is routing meta-models / cloaked test models, not real LLMs).f502bec -
Upload from GitHub Actions: discovery: surface newer flagships from curated families; blocklist: require 2 consecutive bad runs
Verified signaturediscovery: skip a candidate only if we already curate something in its family that is at least as NEW (version-aware), so a future Claude Opus 5 / GPT-6 is still auto-surfaced — the coarse family key alone would have permanently blocked every model from a lab we track. auto-blocklist: replace the single-snapshot threshold with a persisted strike counter (HF dataset 'model-health-strikes'). A model past the failure threshold earns one strike but is NOT excluded; it must stay bad for AUTO_BLOCKLIST_MIN_RUNS (2) consecutive runs before exclusion, and any recovering run resets it. This stops a provider rate-limiting us during one long run from wrongly excluding healthy open models (they get re-attempted next run). load_auto_blocklist is now a cheap read of the strike table (also speeds up backend startup); main.py updates strikes once per run.f28fed1
-
Upload from GitHub Actions: discovery: one flagship per product line; eval: graceful 6h-safe runtime budget
Verified signature- _family_key now strips size-tier/variant suffixes and parameter-size tokens in addition to versions/dates, collapsing size sweeps so auto-discovery keeps a single flagship per product line (highest cost wins). Cuts a recent discovery run from 50 candidates to 14. Specific variants still go in important_models by hand. - main.py honors MAX_RUNTIME_SECONDS: after a model's checkpoint, if the wall- clock budget is exceeded it stops gracefully (exit 0) with remaining models deferred to the next run. The weekly workflow sets 5.5h so a run never hits the GitHub 6h hard-cancel — it exits clean, restarts the Space, and resumes.4047210
-
Upload from GitHub Actions: main: gate publishing on coverage-completeness, not just error rate
Verified signatureThe previous publish gate only filtered high-error models, so a model with sparse-but-successful history (e.g. 5 of ~200 languages) passed and showed an inflated mean — the exact failure that put a small model at rank #1. Now a model is publishable only once it is coverage-complete (zero pending combis: either fully evaluated before this run, or finished in it) AND healthy. Verified against live results-detailed: publishable set drops from 88 (incl. 5-language models) to 19, all with full 221-language coverage.c1041db -
Upload from GitHub Actions: util: retry HF push with backoff; write local snapshot before push
Verified signaturePer-model checkpointing pushes to HF frequently; HF intermittently 429s or drops connections. Retry the push 5x with exponential backoff so a transient failure doesn't kill a multi-hour run, and persist the local JSON before the push so on-disk progress survives a push failure.2a1f0a5
-
Upload from GitHub Actions: main: checkpoint per fully-evaluated model instead of once at the end
Verified signatureThe eval loop previously saved only after ALL batches finished, so a run killed by the GitHub 6h runner cap (or any interruption) persisted nothing while still being billed for the API calls. Now evaluate one model at a time and push to HF after each model completes its entire matrix. Two guarantees: - progress survives interruption (next run skips fully-logged models), so a large onboarding completes across multiple runs; - a model only enters the published aggregate once fully covered AND healthy (publishable_models excludes models past the auto-blocklist failure threshold), so a sparse/broken model can't show an inflated mean. Lifts publishable_models/checkpoint to module level; unit-tested the gate.eaa7534
-
Upload from GitHub Actions: workflow: make huggingface-cli login resilient to transient 429s
Verified signatureThe eval step died when HF's whoami endpoint returned 429 on login. All dataset load/save calls pass the token explicitly, so the CLI login is non-essential — retry it with backoff and never let it abort the run. Also export HF_TOKEN for any internal hub call.b67f6cf -
Upload from GitHub Actions: models: replace claude-opus-4.5/4.6 with 4.8 in curated list
Verified signatureKeep one current Opus flagship lineage: 4.8 (newest) + 4.7. Drop the superseded 4.5/4.6 from important_models.15e8f68 -
Upload from GitHub Actions: discovery: filter out voice/ASR/vision/build endpoints
Verified signatureAuto-discovery was surfacing non-text endpoints (microsoft/mai-voice-2, nvidia/parakeet ASR, glm-4.5v / glm-5v-turbo vision, x-ai/grok-build). Add voice/parakeet/canary/build skip rules and a regex for the bare-"v" vision suffix that the -vision/-vl tags miss.f2add9e
-
Upload from GitHub Actions: restore results snapshots to b7b017f (canonical 40-model state)
Verified signatureThe "restore" I pushed in 5f30ceb was wrong: it re-aggregated the entire results-detailed log over 132 models (not just current cohort), which let sparse-coverage auto-discovered models bubble to the top of the leaderboard with cherry-picked 100% scores. Dashboard rank #1 was google/gemma-4-31b-it (a model with only a handful of evaluated combos), not actual frontier models. Reverting results/{results,models,languages}.json to b7b017f (40 models, 221 langs, 59115 rows — the 2026-03-10 canonical snapshot). Pushed the same files to the HF datasets so the Space backend picks them up. The running eval workflow (26099052615) will replace this with a fresh 99-model snapshot when it completes via the canonical main.py aggregation filter. Until then, the dashboard shows correct historical data instead of broken current data.4e0fc02 -
Upload from GitHub Actions: backend: handle null creation_date in three apply() calls
Verified signatureThe aggregated results may contain rows for models that don't have metadata in models.json (e.g. when results-detailed includes historical models that have since rotated out of important_models). The results+models left-join leaves those rows with NaN/NaT creation_date, which the old "x.isoformat() if x else None" lambdas tripped over because pandas NaT is truthy and a float-NaN doesn't have .isoformat. Replace with "pd.notna(x) and hasattr(x, 'isoformat')" in all three spots: make_model_table, make_language_tier_history, make_license_history. Surfaced after this morning's results-restore widened the aggregate to historical models that aren't in the current cohort; the HF Space returned 500 for every /api/data call ("Network response was not ok" in the frontend).f6a28ed -
Upload from GitHub Actions: preflight: skip eval workflow when OpenRouter balance is too low
Verified signatureIf the OpenRouter account has no usable balance (per-key cap exhausted or account credits depleted), the eval just floods every request with 403s, FatalAPIError aborts, and the workflow goes red. Cron-scheduled runs would do this repeatedly while the operator can't top up. New step: "Pre-flight — check OpenRouter balance" runs first in the workflow. Hits /api/v1/auth/key for per-key limit_remaining; falls back to /api/v1/credits for account-level if no per-key cap is configured. If remaining is below MIN_OPENROUTER_BALANCE_USD (default $1), all subsequent steps are gated off via `if: steps.balance.outputs.proceed == 'true'`, producing a workflow with most steps showing as "skipped" and a clear ::warning:: annotation pointing to https://openrouter.ai/credits. Once you top up, the next scheduled run automatically picks back up without any code change. Threshold can be raised by editing the MIN_OPENROUTER_BALANCE_USD env var on the job (currently $1; bump if you want a wider safety marg…a70e02a -
Upload from GitHub Actions: fast-fail on account-level API errors; refuse to ship runs with >80% errors
Verified signatureToday's workflow run hit a 403 'Key limit exceeded' from OpenRouter on the very first request and would have kept burning through the entire combinatorial matrix tagging every row with status=error. We caught it manually and cancelled before save() fired, but the fragility is real: any future account-level outage (auth, credit cap, key revoke) would flood results-detailed with bogus errors and then auto_blocklist would mass-exclude the auto-discovered models from the *next* run, even after the underlying issue is fixed. Two layers of protection. Layer 1 - propagate account-level failures (evals/models.py + tasks.py): - Add FatalAPIError. Raised by complete() when the OpenRouter error message matches markers like "key limit exceeded", "insufficient credits", "unauthorized", "payment required", "invalid api key". - query() in tasks.py re-raises FatalAPIError instead of swallowing it into status=error. asyncio.gather then surfaces it, the eval loop aborts, and save() …c2afc16 -
Upload from GitHub Actions: unblock workflow: materialize gcloud creds on runner; lazy-init translate client
Verified signatureThe nightly workflow has been silently failing since at least Dec 2025 because the eval pipeline depends on gcloud_credentials.json for the Google Translate baseline but the file is gitignored and there was no GHA secret/step to materialize it on the runner. Also, evals/models.py called translate.Client() at module import time, so any import of the module (including backend.py, smoke tests, any contributor without GCP locally) blew up with DefaultCredentialsError. .github/workflows/nightly-evals.yml - Add a "Write Google Cloud credentials" step that decodes the GCLOUD_CREDENTIALS_JSON secret into gcloud_credentials.json at the repo root + validates the file parses as JSON (fail-fast). - Set GOOGLE_APPLICATION_CREDENTIALS=gcloud_credentials.json in the "Run evaluations" env so the google-auth lookup finds it. evals/models.py - Replace module-level `translate_client = translate.Client()` with a lazy factory `_get_translate_client()` that defers instantiation u…bec2f46 -
Upload from GitHub Actions: guard main.py against partial-scale HF pushes; restore aggregated results
Verified signatureA smoke run earlier today (N_LANGUAGES=5 N_MODELS=150) silently overwrote fair-forward/evals-for-every-language-results with a filtered subset: the aggregation step at main.py:97 filters results_agg to current_models x current_languages, then save() pushes regardless of scale, so a 5-language smoke truncated the dashboard from ~1000 langs to 5. results-detailed was untouched (it merges full history), which made recovery possible. Restoration: re-aggregated the entire results-detailed table (701k rows, 135 models, 221 langs -> 70.9k aggregate rows) and pushed only the `results` dataset back. Local results/*.json snapshots now reflect the restored state. Footgun patch: - evals/main.py: * Add CANONICAL_N_LANGUAGES_FOR_PUSH (1000) and CANONICAL_N_MODELS_FOR_PUSH (100) constants. * Compute ALLOW_HF_PUSH_RESULTS at module load. When either env scale is below canonical, route results/models/languages through save_local_only() instead of save(), keeping the …691e6c2
-
Upload from GitHub Actions: refresh pyproject metadata + README HF frontmatter
Verified signaturepyproject.toml - description: real one (was "Add your description here") - drop bert-score and tiktoken from the dev group; neither is imported anywhere. Kept elevenlabs + jiwer because they pair with the dormant FLEURS/CommonVoice ASR pipeline. README.md (HF Spaces frontmatter) - models: refreshed to 8 representative current frontier models (was 6 stale entries including openai/whisper-large-v3 which is ASR and not active). Added a comment noting evals/models.py is canonical (the full curated list is ~42). - datasets: dropped mozilla-foundation/common_voice_1_0 and google/fleurs (ASR dormant); added the Masakhane benchmarks that actually drive multilingual coverage. - language: tags expanded from English+German only to a small cross-tier sample (CJK, Indic, Romance, Semitic, Bantu, Niger- Congo) for HF discoverability. uv.lock: regenerated.1eccc3f -
clean up stale root files
Verified signatureRemove 13 vestigial files from the Space root. None are read by backend.py. Authoritative copies live in /results/, /data/, /notes/ subdirs. GCS deploy retired.b369319
-
Upload from GitHub Actions: new model
Verified signature78be468
-
Upload from GitHub Actions: added new models
Verified signature83d2972
-
Upload from GitHub Actions: added new models
Verified signature93a8617
-
Upload from GitHub Actions: Merge pull request #28 from datenlabor-bmz/jn-dev
Verified signatureJn dev55b63ea
-
Upload from GitHub Actions: fixed hypoerlinks
Verified signature44a2e08 -
Upload from GitHub Actions: removed mail
Verified signatureba308de
-
Upload from GitHub Actions: fixed button issue
Verified signaturec8949e9 -
Upload from GitHub Actions: updated frontend
Verified signature2d5c7b3 -
Upload from GitHub Actions: updated button
Verified signature4419cd0 -
Upload from GitHub Actions: added disclaimer
Verified signature3a1f6aa -
Upload from GitHub Actions: minor frontend updates
Verified signaturef81f302 -
Upload from GitHub Actions: minor frontend update
Verified signaturec9babd0 -
Upload from GitHub Actions: cleaned up code
Verified signature2586cfe -
Upload from GitHub Actions: added opus 4.5
Verified signature0a17acf
-
Upload from GitHub Actions: add gpt-5.1, gemini-3
Verified signature9ea2dd3
-
Upload from GitHub Actions: update and fixed rendering issues
Verified signature4bfbb64
-
Upload from GitHub Actions: flores filter for available dev split
Verified signature34b05c6
-
Upload from GitHub Actions: model name no bracket stuff
Verified signatureaa92add -
Upload from GitHub Actions: drop normalization
Verified signature972026c -
Upload from GitHub Actions: improve norwegian fix
Verified signature6f0e312
-
Upload from GitHub Actions: add filters
Verified signature3018273 -
Upload from GitHub Actions: add langcodes dep
Verified signatured509a02 -
Upload from GitHub Actions: fix norwegian
Verified signature0cbac6c