# Muse Glimmer ad hoc evaluation — reproduction notes

This is the exact prompt-and-grader bundle behind:

https://wasnotwas.com/writing/muse-glimmer-on-an-rtx-4090-a-good-agent-that-thinks-too-much/

Repository browser:

https://github.com/SamSaffron/wasnotwas-com/tree/main/static/artifacts/meta-muse-glimmer-30b

## Requirements

- Ollama listening on http://127.0.0.1:11434
- Python 3.11+
- Ruby on PATH for executable Ruby graders
- SQLite support in Python's standard library
- Models registered under these exact names:
  - muse-glimmer:17gb
  - gemma4-26b-agentbench:latest

No third-party Python packages are required.

## Muse Modelfile

    FROM /absolute/path/to/muse-glimmer-30B-kquant-17gb.gguf

    PARAMETER num_ctx 32768
    PARAMETER temperature 1.0
    PARAMETER top_p 0.95
    PARAMETER top_k 64

    SYSTEM "Reasoning strength: low"

Create it with:

    ollama create muse-glimmer:17gb -f Modelfile

The Ollama build must include llama.cpp Muse Glimmer support. Ollama 0.32.7's pinned llama.cpp did not; the tested local build used llama.cpp commit 4dee52f.

## Run

From the scripts directory:

    python3 benchmark.py
    python3 addendum.py
    python3 hard_benchmark.py
    python3 language_benchmark.py

Each script writes its result JSON in the current directory. Existing captured outputs are under ../results/.

## Files

- scripts/benchmark.py — base correctness, speed, tool recovery, and context retrieval
- scripts/addendum.py — full-answer timing and additional Ruby/calibration tasks
- scripts/hard_benchmark.py — topology, parser, SQL streaks, optimization, shortest path, and injected tool data
- scripts/language_benchmark.py — Hebrew/English translation and constrained summarization
- results/results.json — base suite raw Ollama responses
- results/addendum.json — full-answer and additional task responses
- results/hard-results.json — hard-suite raw responses and executable grader output
- results/language-results.json — translation and summarization responses
- results/muse-knapsack-4096.json — Muse's successful 4,096-token knapsack retry
- results/muse-merge-1024.json — Muse's successful larger-budget interval-merging retry

## Important limitations

- These are ad hoc tests, not a statistically powered benchmark.
- Most quality tasks were run once per model. Speed probes were repeated where noted.
- The models use different tokenizers and internal reasoning behaviour.
- Executable graders are stronger evidence than visual inspection, but their test coverage is still small.
- The original interval wording was ambiguous about adjacent integer intervals. The addendum uses an explicit condition.
- The addendum's first-pass calibration grader omitted the phrase "does not state", so both correct calibration responses are marked false in addendum.json. The article reports the semantically correct result.
- The first translation request for each model included model-switch loading and is not a warm latency comparison.
- Muse used Vulkan without DFlash. Meta's RTX 5090 CUDA+DFlash numbers are not directly comparable.
- Raw JSON includes the models' returned thinking fields. It contains synthetic benchmark data only.