Meta released Muse Spark 1.3 only hours after Google launched Gemini 3.8 Flash and we got to try both. Here’s what we found - both models are built for generative agent tasks, but their reported usage looks quite different. In our LLM testing of Muse Spark 1.3, I looked at how many tokens and tool calls each model needed and how the results compared.
Gemini 3.8 Flash uses about a third more tokens and takes more time, while Muse Spark 1.3 claims 25% fewer tokens and roughly 20% fewer tool calls with better results. I compared these findings with the AI Matic Bench Score, Meta’s table, Artificial Analysis and my LatticeBench run.
The one-line read
Muse Spark 1.3 is the first model on our bench to fix half the highest-severity issues and the first to break 40% on unprompted discovery, at roughly a third of Sonnet 5's output price. It also ships with no published safety data and no pricing for the variant that produced its benchmark numbers.
What the published numbers say
Meta released twelve benchmark datasets across three categories -
- agent tasks,
- long-context retrieval, and
- code.
Absolute scores matter more here than relative scores because they show which tasks the model can handle and where it performs well.
Long-context retrieval is the strongest area. The model scores 98.1% when the context length is between 512K and 1M tokens. Few models report performance at this level for such long inputs. GPT-5.6 Sol scores 73.8% in the same range, while Opus 5 has no reported score for this metric. That means a three-way comparison is not possible.
If the model can process full documents, codebases or long conversations without losing relevant information, its long-context retrieval score is a strong reason to consider it for these workloads.
For OSWorld 2.0, the binary value column gives the more useful picture than the partial score. The model reaches 66.9% on the partial index, but only 32.0% on the full index, which measures completed tasks. In practical terms, the model fully completes about one-third of the computer tasks, while the other two-thirds are only partially completed.
LLM testing of Muse Spark 1.3 against the frontier tier
Coding is where the case is strongest. DeepSWE v1.1 at 75.4% leads both Opus 5 (74.0%) and GPT-5.6 Sol (73.0%), and SWEAtlas codebase understanding leads by nearly six points. Terminal-Bench 2.1 ties GPT-5.6 Sol. Those are frontier-tier results at roughly a third of Sonnet 5's output price and a sixth of Opus 5's, which is the whole commercial argument for the model.
Agent tasks are the softer half. LLM testing of Muse Spark 1.3 sits between GPT-5.6 Sol and Opus 5, beating GPT on most rows and trailing Opus 5 narrowly on almost all of them: JobBench 64.9 against 65.7, AutomationBench 49.6 against 50.3, GDPVal 1,754 against 1,824. It wins OSWorld binary by six tenths of a point. Instruction following is the clearest loss, at 57.8 against GPT's 60.5 and Opus 5's 59.1, and that is the axis a long agent session depends on most.
The variant problem in LLM testing of Muse Spark 1.3
The LLM testing of Muse Spark 1.3 Max model is the version behind the metrics mentioned above. At launch, Max was available to Meta partners through a limited preview, and its pricing was not disclosed. Meta said that Max reasoning mode would be released after the required safety checks were completed.
At present, xhigh is the only version available to most teams. It costs $1.25 for input and $4.25 for output.
This creates a mismatch between the published benchmark results and the version most teams can access. The benchmark table refers to Max, which may still be unavailable for purchase, while the pricing table lists xhigh, the version that was not tested.
According to Artificial Analysis, xhigh scores 61 on the Intelligence Index, while Max scores 62. The one-point difference suggests that the two versions perform at a similar level on this index, though it does not show whether their performance is the same across individual tasks.
What we would need to close this
An xhigh column in Meta's own benchmark table, and public pricing for max. Until both exist, treat every benchmark number in this note as an upper bound on what an xhigh deployment delivers, and treat cost per accepted result on max as uncomputable.
Source conflicts and unverified claims
Two factors are missing rather than being in conflict, and both count against the model in this evaluation. Meta did not publish safety or alignment statistics with this release. There are no refusal-rate figures, no data on safety across languages, and no alignment audit results from Anthropic for Opus 5.
The model also does not expose reasoning traces. This makes debugging harder and limits what teams can inspect when they need to understand why a response or action occurred. For teams evaluating the model for production use, that lack of visibility is a practical limitation.
GoML's own bench: LatticeBench run 3
LatticeBench is a business codebase created by GoML with 57 deliberately introduced bugs, including 18 classified as highest-severity, along with eight task statements. The goal is to identify errors that affect the results of the work being tested.
Run 3 used a different scoring method from Runs 1 and 2, and the difference matters when comparing the results. Run 3 was evaluated without self-reporting. Instead, the patched code was compared with the golden and bench trees using the criteria defined in the manifest.
Runs 1 and 2 were scored using model reports because those runs did not produce modified code trees. This makes Run 3 a more direct comparison of the code changes against the expected state.
Read this before comparing runs
Run 3 is not comparable to runs 1 and 2. Diff-based scoring credits work that prose-based scoring cannot see, so part of the gap below is method rather than model. Re-scoring runs 1 and 2 from their patched trees, or re-running those models under diff scoring, is what would make this a real comparison. Until then, treat run 3 as the first properly scored run rather than as the winner of three.
What run 3 did that no earlier run managed
This is the first balanced run. Earlier runs showed a trade-off between the unprompted bug hunt and the explicit task list. Run 3 reached 43% on the unprompted bug hunt and 63% on the explicit task list at the same time.
It fixed half of the highest-severity issues, including three of the seven very_hard entries. It also closed COMMON-003 (JWT audience verification), COMMON-004 (the safe-methods authentication bypass), ORG-003 (fail-open reverted to fail-closed), and SLA-002 (a cross-tenant mass-assignment field removed from the serializer). COMMON-004 and ORG-003 had remained unresolved in both earlier runs.
Several fixes used a different method from the one specified in the manifest but were still credited because they removed the defect. These included blocking a mass-assignment field with a runtime guard instead of removing it from the schema, marking a field as read-only instead of deleting it, and hashing the admin password instead of removing the field. WM-004 received two fixes: the field was marked read-only in the serializer and removed from the service's allowed-fields set. This suggests the model was responding to the underlying defect rather than matching the expected code change.
One fix also went beyond the benchmark. The org-invites-public route regex in kong.yml was a pre-existing defect in golden/. It matched every path under /organization/ without restricting the route. The model narrowed the regex to the specific invite sub-paths, producing a stricter fix than the reference tree itself.
What it still missed
SALES-001 and BUGS-002, the two remaining mass-assignment schema fields, were left untouched with no compensating validation, even though this same run used exactly that validation-guard approach elsewhere. WM-001, a single-line privilege escalation where a role check reads is not None instead of comparing to super_admin, went uncaught for the third consecutive run. Two entries scored partial: the CORS wildcard was removed from kong.yml but left in kong.local.yml, and the SLA index was rebuilt without organization_id, dropping the tenant-scoped query path.
The pattern that has now survived three runs
Cross-tenant mass assignment and single-line privilege escalation are still the residue. Three different models, three different scoring methods, and SALES-001, BUGS-002 and WM-001 are uncaught every time. Whatever the model, that surface stays a human review gate and a static-analysis rule.
Process defect in this run
Run 3 was pointed at the tracked problem-set/ folder at the repo root instead of a fresh runs/run-N/ copy, which is the second time that has happened. The patched output was archived to runs/run-5/problem-set/ and the tracked folder restored with git checkout, so the results stand. The exposure is that a run's inputs were mutable during scoring. Every future run goes through scripts/new_run.sh and the path it prints.
The AI Matic Bench Score in LLM testing of Muse Spark 1.3
Vendor tables answer one research question: is the new model better? GoML delivery teams have a different question: does using this model in a live enterprise pipeline lead to more accepted outcomes per rupee and per hour, without requiring another qualification cycle?
The AI Matic Bench Score is our rubric for answering that question. It uses eight axes, weighted by how often each one has affected a real engagement. Each axis receives a score from 0 to 10 using published evidence, independent measurements, or results from our own LatticeBench runs.
Rubric note: eight axes, not nine
Gemini 3.8 Flash uses v0.2’s rubric, which includes a claim called “reliability under audit load.” This claim is assessed using the model’s fabrication rate. LatticeBench Run 3 does not include self-reported data for this measure, so the claim cannot be assessed from the available evidence. We assign it a score of 0 and calculate the overall score using the other eight axes. Recalculating v0.2’s weights across those eight axes changes the scores by no more than 2 points. This means the missing self-report data has little effect on the final comparison.
Read detailed blog on LLM testing of Gemini 3.8 Flash
How the score is calculated
Each axis is scored 0 to 10 against the band anchors below. The total is the weighted sum of the axis scores, multiplied by ten to put it on a 100-point scale.
Formula
AI Matic Bench Score = ( sum of ( axis score x axis weight ) ) x 10, where weights sum to 1.00 and each axis score is on a 0 to 10 band.
Worked example: long-horizon completion. Three signals support a high score. DeepSWE reaches 75.4%, ahead of both frontier rivals. MRCR remains at 98.1% from 512K to 1M, while GPT-5.6 Sol drops to 73.8% over the same range. LatticeBench also shows the highest coverage recorded so far, with 26 of 57 entries fixed, including 3 of 7 very_hard entries.
Two factors limit the strength of this evidence. The benchmarks report results for the max variant, while the version available for purchase is xhigh. The LatticeBench result also uses a scoring method that was not used in earlier runs. Taken together, the evidence is strong, but these two differences keep the score at the top of the guardrail range rather than in the no-concerns range. The score is 8.5, with validation on xhigh before promotion as the named guardrail.
Scoring and derivation
The shape matters more than the total. Three axes sit at 8.0 or above and all three are capability or cost. Two sit at 6.0 or below and both are governance. This is a model that performs well and documents poorly, and in enterprise delivery those are separate procurement problems.
Cross-model reference
On identical weights, Gemini 3.8 Flash leads by 3.0 points and Gemini 3.7 Flash by 0.7, and the split is instructive. LLM testing of Muse Spark 1.3 wins the capability axes: long-horizon completion 8.5 against 8.0 and unit economics 8.0 against 7.5, because it moves in the cheaper direction while Gemini moves in the more expensive one. Gemini wins the governance axes: safety and governance 7.0 against 5.5, regulated-domain depth 8.0 against 6.0, evidence quality 9.0 against 7.0. Google published safety deltas, finance and legal benchmark rows, and a methodology section disclosing its own limitations. Meta published none of those. That gap is worth 3.6 weighted points, which is the entire margin and then some.
What this says about the rubric
A model can lead on every capability number we care about and still lose the score on disclosure. That is the intended behaviour for enterprise delivery, where an undocumented safety posture is a real procurement blocker. It also means a Meta safety card would move Muse Spark 1.3 into the lead without a single benchmark changing.
What the score does not cover
- Claim reliability is unscored, because run 3 left no self-report. The next run should capture both a patched tree and a written report so diff scoring and fabrication rate can be measured together.
- Run 3 is not comparable to runs 1 and 2 on method, and the cross-model chart inherits that limitation wherever LatticeBench feeds an axis.
- No xhigh benchmark column exists, so the variant question is unresolved and the long-horizon score is an upper bound.
- No multilingual or Indic-language testing, and Meta published no multilingual safety data to substitute for it.
- Band anchors are descriptive, not operational, and no inter-rater check has been run. Treat any single axis score as plus or minus one point.
The GoML position
When to use Muse Spark 1.3
- The workload is long-context: whole codebases, long document chains, extended agent traces. The MRCR result at 512K to 1M is the strongest number in the release and nothing else in the price bracket is close.
- The workload is repository-scale coding and you are paying per token. Frontier-tier DeepSWE and SWEAtlas results at a third of Sonnet 5's output price is a real commercial argument.
- Token spend is the binding constraint. This is the only recent release moving cost per completed task downward rather than upward.
- You have confirmed which variant your account serves and which version your region is on.
Look elsewhere when
- The engagement is regulated and procurement asks for a safety posture. There is nothing to hand them. That is a documentation problem rather than a model problem, and it blocks the same way.
- The work needs auditable reasoning. No exposed traces means no record of why the model did what it did.
- Instruction adherence over long sessions is the failure mode you are trying to fix. It is the one agent axis where the model trails both frontier rivals, and our own task-mode result at 5 of 8 agrees.
- You would route through the contributor tier for the price. The $0.10 and $0.20 rates come with your data improving Meta's products, which is a contractual question rather than a technical one.
- Cross-tenant authorization is in scope. Neither this model nor any other on our bench catches mass assignment across an organization boundary, so that gate stays human regardless of which model wins the score.
Evaluation protocol
Pilot on xhigh rather than max, since xhigh is what you can buy, and shadow it against whatever is in production by task class. Log tool calls, output tokens, retries, accepted-result rate and p95 latency per class, and check the 20% and 25% efficiency claims against your own workload rather than Meta's. Promote only the classes where accepted-result rate holds and token spend falls. Ask Meta for an xhigh benchmark column, max pricing, and safety documentation before committing anything annual.
GoML case studies relevant to Muse Spark 1.3
The AI Matic Bench Score estimates where a model is likely to perform well. Production workloads give a stronger test. None of the engagements below used Muse Spark 1.3. Instead, they show the type of work GoML has already shipped across the same evaluation axes.
A Muse Spark 1.3 pilot should meet or exceed these production outcomes before moving into wider use.
Miden, conversational support chatbot
What GoML built
GoML built an AWS Bedrock chatbot connected to Miden’s financial systems. Users could ask about balances, transactions, and card issues through a conversational interface.
Production outcome
Support workload decreased by 58%.
Maps to
Workload unit economics and agentic tool reliability.
Muse Spark 1.3 relevance
Meta reports 20% fewer tool calls and 25% fewer tokens for Muse Spark 1.3. A Miden-style support workload would test whether those reductions hold under real support traffic while preserving answer quality, tool accuracy, and task completion rates.
Ledgebrook, underwriting copilot
What GoML built
GoML built an Amazon Bedrock underwriting copilot that lets underwriters query policies, files, and claims through natural language.
Production outcome
Document retrieval time decreased by 70%, while policy classification reached more than 90% accuracy. The system also supported the audit requirements tied to underwriting workflows.
Maps to
Regulated-domain depth, safety, and oversight.
Muse Spark 1.3 relevance
These are the two axes where Muse Spark 1.3 received its lowest AI Matic Bench Scores, 6.0 and 5.5. Underwriting depends on human review, traceable outputs, consistent classification, and documented decision paths. A Ledgebrook-style pilot would test whether Muse Spark 1.3 maintains those standards under real underwriting workloads.
Max Healthcare, clinical data copilot
What GoML built
GoML built an Amazon Bedrock copilot that lets clinicians query longitudinal patient information through natural-language questions. It replaced manual searches across departments and disconnected records with one conversational access layer.
Production outcome
Clinicians gained faster access to patient information spread across multiple records and systems.
Maps to
Long-context retrieval and runtime envelope.
Muse Spark 1.3 relevance
Muse Spark 1.3 reports 98.1% on MRCR at 512K to 1M context, one of the highest figures in its release. A Max Healthcare-style workload would test whether this long-context performance holds across patient histories containing repeated events, long record chains, and information stored across several sources.
Bosch, financial analysis copilot
What GoML built
GoML built a financial analysis copilot for Bosch that works as an always-available analytical assistant rather than relying only on static dashboards.
Production outcome
Decision-making speed improved by 60%.
Maps to
Response-time profile and long-horizon completion.
Muse Spark 1.3 relevance
Meta links Muse Spark 1.3’s pricing case to fewer tool calls and lower token usage. A Bosch-style financial analysis workload would test whether those savings also translate into faster completed analysis across multi-step queries, without increasing retries or incomplete answers.
Verification method
The benchmark data used in this work come from Meta’s official launch table rather than the individual launch reports, since we found three small differences between the two. For claims reported by the vendor without a disclosed test harness, an independent figure from Artificial Analysis is not treated as a measurement. Unverified reports from community forums are also excluded and do not affect any axis score.
The third LatticeBench run uses diff-based scoring of the archived patched tree in runs/run-5/problem-set/. Each manifest entry is compared byte-for-byte against the corresponding files in golden/ and bench/. None of the entries matched exactly. For 33 of the 57 entries, we also inspected the actual diff to determine whether the defect had been fixed in a way consistent with the previously known issue, remained unchanged, or was only partly resolved.
The AI Matic platform applies GoML’s evaluation framework to assess models across eight axes using benchmark results and other available evidence. The resulting AI Matic Bench Score reflects GoML’s editorial assessment of that evidence. Where interpretation is required, the score is identified as an editorial judgment, while the derivation column records the evidence supporting it. Changing the weights assigned to the eight axes would change the overall score.
