A First Look at Qwen3.5 Quantization and Model Size

The idea

Simply put, I wanted to find out: how much do parameter count and quantization bit width actually affect a model’s capabilities?

Most benchmarks I can find online test harder tasks. What I need to know is: can this model reliably handle simple things such as formatted output and string manipulation?

Well, if there isn’t one already, I’ll write my own!

What this benchmark measures

I designed five dimensions, worth 110 points in total:

Dimension	Points	Tasks
Precision	30	Counting letters in words, counting characters, reversing strings, date arithmetic, decimal comparisons, balanced brackets
Instruction following	25	Strict output constraints, structured JSON, formatting restrictions
Practical tasks	20	Information extraction, spam classification, TODO extraction, ticket routing
Stability	15	Consistency across repeated runs, long-output formatting, multi-turn memory
World knowledge	20	Common-sense reasoning, exact facts

The code

The core evaluation logic is in eval.py. Every task has an explicit expected output, and scoring uses exact matches:

def judge_text_exact(expected: str, actual: str) -> Tuple[bool, Dict[str, Any]]:
    actual_norm = normalize_text(actual)
    expected_norm = normalize_text(expected)
    return actual_norm == expected_norm, {
        "expected": expected_norm,
        "actual": actual_norm,
    }

I didn’t use LLM-as-judge because I don’t have the money.

Models tested

This time I tested the Qwen3.5 series:

Parameter counts: 0.8B / 2B / 4B / 9B

Quantization: Q3_K_M / Q4_K_M / Q5_K_M / Q6_K / Q8_0

That’s 20 combinations, all tested. I used the llama.cpp server; give it a try if you need concurrent requests to GGUF models.

Straight to the results

An hour later, the results were in! First, the heatmap:

Hey! See that? The winner isn’t 9B-Q8_0. It’s 4B-Q5_K_M!

A few counterintuitive findings

1. 4B-Q5_K_M > 9B-Q8_0

The Q5 version of 4B actually scored higher overall than the Q8 version of 9B. Let’s break it down:

4B-Q5_K_M: Precision 16.5 | Instructions 23.3 | Practical 19.0 | Stability 15.0 | Knowledge 17.0
9B-Q8_0:   Precision 20.5 | Instructions 21.7 | Practical 18.0 | Stability 15.0 | Knowledge 15.0

2. 9B-Q4_K_M > 9B-Q8_0

Within the 9B series, Q4 actually beat Q8 (82.12 versus 81.97). A tiny gap, but still surprising:

9B-Q4_K_M: Precision 19.0 | Instructions 23.3 | Practical 18.0 | Knowledge 15.0
9B-Q8_0:   Precision 20.5 | Instructions 21.7 | Practical 18.0 | Knowledge 15.0

Instruction following again! Q4 did noticeably better there, offsetting its disadvantage in precision.

3. 0.8B-Q5 > 0.8B-Q8

The tiny 0.8B model was even more dramatic:

0.8B-Q5_K_M: Overall 55.30 | Instructions 13.33
0.8B-Q8_0:   Overall 48.48 | Instructions 8.33

Q5 beat Q8 by almost seven points! Five of those came from instruction following. This needs more experiments, though: was it chance, or can quantization actually improve scores? I’ve heard of the latter happening before.

Looking at each dimension

Precision: where 9B has the advantage

Model	Precision
9B-Q8_0	20.5/30
9B-Q6_K	20.5/30
9B-Q5_K_M	18.0/30
4B-Q5_K_M	16.5/30
2B-Q8_0	13.5/30
0.8B-Q8_0	7.0/30

This was the only dimension where 9B clearly led. More parameters really do help with precise reasoning tasks such as reversing strings and date arithmetic. The 0.8B model fell apart here, scoring only seven points.

Instruction following: 4B is unexpectedly strong

Model	Instruction following
4B-Q5_K_M	23.33/25
4B-Q6_K	23.33/25
4B-Q8_0	23.33/25
9B-Q4_K_M	23.33/25
2B-Q3_K_M	20.0/25
0.8B-Q5_K_M	13.33/25

The 4B series did especially well! Q5, Q6, and Q8 all scored 23.33 out of 25. Meanwhile, 9B-Q8_0 managed only 21.67.

Simply put: 4B might be a sweet spot, with enough capacity to learn instruction following without overfitting to other things.

World knowledge: 4B actually holds its own against 9B

Model	World knowledge
4B-Q5_K_M	17.0/20
4B-Q6_K	17.0/20
4B-Q4_K_M	17.0/20
4B-Q8_0	17.0/20
9B-Q8_0	15.0/20
9B-Q6_K	14.0/20

Wait, 4B knows more than 9B?! I checked the data. It mostly came down to common-sense questions, such as the trick question beginning “Xiao Wang’s eldest sister is called Spring…” that 4B answered correctly and 9B missed.

It makes me suspect that larger models may be more easily misled by surface patterns, stumbling on questions that require going against an intuitive answer.

Stability: most models are very steady

Almost all the models earned the full 15 points for stability. Only the Q3 and Q4 versions of 0.8B slipped, scoring 13.33.

In other words: unless the model is extremely small, like 0.8B, quantization doesn’t seem to affect stability much.

My suggestions for each size

Based on this experiment, here are a few pointers for anyone running models locally:

0.8B: a toy

Overall scores were 40–55, with severe problems in precision and instruction following. Suitable only for very simple text classification or playing around. Don’t expect it to handle serious tasks.

2B: usable, but unreliable

Scores of 62–68 are much better than 0.8B, but precision was only 10–13. Fine for tasks with low accuracy requirements; be careful with string manipulation and formatted output.

4B: the sweet spot in this experiment

Scores of 75–83 make this the most balanced choice. The Q5_K_M version even beat every 9B variant overall! For a general-purpose local model, 4B-Q5 is my top recommendation.

9B: a higher ceiling, with pitfalls

Scores were 77–82. It certainly had the best precision, but sometimes followed instructions less well than 4B.

Limitations of this benchmark

Honestly, this evaluation has its problems. Everything uses exact matching, so even an extra “Sure, the answer is:” prefix counts as wrong. Real use might not be so strict. There are no creative-writing or coding tests. It favors rule-based and structured tasks and says nothing about the aesthetic quality of open-ended generation. Still, I think it’s a serviceable benchmark: it can give you a rough idea of a locally deployed model’s quality in just a few minutes.

Final thoughts

After running these 20 models, my biggest takeaways are:

Quantization doesn’t weaken every capability equally. Some dimensions are sensitive to it; others aren’t.

Parameter count and bit width aren’t simply interchangeable. A 4B-Q8 isn’t a substitute for a 9B-Q4. (Laughs.)

4B may be the sweet spot for local deployment. We already saw evidence of that in the Qwen3 2507 era.

Thank you for reading this far! You’ve worked hard today, too!