Why this project exists
So I was running the custom Deepseek I made and was loving it for two reasons - it was extremely fast, and it really was a jump in capability for me running a local model. To be fair, it was also by far the largest model I’ve ever run. But I had a few complaints with it - its prose / responses are littered with AI slop and no matter how many guardrails you put into your harness, it sounds like the worst of AI all the time. And while it was a massive jump in ability, it had a tendency to break down in the details. Like it made a deck for me and the design was just slightly off from slide to slide. Everything was close… but just not quite there.
GLM 5.3 Flash came out and all of the benchmarks were a jump from Deepseek so I gave it a shot. And it was terrible. The first three prompts I gave it made the model spin out and not respond for about 30 minutes each time. Something wasn’t right with it. So I switched back to Deepseek.
Then I kept reading everyone else saying how good GLM was. I know sometimes it’s not the model, it’s the runner - so I downloaded it again. I tried three variants with two runners and all of them were terrible. But just like half of the other projects here, I got stubborn and believed I could fix this.
So here we are.
The model
Something to note here - GLM 5.3 Flash is a very large model, about 10gb larger than Deepseek v4 for the weights alone. This one pushes the dual DGX system to its limits. I say this because most of the variants I was combing through were all trying to quant the model just right to fit on this hardware (since most people I see online running models of this size have the same hardware I do).
After trying a few different ones I opted for a quantization type I’ve started to see but never used: EXL3. I’ve read a few articles on this, and I’ll be honest - I don’t understand it well enough to explain in any way. Bottom line: it’s newer, and supposedly it keeps a high quality and is very fast (both in the quantization process itself and in usage). The specific model I settled on was this one: neko-legends/GLM-5.3-Flash-Uncensored-EXL3 · Hugging Face from Neko Legends. And yes, as always I go for the abliterated models.
So we had our model, and now it was time to pick the runner.
The runner - vLLM
So we started with the suggested runner from the model card - a customized version of vLLM from MiaAI Lab (who is highly regarded in the community for making these custom builds). We started with her runner here: MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks: GLM-5.3 Flash EXL3 for 2x DGX Sparks but it was also a bit prone to spinning out of control.
We did some digging online and it turns out we weren’t the only ones having this issue. And it looked like another similar custom runner might have had that problem fixed, so we next moved on to this one: Entrpi/glm-5.3-flash-exl3-2x-spark: GLM-5.3-Flash (EXL3 4bpw + DFlash2) on 2x NVIDIA DGX Spark: one-shot installer, 33-74 tok/s c1, 1.3M+ context, vision from Entrpi, another well-regarded community dev.
This one worked! But there were issues - it was still pretty slow, and it still had some spinning. With some digging we found quite a few upstream PR’s into vLLM to address some of the issues we were finding (again mostly related to how the runner guards against reasoning that won’t stop). But some of the other PR’s we found pointed to other aspects of the runner that made me want to dig deeper. Most of the libraries needed to make this model and this new quant type work weren’t compatible with vLLM when the model came out (which goes back to what everyone says - when a model comes out it’s not going to be supported well, you have to wait a few weeks and let it bake before you should start using it). Entrpi took these libraries and custom bolted them on. In a few cases there were workarounds / fallbacks - one of the libraries to support the backend was a version that wasn’t meant for this hardware. It worked, but clearly the inefficiency was contributing to the low tok/s we were seeing.
Our first stroke of luck - z.ai updated the jinja template for GLM and that fixed quite a few of the spinning issues we had (more evidence that you just have to wait a few weeks before trying new models). So we added that to the list of things to update and test out.
And then the next day vLLM came out with a new version - v.29. So I sat down with my agent and came up with a plan to code our own version. We started to research and laid out all of the dependencies for the model and which ones had upstream versions that fit our requirements:
- Native compatibility with GLM 5.3 Flash
- Native compatibility with vLLM v.29
- Native compatibility with DGX Spark
These requirements create a pretty narrow list. But we identified every library and went to work creating our own custom one. I won’t go into the full list, it’s on my GitHub, but there were 22 total updates we made to get it to work. Our runner is here:
We finished the build, and we fired it up! And it gave us an error. My agent tells me, “this failed because this is the older method of EXL3 quantization, and this new library is expecting the new quant instead.”
Are you kidding me? We spent two full days coding this and I went through over a week’s worth of tokens only to find out that this wouldn’t work with the model we were trying to code for.
Back to the model
Ok, so after a few deep breaths we came up with the next part of our plan - it actually wasn’t hard to update the EXL3 quant to work with the new format. It’s called “rank slicing” and has to do with taking the model and setting each shard for splitting between the two nodes of our dual DGX cluster. But there apparently isn’t a tool to do this, so we had to build that too. It’s in the same repo as our runner above.
So we sliced the model and created a new version:
We tested it and it worked really well! There was a 15-25% speed increase (we’re still testing but we’re seeing that consistently across the board). And we haven’t had any spinning. We’re still tweaking things for speed - we’re using a dflash drafter model (this one: incoai/GLM-5.3-Flash-DFlash2 · Hugging Face) and the runner and draft model are made to work with 7 draft tokens.
So we made our 23rd update to the runner to allow for fewer draft tokens. In theory we probably should have attempted to train the draft model also, but that’s a massive endeavor and in general you can take a draft model and reduce the number easily. You have a lot more issues if you try to increase the number. Since our drafter was made to predict 7 tokens, going down to 5 shouldn’t have had much of an issue.
We’re still testing it - so far the token generation has dropped a bit. On average our numbers were the following:
- Baseline original runner: 22.65 tok/s
- Our runner with 7 tokens: 26.02 tok/s
- Ours with 5 tokens: 24.32 tok/s
But this doesn’t tell the whole story. This is only the first day of testing, and it’s all lower context size tests. So yes, token generation goes down slightly, but we haven’t yet been able to do full wall time (end-to-end time from prompt to response) testing on large context prompts. We have our context set to 800k and when you have very large context work, extra dflash drafting can impact that heavily. The increase in token generation speed is often not worth the offset you lose in prompt processing.
We’ll see - the numbers will tell us whether we end up at 5 tokens or 7 for our drafter. Either way we have some real increases.
And in a rarity for me - the benchmarks on Artificial Analysis (https://artificialanalysis.ai) actually line up to my experience: GLM 5.3 Flash is significantly better than Deepseek v4 Flash for long horizon agentic ability. This thing can one-shot almost anything I’ve thrown at it, and it gets it right on the first try. I am genuinely impressed and finally feel like I have frontier capability at home. It’s slower for sure (Deepseek was consistently getting over 50 tok/s TG), but the end result is much faster when I can get exactly what I want in one prompt instead of having to steer the model continually to iterate until I get the same result. One long prompt response is significantly faster than a dozen short ones.
Status
I thought this project was complete, but new upstream PR’s into vLLM added speed boosts and compatibility / stability with my system. We’ve been implementing them and fixing bugs and now are on release 5 of this runner. We’ve seen some very good performance improvements across the board and will continue tweaking our updates.
I’ve also seen the number of downloads of this runner much higher than I expected so I’m going to continue to move this project forward.