Custom DeepSeek Stack

Why this project exists

I finally acquired enough hardware to run a frontier-class AI model locally, but these larger ones take up every resource I have. Prior to trying to run a model of this size, I had multiple smaller models each with specific purposes (orchestration, coding, tools, etc.), but the ClearPing project showed me - if I want to be able to build complete projects locally I need a more capable daily driver (read: BIGGER).

I got DeepSeek v4 Flash 0731 working with a large context size and the output rivaled what I was getting from ChatGPT in many ways. But when you're trying to build a website, one key functionality is "vision" - where a model can interpret images. You need this for art asset creation (try explaining to a model that can't see how to change an icon they created), but you also need it for page layout and automated testing.

We had one main problem to solve: DeepSeek v4 didn't have vision, and the model is too big to run a separate vision-capable model alongside it.

To solve this problem, a few people in the community grafted on a vision tower from K2's Kimi model, but that added 8gb to the weights and that was just over my threshold to host it. So I found a smaller vision tower made to graft onto DeepSeek that was only about 800mb in size. This would fit perfectly for my needs.

Once we made the model, we couldn't find a version of vLLM for inference providing that would run it. So we had to make that too.

The model

We started from the abliterated DeepSeek fine tune made by Cebeuq we were already running (see my writings page for why I run abliterated models), so we knew it worked well. We grafted the separate vision tower on top of it, which was a relatively easy process. It worked and we were happy with the results. This model can be found here:

Then a newer, much narrower abliteration came out by Drowzeys. I didn't really care about having "zero refusals" - I cared about it executing well. So we decided to make a new version based on this checkpoint. We grafted the same vision tower onto the newer base since we already proved that out and liked the results. Honestly, I didn't realize the first version was worse until we used the new one. The difference was real. It's hard to quantify, but orchestration, tooling, and coding all felt tighter. It was immediately more capable and coherent. Truth be told it lost a little bit in the personality; it's a little stiff to talk to. But it's a worthwhile tradeoff, and we tell everyone these differences in both of the model cards online so they can choose which one to use. The new model can be found here:

Both fine tunes were made in similar manners: this is a component composition, not weight averaging. The language shards stay byte-identical to their source; the vision tower and projector load separately and the custom runtime inserts the image embeddings at the image token. The result is a vision-capable model we can actually serve.

Note that links to the original models we built ours from are in the model cards for each of our models above.

The runner - vLLM

Let me start with: I stick with vLLM for inference because it handles concurrency and prompt processing better than any other I've seen (I've tried llama.cpp and its derivates like LM Studio, SGLang, Atlas, and VeloGB). Because my hardware stack is Nvidia's DGX Spark boxes, there are custom versions of vLLM for that hardware. A hint for anyone who is interested in DGX - look for sm121 or GB10 when you want to find software made specifically for this box.

Eugr has a popular custom version of vLLM that runs amazingly on the DGX but upstream vLLM didn't support DeepSeek in its latest official release. Anemll created a custom version of vLLM specifically for DeepSeek that actually ran pretty well for us when we used the original Cebeuq version of DeepSeek. Our version wouldn't start on this runner though. The extra vision tower creates a mismatch in what the runner expects to find and it failed at boot.

So I put my agent to work (I used my agent running GPT Sol 5.6) and we updated the Anemll runner to be able to host our model. We tested it and it looked like everything was working. It seemed slow, but I was throwing a lot of larger tasks at it, so at first I thought nothing of it.

Then it stayed slow so I started looking at the logs and found that the vision tower broke the Dspark speculative decoding. This meant our token generation speed went from about 50-60 tok/s down to about 14. So we went back with this information and dug in and was able to fix the runner. Speed went back up to pre-grafting levels and we were happy. We also added in a patch from Eugr's repo that improved performance on the DGX so that's a plus. Both versions (1.0 and the fixed 2.0) of the runner with release notes are hosted here on my GitHub site:

For anyone attempting to use this version, one warning on something I recently found out and haven't yet added to the GitHub readme - this version of vLLM has a bug for how it reports cache size. It was telling me I had enough cache size to hold about 1.8M tokens, so I set it for 800k context window at two concurrent lanes and thought I had room to spare. The real cache is actually significantly smaller, more like 1.1 - 1.2M.

We had it running, but the Anemll version was pinned to v25.2 of vLLM and 27.2 was out with better DeepSeek support. It wasn't fully supported, but it was most of the way there and there were upstream PR's we could backport that would fill most of the gaps. So I had a new goal - let's get off someone else's checkpoint as a base and create one from vanilla vLLM ourselves. We started on version 3.0. We had it fully built in a separate repo since I no longer needed to call out someone else's work and after initial testing we uploaded it to GitHub where I could say I had a clean version on the latest and greatest. And 15 minutes later vLLM released 28.0 with full DeepSeek support. Ugh. The 3.0 repo is still marked as private, but I kept it running locally - mostly because this version changed the backend from B12X to Marlin, which I keep hearing is more performant, so we wanted to test that theory (turns out it's about the same).

I found some issues with it though - I tried to dial back the Dspark drafting from 5 tokens down to 3 and it broke everything (speculative decoding helps token generation speed but slows down prompt processing, which also slows inversely to context size; so with a huge context window the gain from 5 drafted TG tokens isn't worth the extra slowdown in PP). My agent explained exactly why the drafting broke, but this is going way too deep in the model architecture for me to understand it enough to regurgitate here. We started to go down the path of fixing it, but we spent an entire day and honestly got nowhere.

Then we tried to build a new one from scratch. We started with the new vLLM version 28.0 base and incorporated even newer upstream PR's that fix the Dspark drafting issue we were trying to solve ourselves. The new 28.0 version fundamentally changed how a few of DeepSeek's quirks function, so there wasn't anything from the old Anemll runner that we could use now. And it looks like Marlin and Triton now both fully work in addition to B12X as well. I also wanted this to be more cutting edge so I am pinning it to the newest libraries for Flashinfer, Transformers, B12X, and the other dependencies. And we're on the 13.3.1 version of the Nvidia CUDA driver (our old version used 13.0). Another tip for anyone working with CUDA - there's a bug in 13.2 on DGX that affects many models; stay away from that if you can.

We spent a week trying to get this to work and it just didn't. At some point you can't keep banging your head against the wall, so we put it aside. We're back on the 2.0 version.