Heretic DGX

Why this project exists

One of the main applications used to abliterate models is called "Heretic" by Philipp Emanuel Weidmann (who goes by "p-e-w" online). From my limited testing, this software wasn't made to work on a model that's split across two systems (via tensor parallelism). This means a setup like mine where I can run larger models across two computers isn't able to use Heretic to abliterate a model of that size.

As with every other software project on here - when I find something I want to use and it doesn't exist, I build my own.

The build

As I've said before, I'm not a developer, so I understand code up to a point, and the rest I have to trust my AI agent on certain details. I bring this up because I didn't go through the Heretic codebase; it's not a complete black box to me, but I also don't understand it fully.

From what I can tell, Heretic works in a multi-step process: it takes a downloaded model (the weights) and loads those into the software. Generally for software like this, that means the entire model has to fit in memory before it can be worked on, but in this case it looks like it can be streamed from the hard drive. So in theory you can work on a model larger than you can handle, but in this case it seems like it would be prohibitively slow to do so. If I want to abliterate a model that's 200gb in size, I don't want my computers offline for most of the week while this runs.

After it loads the model it determines how to train that specific model (which it should be noted, models have different architectures, and if there's a niche model with a nonstandard architecture, it likely won't work without an update to Heretic) and creates what's called a LoRA adapter (LoRA stands for Low-Rank Adaptation, and is an efficient way to post-train an existing model). You then take the LoRA adapter and use that to train the original model on removing its refusals. It then can create a final copy of the model with the training baked in, and the refusals are either reduced or removed. You can set the level of ablation you want in Heretic, keeping in mind the more you try to change the model from its original state, the higher the likelihood you can lobotomize it.

There were some interesting problems to solve in this build, and some of them go against how the DGX is made to operate. Most notably, when you have a model that has to run across both nodes of the system, the best way to run that is to have a copy on each node. This way each node can load the model into memory from its own drive, which takes significantly less time to spin up. It's possible to have a copy of the model on just one box and stream it to the other, but it's not recommended. For Heretic, that creates the slight issue of "which file am I abliterating if there are two copies?"

In the Nvidia DGX ecosystem, there's a specific type of networking port on these boxes using a cable called a ConnectX7 cable. It gives a full 200gb/s duplex bandwidth across these, giving it almost zero latency. It actually creates a separate network with its own IP addresses for these two nodes and talks over a separate protocol. So for our update, Heretic on launch had to find this separate network, identify both nodes, and be able to route traffic in each direction on this network. Then we had it load the model in from one of the nodes and stream it to both to work on. Then it had to use the DGX as intended and work on both boxes as if it was one large pool of memory and gpu. This took the most work - once we got past the networking issues, the rest of Heretic functioned relatively close to original intent.

The final major changes are because I'm not technical enough to want to use every command to do every step the right way. So two more changes were needed: the first was a simple one; Heretic would create the LoRA and then you had to tell it to use the LoRA in a separate step and I didn't want to have to do that. So we updated it to just create the end model as the output.

The other took more work: if you have a quantized (compressed) model, Heretic by default was saved final output copies in bf16 (uncompressed) format, which in some cases increased the size by 2-4x (note here that I'm not sure if this was the default or just what was happening in my case because I didn't know every flag to pass in). For me this wasn't acceptable; I couldn't have a model that was too large to run, so giving me an uncompressed version isn't something I could use. Also if it saved these massive files, I would run out of disk space quickly. Additionally, and even more importantly, there are some really good quants out there of these models, ones that are done much more expertly than I can do, and I didn't want to waste that work on requantizing a model that wouldn't work as well (the worse a quantization is, the more incoherence you'll find in usage). So my final requirement was that the output model had to stay in the original quantization.

Status

Thankfully after a few days of development it looked like the software works. For now we can call this project complete, but if p-e-w decides to update his software in a meaningful way, I'll likely piggyback and update mine.