Locking Pretrained Weights through Deep Low-Rank Residual Distillation

The standard of open-weight language fashions has dramatically improved lately. Sharing weights drastically facilitates mannequin adoption by enabling their use throughout various {hardware} and software program platforms. In addition they permit for extra open analysis and testing, to the extent that customers can use them as checkpoints, fine-tune them in response to their wants, and probably redistribute them. In some circumstances, nevertheless, considerations on modifying these weights in the direction of unauthorized makes use of could outweigh the professionals of giving customers such a freedom. Defending towards such adaptation is non-trivial: since an adaptive attacker can observe all weights and architectures by definition, they will reverse easy structural defenses, and use optimization to defeat the best locking mechanisms. On this work, we exploit the inference–coaching asymmetry of computerized differentiation as a novel protection axis. We suggest DLR-Lock, a way the place the purveyor of the mannequin purposely replaces every pretrained MLP of their mannequin with a deep low-rank residual community (DLR-Internet) of comparable parameter depend, forcing activation reminiscence that grows linearly with depth throughout backpropagation. DLR-Nets are effectively educated through module-wise distillation. We present that, past this reminiscence overhead, DLR-Lock ends in architectural mismatches that complicate the optimization panorama of ordinary fine-tuning, and a backward go that incurs disproportionately extra overhead than the ahead go. Our protection succeeds in withstanding adaptive attackers with full data of the protection technique whereas preserving the unique mannequin’s capabilities. Experiments on LLM validate these claims.

† College of Tokyo** Work accomplished whereas at Apple