Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask

This paper runs a series of experimental ablation studies to better understand the limits of the Lottery Ticket Hypothesis, and investigate variants of the initial pruning and masking procedure under which its effects are more and less pronounced. It is first and foremost a list of interesting results, without any central theory tying them together. These results include the observation that keeping pruned weights the same sign as their "lottery ticket" initialization seems more important than keeping their exact initial magnitudes, that taking a mixed strategy of zeroing pruned weights or freezing them at initialization can get better results, and that applying a learned 0/1 mask to a re-initialized network can get surprisingly high accuracy even without re-training.

While it certainly would have been exciting to have a paper presenting a unified (and empirically supported) theoretical understanding of the LTH, I respect the fact that this is such a purely empirical work, that tries to do one thing - designing and running clean, clear experiments - and does it well, without trying to construct explanations just for the sake of having them. We still have a ways to go in understanding the optimization dynamics underlying lottery tickets, but these seem like important and valuable data points on the road to that understanding.