Zyphra拟合持续训练LLM可塑性损失缩放定律

Zyphra fits a scaling law for plasticity loss in continuously trained LLMs. What can we do to push the point of rigidity onset towards infinity? I recall Sutton's team could only come up with continual backpropagation (random reinitialization of some units)… suboptimal.