Large Language Models often forget previous abilities when adapted for new functions. A novel self-distillation strategy seeks to minimize skill decay and streamline model administration.
A recently developed fine-tuning method endeavors to overcome “catastrophic forgetting,” a significant hurdle that frequently complicates successive model updates within corporate environments.
Experts from MIT, the Improbable AI Lab, and ETH Zurich have unveiled a fine-tuning approach engineered to enable models to master novel tasks without losing their established proficiencies.
To safeguard against the deterioration of current functionalities, numerous organizations opt to compartmentalize new tasks into distinct fine-tuned models or adapters. However, this segmentation elevates expenses and introduces additional governance challenges, compelling teams to consistently re-evaluate models to prevent performance degradation.
This novel technique, known as self-distillation fine-tuning (SDFT), is specifically engineered to mitigate this inherent compromise.
The researchers explained that SDFT “utilizes in-context learning by employing a demonstration-conditioned model to act as its own instructor, thereby producing on-policy training signals that retain existing functionalities while fostering the acquisition of new proficiencies.”
They further stated that SDFT consistently surpasses Supervised Fine Tuning (SFT) “in both skill development and knowledge assimilation tasks,” delivering superior accuracy for new tasks “while considerably diminishing catastrophic forgetting.”
Through their experiments, the researchers observed that this methodology allows models to progressively acquire new skills without compromising their performance on previously learned tasks, a feature that promises to simplify the ongoing updating and specialization of production models for businesses.
Addressing the challenge and proposing a remedy
Even with the swift progress in foundation models, the majority of enterprise AI systems typically remain unchanged after deployment. While prompting and retrieval can modify behavior during inference, the model’s core parameters do not evolve to incorporate fresh skills or information.
Consequently, every subsequent fine-tuning cycle introduces the danger of catastrophic forgetting, where improvements on a recently learned task lead to a decline in performance on prior tasks.
“For the advancement of future foundation models, it is crucial to overcome the challenge of continual learning: empowering AI systems to persistently learn and evolve, much like how humans amass knowledge and perfect abilities over their lifetimes,” the researchers pointed out.
Reinforcement learning presents an avenue for training using data derived from the model’s intrinsic policy, thereby mitigating forgetting. Nevertheless, this approach generally necessitates explicit reward functions, which can be difficult to define for all scenarios.
SDFT proposes an alternative strategy. Rather than deducing a reward function, it harnesses the model’s inherent in-context learning capability to produce on-policy learning signals directly from provided demonstrations.
Throughout the training process, the identical model assumes a dual function. A ‘teacher’ variant is trained using both the query and expert examples. Conversely, a ‘student’ variant is exposed solely to the query, mimicking actual deployment conditions. The student then adjusts its parameters to conform with the teacher’s predictions based on its self-generated outputs.
“Within sequential learning experiments, SDFT allows a singular model to acquire numerous skills incrementally without experiencing performance degradation, thereby solidifying on-policy distillation as a viable route for continuous learning from examples,” the researchers affirmed.
Obstacles to address
SDFT presents a highly pragmatic solution, as this approach eliminates the necessity of managing extensive “model zoos” comprising distinct adapters or fine-tuned versions, as noted by Lian Jye Su, a principal analyst at Omdia.
Nonetheless, its applicability in commercial deployment is yet to be fully determined, given that several significant challenges still persist.
For example, SDFT demands considerably more training duration and approximately 2.5 times the computational resources compared to conventional SFT. Furthermore, its efficacy relies on base models possessing robust in-context learning capabilities.
Sanchit Vir Gogia, the chief analyst at Greyhound Research, also cautioned that SDFT does not remove the requirement for regression testing frameworks. Since the model learns from its internally generated sequences, organizations must guarantee reproducibility via stringent version control and comprehensive artifact logging.
“While consolidation streamlines the number of models, it transfers operational complexity to the intricacy of governance,” Gogia stated.
According to Su, the expenditures could be balanced by preventing the catastrophic forgetting of crucial context and the necessity for intricate reward functions in reinforcement learning. However, its widespread adoption by enterprises might still be some time away. “SDFT is anticipated to be initially trialed in areas such as internal developer tools and general-purpose assistants, where the potential for a ‘self-generated error’ carries less risk than in highly regulated sectors like financial or medical decision-making,” remarked Faisal Kawoosa, founder and lead analyst at Techarc.
