Two-Stage Coarse to Fine Tuning: Optimizing Training in Deep Learning Architectures

Abstract

This paper presents a novel training approach designed to improve the efficiency and performance of deep learning models by employing a two-stage Coarse-to-Fine Tuning strategy integrated with the Simple Selective Freezing (SSF) technique.  Coarse training of deep learning architectures is performed first, and the coarse model is then refined by applying the selective freezing technique. The proposed approach is applicable to deep learning architectures obtained manually or by NAS methods. Experiments, conducted on the CIFAR-10 dataset using the deep learning architecture produced by DARTS, demonstrate that the proposed Coarse-to-Fine tuning-based training approach produces better accuracy with fewer parameters compared to the traditional training method. Additionally, the cross-dataset transferability of our approach has been evaluated by applying the learned cell from CIFAR-10 to CIFAR-100 and MNIST datasets, showcasing its robustness. The proposed approach also demonstrates efficacy across varying data percentages used for training, showing improvements in both complete and portions of datasets. Its improvements in scenarios with limited data are particularly notable, attributed to the two-stage Coarse-to-Fine Tuning strategy designed to enhance generalization in data-constrained environments. This adaptability stems from the fact that the generic structure learned by the first portion of the cells of the network during coarse training is frozen during fine-tuning. These extensive experiments indicate that the Coarse-to-Fine Tuning approach enhances accuracy while reducing computational efforts, and improves generalization, making it a versatile and efficient solution for training deep learning architectures with various datasets.