Line-level software defect prediction (LLDP) is crucial for locating and modifying defective code and has drawn increasing attention from both academic and industrial communities. Recently, a series of deep learning-based LLDP models have been proposed. However, there is no systematic comparison of these LLDP models. In this paper, we aim to compare the performance of these models, evaluate their consistency in different scenarios, and investigate whether fusing handcrafted features can improve the prediction performance. To this end, we conducted a comprehensive empirical study on eight recent state-of-the-art models using 32 benchmark datasets from nine software projects’ releases. We evaluated them in both cross-version and cross-project scenarios with four widely used performance metrics, including AUC, Recall@Top20%LOC, Effort@Top20%Recall, and IFA. Moreover, we investigated the impact of handcrafted features on LLDP models based on two different fusion strategies. The results show that 1) the difference among these models is statistically significant; 2) no model is always superior across all performance metrics but Bugsplorer generally performs best except IFA; 3) model performance rankings are inconsistent between cross-version and cross-project scenarios, revealing that a model’s effectiveness is highly scenario-dependent; 4) furthermore, fusing handcrafted features significantly improves model prediction performance, and the fusion strategy also matters. In conclusion, the selection of an LLDP model should be guided by specific scenarios and performance metrics, and the hybrid model that combines deep learning representations with handcrafted features is a promising alternative for LLDP.