To enhance the efficiency of last-mile delivery and reduce operational costs, this paper constructs a multi-drone cluster-based collaborative delivery model centered around truck docking points. With the objective of minimizing total operational costs, an adaptive large neighborhood search algorithm based on deep reinforcement learning (DRL-ALNS) is proposed for solution. This algorithm employs an Actor-Critic architecture, dynamically selecting a combination of destruction-repair operators based on the multi-dimensional state features of the current solution. The Solomon benchmark test demonstrates that DRL-ALNS achieves a mere 0.6% difference from the Gurobi exact solution on small and medium-sized problems, and reduces costs by 4.69% and 1.0% compared to the two-stage hybrid algorithm and the improved hybrid ant colony algorithm, respectively, on medium and large-sized problems. A case study in a district of Chengdu shows that joint delivery reduces costs by 26.85% and driving distance by 48.55% compared to pure truck delivery. Sensitivity analysis reveals that the impact of endurance radius on costs is more significant, with a 20km endurance radius and a load capacity of 15-20kg being an economically reasonable configuration. The research findings can provide decision-making references for logistics enterprises to optimize last-mile delivery routes.