This study focuses on addressing the technical bottlenecks of traditional manual inspection methods for old residential building defects, such as low efficiency, strong subjectivity, and high missed detection rates. Leveraging the advancement of artificial intelligence and computer vision technologies, we propose an intelligent detection approach based on the improved YOLOv11-seg model to identify common building surface defects (cracks, spalling, and algae).First, an independent dataset was constructed, containing 11,702 color images (640×640 pixels) with detailed manual annotations, which were randomly divided into training, validation, and test sets at appropriate ratios to ensure the reliability of model training and evaluation. Second, targeting the challenges of multi-category, multi-scale defects and complex backgrounds in old buildings, two optimization modules—DCA (Deformable Convolution and Channel Attention fusion structure) and DS_C3K2 (Backbone optimization structure integrating Dynamic Convolution and SE Attention)—were introduced into the original YOLOv11-seg architecture. These modules enhance the model’s ability to perceive spatial features and express channel selectivity, thereby improving the accuracy of defect contour extraction and reducing missed and false detections. Experimental results show that the improved YOLOv11-seg model outperforms YOLOv5-seg and YOLOv8-seg in key metrics: in bounding box detection, its mAP50 reaches 0.845 and precision is 0.872; in mask segmentation, its mAP50 is 0.812 and recall is 0.747. Additionally, an integrated visual operation interface was developed to support functions such as image upload, automatic detection, segmentation visualization, and report generation, reducing the technical threshold for engineering application. This research provides an efficient and reliable technical solution for the quality detection of old residential buildings, supporting urban renewal and public safety governance.