Ship object detection is the core part of the maritime intelligent ship safety assistance technology, which plays a crucial role in ship safety. The object detection algorithm based on the convolutional neural network has greatly improved the accuracy and speed of object detection, which YOLO algorithm stands out among the object detection algorithms with more excellent robustness, detection accuracy, and real-time performance. Based on the YOLO v4 algorithm, this study uses the k-means algorithm to improve clustering at the input side of image data and introduces relevant berth data in the self-organized dataset to achieve detection of ships and berths for the lack of detection of berths in the existing ship detection algorithm. The experimental results show that the mAP and F1-score of the improved YOLO v4 are increased by 2.79% and 0.80%, respectively. The improved YOLO v4 algorithm effectively improves the accuracy of ship object detection, and the in-port berth also achieves better detection results and improves the ship environment perception, which is important in assisting berthing and unberthing.