Table tennis is a globally popular sport that has developed rapidly and prosperously in recent years. Table tennis competitions have also become well-received large-scale sports events. Characterized by a fast rhythm, high technical requirements and excellent viewing value, table tennis is widely favored by the public. It not only occupies an important position in competitive sports events, but also serves as a common leisure activity for people of all age groups. With the widespread promotion of table tennis competitions across the world, there is an increasing demand for improving the viewing quality and technical level of matches. In actual competitions, restricted by the small size and high moving speed of the table tennis ball, artificial referees are prone to judgment deviations and inaccuracies. To solve the above problem, this paper proposes a table tennis motion tracking system based on machine vision. First, the imported table tennis match video is decomposed into sequential frames. Then, the RGB images after frame extraction are converted into HSV color space images. Combined with the orange color feature of the table tennis ball, irrelevant color interference information in the background is filtered out to obtain a single-target image containing only the table tennis ball. Next, hierarchical calculation is conducted on the processed images to acquire the center position of the ball and realize preliminary motion tracking. To optimize tracking precision, the Kalman filter algorithm is adopted to predict the subsequent movement position of the ball based on continuous coordinate data. Practical test results demonstrate that the system can stably track the moving table tennis ball and display the tracking and prediction results in the video in real time.