This article is based on data mining research scene data mining in the key data extraction of physical education videos. Combined with the current status of physical education, the application value of scene data mining technology to sports events and physical education is analyzed. The results show that data mining enhances the probability of victory in football matches, basketball matches, and track and field matches, and data mining improves effective frame extraction in sports teaching videos. Data mining is a technology that quickly processes data and information, which improves the effective rate of teaching videos by 7.6%. It is widely used in 98.6% of physical education. The use of data mining in physical education can promote the development of physical education in colleges and universities.