Under the background of the era of big data, the physical education teaching evaluation system of colleges and universities has more characteristics and manifestations of the new era. At the same time, big data also brings more requirements for the improvement of the physical education teaching evaluation system of colleges and universities. Under such an environment and trend, colleges and universities need to make certain reform measures to meet the challenges calmly according to the requirements of the new era. This paper makes a systematic study on the construction of the evaluation system of physical education teaching in colleges and universities, expounds the theoretical basis of building an efficient evaluation system of physical education teaching, analyzes the components of the evaluation system, and gives the construction path of the evaluation system of physical education teaching in colleges and universities. The purpose of this paper is to provide some reference information for physical education in higher education and the cultivation of talents' comprehensive quality.