Accurate identification of abnormal driving behavior is very important to improve driver safety. Aiming at the problem that threshold or traditional machine learning methods are mostly used in existing studies, it is difficult to accurately identify abnormal driving behavior of vehicles, a method of abnormal driving behavior recognition based on smartphone sensor data and convolutional neural network (CNN) combined with long and short-term memory (LSTM) was proposed. Smartphone sensors are used to collect vehicle driving data, and data sets of various driving behaviors are constructed by preprocessing the data. A recognition model based on a convolutional neural network combined with a long short-term memory network was constructed to extract depth features from data sets and recognize abnormal driving behaviors. The test results show that the accuracy of the model based on CNN-LSTM can reach 95.22%, and the performance indexes can reach more than 94%. Compared with the recognition model constructed only by CNN or LSTM, this model has higher recognition accuracy.