Computer-assisted surgery has occupied an important position in modern surgery, further stimulating the progress of methodology and technology. In recent years, a large number of computer vision-based methods have been widely used in surgical workflow recognition tasks. For training this method, a lot of annotated data are necessary. However, the annotation of surgical data requires expert knowledge and thus becomes difficult and time-consuming. In this paper, we focus on the problem of data deficiency and propose a knowledge transfer learning method to compensate a small amount of labeled training data. To solve this problem, we propose an unsupervised method for pre-training a Convolutional De-Convolutional (CDC) network for sequencing surgical workflow frames, which performs convolution in space (for semantic abstraction) and de-convolution in time (for frame level resolution) simultaneously. Specifically, through transfer learning, we only fine-tuned the Convolutional De-Convolutional network to classify the surgical phase. We performed some experiments for validating the model, and it showed that the proposed model can effectively extract the surgical feature and determine the surgical phase. The accuracy, recall, precision of our model can reach 91.4%,78.9%,82.5% separately.