Aiming at the problem of broadband stability of voltage source converter based high voltage direct current (VSC-HVDC) system, a control strategy of converter is designed to improve its small signal stability. Based on the modeling of the flexible HVDC system, the overall structure of the internal and external loop control of the VSC-HVDC converter is established. DC control and AC control damping controller are designed respectively in phase locked loop and PI control. The state space equation of the control system is established to analyze its performance and design its parameters reasonably. The stability of small signal of the system is improved effectively through optimization of several links.