Thermite welding is widely used to join rails to get a continuous welded rail. The HAZ formed on two sides of the thermite welded joint. The soft zones in the HAZ of thermite weld are weaknesses that must be strengthened for the rails to give good service. A groove was machined on the running surface of the rail in the HAZ of the thermite weld, and it was filled using the FCAW process to improve the microstructures and hardnesses of the soft region, without adversely affecting the thermite welded rail. In this paper, the improvements of FCAW over the original HAZ of thermite weld determined through the differences in the microstructures and hardnesses are discussed. An optical microscope and a field emission scanning electron microscope (FESEM) were used to characterize the microstructures and an atomic force microscope (AFM) was used to analyze the interlamellar spacing of pearlite. The micro-Vickers hardness profiles are also compared between the original HAZ zone and that obtained after improvements obtained by FCAW. The spheroidized microstructures of the original HAZ were changed to very fine pearlite microstructures by the FCAW process. The hardness of the soft zone was approximately 250 HV, whereas the value of the improved zone was approximately 330 HV. The conclusion is clear that the FCAW process can effectively improve the microstructures and hardnesses of the soft regions found in the HAZ of conventional thermite welds.