Farms in many countries use fenced pastures—known as paddocks—for keeping livestock. These farms usually spread over many hectares with paddocks of various shapes and sizes. Farm managers face the difficulty of counting their animals on regular basis and recently the use of an unmanned aerial vehicle (UAV) has been proposed for counting livestock. To get the data from only a respective paddock, the fence lines must also be detected correctly in the aerial images and videos. This will help to keep the data from each paddock separate. A unique system based on a combination of deep learning and machine learning is proposed to accurately detect the fence lines in the videos recorded by a UAV at an altitude of 80 m.