We used the Coordinate Response Measure (CRM) procedure (Bolia et al, 2000) to examine whether a non‐acoustic characteristic of speech ‐voice familiarity‐ can affect a listener's ability to separate competing voices. We tested 27 listeners, aged 45‐79, with their spouse's and two novel voices (other listeners' spouses). Couples had been living together at least 18 years. On each trial, two different talkers produced two of four callsigns (one being the target 'Baron'), two of four colours and two of eight numbers, and the participant responded by indicating the colour‐number combination to which 'Baron' was told to go. We tested three conditions: spouse‐voice target phrase with a novel‐voice masker phrase (F/N); novel‐voice target with spouse‐voice masker (N/F); novel‐voice target with (different) novel‐voice masker (N/N). Compared to N/N, performance was significantly better in F/N, and, crucially, N/F conditions, indicating that listeners can exploit learned characteristics of a masking voice to help them track a novel voice. Furthermore, whereas the younger participants (under 60) benefited from having a familiar voice as target or as masker, the older group (over 60) benefited only when the familiar voice was the focus of attention, suggesting that the ability to use voice familiarity to segregate sounds changes with age.