Emergent large language model (LLM)-based digital avatars are poised to have broad applications across domains. However, LLMs, together with auxiliary services such as text-to-speech, require processing time and can generate undesirable conversational response delays. There is still limited understanding of how such delays affect human-avatar interaction, particularly the point at which a delay changes from acceptable to unacceptable. In this paper, we first conduct a preliminary study in virtual reality (N = 30) to investigate the effects of response delays ranging from 0 to 16000 ms. The results show that delays between 0 and 2000 ms are generally acceptable, with around 1000 ms perceived as the most natural. Noticeable degradation begins by 4000 ms and becomes pronounced at 8000 ms. Based on these findings, we implemented a turn-taking scheme that uses a lightweight LLM to generate contextualized conversational fillers within 1000 ms. We then compare this approach with commonly used canned lexical and non-lexical fillers in a user study (N = 63). Canned non-lexical fillers provide no improvement and can worsen the experience, while canned lexical fillers offer only limited benefits. In contrast, contextualized fillers more effectively preserve conversational realism, social context, interactivity, anthropomorphism, and animacy, while reducing discomfort. These findings show that faster responses are not always better and that filler effectiveness depends on contextual relevance. This work provides design guidance for more natural and socially appropriate digital avatars and conversational user interfaces.