Anthropomorphic AI: A Toolkit for Authoring and Interacting with Intelligent Virtual Agents for Extended Reality

Intelligent Virtual Agents (IVAs), which embody an artificial intelligence (AI) in a humanoid representation, have enormous potential for immersive extended reality (XR) environments to enable natural and engaging human-AI interactions. With recent advances in large language models (LLMs) in simulating human-like text responses, interest in anthropomorphic embodied IVAs has grown across XR research and application domains. However, toolkits for authoring and interacting with IVAs in research remain sparse.

Therefore, we present Anthropomorphic AI, a flexible and scalable open-source research toolkit for authoring and interacting with embodied IVAs with rich multimodal capabilities, including speech, gaze, gestures, facial expressions, and vision. Our system enables developers to create various embodied anthropomorphic IVAs by customizing behavior through expressive nonverbal cues, selecting and combining different foundation models, speech-to-text (STT) and text-to-speech (TTS) methods, and adapting the system prompt to guide interaction. We also integrate various features such as proximity detection, trajectory-based action recognition, and vision-based multimodal prompting for supporting natural human-IVA interaction in immersive XR. We evaluate the toolkit through four use case demonstrations, a pilot developer evaluation, and an end-user study in immersive VR, showing its capability in generating socially engaging, highly usable, and likable anthropomorphic IVAs for immersive XR applications.