Aesop: A Visual Storytelling Platform for Conversational AI and Commonsense Grounding

Abstract

We present a new collaborative visual storytelling platform, Aesop, for direction and animation. Our system operates in two main modes, common sense grounding (annotation) and conversation. The Aesop system senses the human state and input using a natural language parser and human gesture monitoring for natural interactions. The interface consists of a 3D animation software and a web controller to interact with the internal state of the system. For knowledge representation, we formulate novel knowledge graphs which enable spatio-temporal event representation. Aesop thus enables 3D spatial and temporal reasoning which are both essential for storytelling. Finally, the system utilizes a dialog manager to track the conversation state and manage goals. Aesop provides a rich platform that enables research in language, gestures, vision, and planning in the context of storytelling.