Agent Outline

Project description

The project entails a modular AI agent used for communication with clients. This communication can take the form of text, audio or video chats. The user gets to choose which type of conversation they want and they can choose to change whenever they see fit.

The system will dynamically create a new system prompt to feed the AI in order to ask the client/user certain questions needed by the company. These are all stored in a database and will be fetched and itterated through depending on what data the company needs of the client.

Context

The project is regarding improvements of business to business communication, by improving the communication pipeline more time can be saved on both ends of the conversation. Through the use of an LLM powered chatbot we achieve this goal.

Results

The result of our project is a fully deployed AI-powered chatbot with text, audio, and video capabilities that can interview clients in an interactive and engaging way. The application combines a modern web frontend, a Python backend, AI models, voice processing, avatar animation, authentication, and database functionality into one complete product.

The frontend was built with React and written in TypeScript, providing a responsive and user-friendly interface. The backend was developed in Python and handles the application logic, communication between services, and integration with the AI components.

We deployed three AI models on RunPod to support the main intelligent features of the application: a large language model for generating text responses, a text-to-speech model for converting generated text into audio, and a speech-to-text model for transcribing user speech into text. In addition, we integrated Rhubarb as a mouth cue generation system for the avatar, allowing the avatar’s mouth movements to be synchronized with the generated speech.

The frontend is deployed on Vercel, while the backend is deployed on Render. The mouth cue generation service is also hosted together with the backend on Render. The databases and user login/authentication system are managed through Supabase.

Overall, the project resulted in a complete end-to-end application that functions as an AI interviewer chatbot, capable of communicating with clients through text, audio, and video while using an animated avatar for a more natural interaction.

About the project group

Our group is composed of people originating from separate parts of the Fontys IT education, some originating from game design or cyber security, while others originate from the AI specializations. All of our expertise is combined into this project.