AI-Powered Assistant Development – AI Hotel Concierge
Highlights
- Agentic AI pipeline setup
- Voice interaction and AI avatar communication
- Vector database storage for RAG
- Integration with Heygen via native APIs
- LLM integration for AI agent training
- AI-assisted engineering with Claude
Client
The client is a US startup that approached SoftTeco with an idea to create an AI-powered virtual assistant for Cisco Webex Desk Pro devices, commonly used in hotels, airports, and similar environments. The client wanted the assistant to communicate with users via audio and video formats, thus enhancing user experience and personalization. Since the client needed strong expertise in AI development, they selected SoftTeco as a recognized AI vendor.
Challenge
The product needed to combine several independent technologies within a single conversational experience: speech recognition, an AI avatar, LLM-based reasoning, RAG, ERP connectivity, and payment processing. The main challenge was therefore not only to implement an AI assistant, but to coordinate multiple third-party services with different APIs, data formats, latency patterns, and failure scenarios while maintaining a natural experience for the user. Since HeyGen provided the avatar-facing part of the solution, SoftTeco was responsible primarily for the backend, AI orchestration, integrations and cloud architecture.

Ready to bring your AI vision to life?
Let’s talk about how SoftTeco can help you transform it into a high-quality, market-ready solution.
Solution
SoftTeco developed the backend for an AI-powered concierge that transforms Cisco Webex Desk Pro into an intelligent customer-service terminal. When a user speaks to the device, Whisper converts the audio into text and passes the request to an agentic AI pipeline. The agent interprets the request, uses available tools and the RAG knowledge base when necessary, and generates an appropriate response. The text is then sent to HeyGen, which converts it into audio and video and presents it through the AI avatar.
The solution can also retrieve current services and pricing from an ERP system. When the interaction involves a purchasable service, payment can be initiated through an integrated Stripe Terminal. The backend was deployed on AWS and designed around independently scalable services. Because the system depended on several external APIs and AI services, engineering work involved continuous analysis of API contracts, response structures, integration edge cases, and failure conditions.
SoftTeco engineers used Claude throughout the project to analyze technical documentation, work through third-party integration requirements, refine implementation logic, generate test scenarios, and review code changes. Its use was integrated into the existing engineering workflow, with engineers validating the output and retaining responsibility for architecture, security, technical decisions, and final code ownership.
Tech Stack
AI & RAG
OpenAI
Whisper
Claude
How it works
We started working on the client’s idea by analyzing and estimating possible solutions and available providers for integration. As a result, we selected Heygen as a provider of AI avatars to be integrated with the Cisco Webex Desk Pro and our backend. The integration with Heygen was performed via native Heygen APIs.
Since Heygen delivered the frontend part of the solution, we were responsible for the backend development. We used Agentic AI as the base for the backend and selected Vector database as the RAG storage of choice. To connect frontend and backend, we set up an Agentic AI pipeline which interprets user queries and uses ReAct agents and RAG approach to return requests in the needed format.

The pipeline operation starts with the user query in the audio format being received by Whisper, an OpenAI transcriber that converts speech into text. The transcribed text is then sent to our pipeline and is processed by the ReAct agent with Tools. The response to the query is generated in the text format and is sent to Heygen, which transforms the text into audio and video and presents it to a user via the AI avatar. To ensure the answers are relevant and accurate, we used LLMs for model training.
In addition to voice recognition and customer service, the solution also has a card reader integration with Stripe Terminal WisePOSE and an ERP integration to retrieve services, pricing, and record completed orders. Thus, users are able to instantly pay for specific services, like room booking or purchasing an additional service. The solution is deployed on AWS for better scalability and availability and is based on the microservices architecture to ensure independent and rapid scaling.
Looking for a trusted AI development partner?
Our engineers specialize in complex integrations, real-time analytics and scalable architectures for enterprise solutions.
Results
SoftTeco delivered a market-ready AI concierge combining conversational self-service, relevant information, and transactional functionality in a single terminal experience. AI also supported the engineering process: SoftTeco engineers used Claude for implementation, integration analysis, testing, and code review across a complex backend. The resulting architecture provides a scalable foundation for new services, integrations, and AI-powered scenarios.

