What each modern chatbot development team needs to know

As someone who has worked in Conversational AI for the past six years, I've seen the industry's transformation from traditional NLP and entity-based solutions to GPT-powered AI Agents.

In this post, I collected different topics that the modern chatbot development team needs to cover. For some topics, I also tried to share resources and links where you can learn more.

I think this guide will be useful both for Conversational AI agencies trying to improve their work, as well as internal chatbot development teams.

I split all skills into two categories, non-technical expertise and technical expertise, however, it's important to say that those often overlap and even so called "non-technical skills" require a certain level of technical knowledge.

Let's get started!

✨ Non-technical expertise

1. User research for AI Assistants: What do your users actually need?

Every Conversational AI project, be it AI Assistant or Voice Assistant, should start with user research.

Understanding your users' needs, pain points, and interaction preferences is crucial to creating an effective AI assistant.

You need to know how to analyse the behaviour of your potential users to get insights into how you need to design your product. User research techniques can include:

• User interviews: Conducting one-on-one interviews with potential users to understand their needs, pain points, and expectations from an AI Assistant.

• Surveys & questionnaires: Collecting quantitative and qualitative feedback from a larger audience to identify common patterns in user preferences. This will later define your conversation design.

• Conversation log analysis: Observing users interacting with chatbots in real-life scenarios to gather insights on usability and pain points, helping you improve your product further.

• Wizard of Oz testing: Simulating chatbot responses with a human operator before developing an AI-driven system to gather insights on user expectations.

• User personas: Developing detailed profiles of target users, including demographics, goals, challenges, and behavioural patterns.

Resources:

• Check mentors on ADPList with experience in UX & Conversation design. Here are great tips on how to find a mentor: https://medium.com/design-bootcamp/how-to-find-and-become-a-conversation-design-mentor-866bfd44a322

• Check CDI courses: https://www.conversationdesigninstitute.com/courses

2. Process mapping: How are things currently done?

Before implementing AI solutions, teams need to document and understand the current processes they aim to automate. This ensures that AI is improving workflows rather than blindly replacing them.

Key steps in process mapping:

• Identify key workflows: What tasks are being performed, and who is responsible for them?

• Create a visual representation: Use tools like Miro or draw.io to map out the workflow.

• Highlight inefficiencies: Identify bottlenecks, pain points, and opportunities for automation.

• Use the process map as a baseline: This ensures AI development is guided by real needs rather than assumptions.

Process mapping is an essential skill for chatbot developers, as it helps determine what to automate and how automation can provide real value.

3. Conversation analysis: How do real humans do that? And what does having a good conversation mean?

To design human-like AI conversations, teams must analyse real-world conversations to understand natural dialogue flow, tone, and best practices.

Key areas of analysis:

• Customer support interactions: Understanding how agents handle queries, objections, and escalations.

• Sales conversations: Identifying successful persuasion techniques.

• Production chat logs: Learning from real user interactions rather than relying on scripted examples.

From these conversations, teams must extract playbooks and conversation rules, which define what makes a conversation feel natural and engaging.

4. Expertise mapping and knowledge transformation: How to convert expert knowledge into the format that AI can train on?

AI systems need structured expert knowledge to function effectively. The challenge is converting expert insights into a format AI can train on.

Step 1: Extracting expert knowledge

• Conduct structured interviews with domain experts (e.g., therapists, customer support leads, doctors).

• Ask the right questions: How do they structure interactions? What are key decision points?

• Gather training materials (e.g., onboarding guides, playbooks, FAQs).

Step 2: Transforming knowledge for AI

• Convert insights into structured training data.

• Create fine-tuning datasets or documents for Retrieval-Augmented Generation (RAG).

• Develop effective AI prompts based on expert inputs.

This skill ensures that AI assistants operate with domain-specific expertise and accuracy.

5. Knowledge management: How to maintain always changing data after the release?

The landscape of AI knowledge management has changed a lot with now that we are dealing with LLMs and RAG systems. Unlike traditional chatbots with static responses, modern AI assistants deal with big amounts of dynamic, raw data from multiple sources. This creates new challenges for teams.

Modern chatbot development teams need to not only know how to prepare data for training, described in step 3, but also how to maintain the data after the release. Skills that became important now:

Key considerations:

• Setting up structured processes for regular data verification and updates.

• Defining responsibilities within the team, who reviews and updates the data?

• Understanding data change frequency, which pieces of information evolve more frequently?

• Maintaining consistency across multiple data sources.

• Resolving contradictions to prevent misinformation.

• Building feedback loops for expert validation and continuous improvement.

• Implementing hierarchy in data sources, ensuring AI references the most reliable, up-to-date information first.

A robust knowledge management system ensures long-term AI reliability and accuracy, keeping responses aligned with the latest business changes.

Working with data in the modern chatbot development teams also means having the skills to organise ongoing knowledge management. This includes

• Creating sustainable processes for regular knowledge updates

• Establishing documentation standards

• Building systems for continuous expert feedback

• Maintaining version control of knowledge bases

6. Conversation design: How will AI conversation look like?

This crucial skill determines how your AI assistant communicates. The Conversation Design Institute (CDI) offers comprehensive courses in this area. I personally followed the one from Conversation Design Academy and can really recommend this one.

Here is a great overview of other conversation design courses: https://uxplanet.org/the-ultimate-conversation-design-course-guide-3581e66d23e7

Resources from Google: https://developers.google.com/assistant/conversation-design/welcome

✨ Technical expertise

1. Chatbot development technologies: What would be the best tool for the job?

An in-depth hands-on overview of existing technologies is important when you are just starting your project. Always keep an eye on what's out there or hire a consultant to worked with multiple tools to give you an honest recommendation.

Teams should be familiar with various platforms and tools:

• Rasa

• DialogFlow

• VoiceFlow

• Landbot

• ManyChat

If your team is building for voice technology:

• Vapi

• Bland AI

• Air AI

• LiveKit

• Retell AI

2. How to prepare training data?

Data preparation is crucial for AI performance. Teams should know how to:

• Gather data efficiently: Sources include web crawling, paid datasets, and custom data logging.

• Structure data correctly: Organise it in formats suitable for training, such as JSON, CSV, or embeddings.

• Use metadata effectively: Tagging data with attributes like timestamps, sources, relevance scores, and categories.

• Identify high-quality data: Filtering noise, removing duplicates, and ensuring data accuracy.

• Handle data versioning: Keeping track of changes and updates to prevent inconsistencies.

Good training data is the foundation of a performant AI model. Teams must ensure their data is accurate, well-structured, and continuously updated.

3. NLU & Entity detection:

While most of the modern AI Assistants use LLMs, it's important to know concepts such as intent detection and entity detection and know when to use old school ML techniques which work well&fast&cheap and when to use LLMs.

4. Prompt engineering: How to write a prompt so that AI follows your instructions?

Essential for effective AI interactions:

• Prompt engineering guide for chatbots

• Function calling documentation

5. Vector databases & Advanced RAG

Teams need to know which vector databases are available, what are their strong/weak points and which to use when.

Resources for understanding advanced retrieval:

• Advanced RAG techniques overview

• Video guide on RAG

6. Fine-tuning: When to use fine-tuning and when to use RAG?

Understanding when to fine-tune a model versus when to rely on retrieval-based augmentation is critical.

• Fine-tuning guide

• RAG vs. Fine-tuning comparison

• IBM's perspective on fine-tuning

• Microsoft's fine-tuning considerations

7. LLM hallucination reduction

Understanding and mitigating false outputs:

• Comprehensive guide to LLM hallucinations

8. Evaluation methods

Teams should implement both quantitative metrics and qualitative analysis to test AI models effectively.

• Guide to LLM evaluation metrics

• Experimental environments for prompt testing

Resources for testing AI assistants:

• LLM evaluation setup and metrics

• Software testing approaches for LLMs

• Evaluating LLM-based applications

• LLM system optimization

Evaluation concepts:

• LLM as a judge

• Libraries like Deep Eval & RAGAS, calculating hallucination scores

• How to set up manual evaluations

• Experimental environments to play around with your prompt

• Creating simulators

Both technical metrics, but also qualitative analysis is important

As well as knowing which metrics to track.

9. Ethical considerations

Ethics play a crucial role in AI development. Developers should be familiar with:

• Bias mitigation strategies

• Privacy best practices

Resources:

• Ethical Explorer toolkit

• OpenAI Model Spec document

• Claude's Constitution

Conclusion

Building effective chatbots requires a diverse set of skills in both technical and non-technical domains. Teams need to continuously update their knowledge as the field evolves rapidly. While this guide covers the core competencies needed for chatbot development, note that voice interaction introduces additional complexity and requires its own technical stack.

Feel free to reach out if you'd like to learn more about voice-specific development considerations.