Best practices for chatbot development

Chatbot development is an complex process that requires the symbiosis of multiple disciplines - from user experience and conversation design to natural language processing and data analytics - in order to create a smooth and user friendly conversational interfaces. With so many components to consider, it's easy for chatbot projects to go off track if the foundational principles of good chatbot design are not established from the start.

After building chatbots for clients over the last 6 years, I've observed differences between successful and unsuccessful projects. In this blog post, I want to share 12 key principles that I've found are crucial for developing successful chatbots that solve real problems for businesses while meeting user needs.

Whether you're building your first chatbot or improving an existing one, I hope this overview will help you reflect on your current chatbot development processes and get inspiration on what you can do differently.

Let’s get started!

Table of content

1. Principle #1: Think of chatbots as tools for solving problems
2. Principle #2: Do your research
3. Principle #3: First design, then develop
4. Principle #4: Use conversation design principles
5. Principle #5: Involve industry experts in conversation design process
6. Principle #6: Know how to effectively train AI models
7. Principle #7: Release your first version ASAP, get feedback and iterate
8. Principle #8: Log all conversations and regularly analyse them
9. Principle #9: Automate testing
10. Principle #10: Track key performance metrics
11. Principle #11: Collect user feedback after the release
12. Principle #12: Stay updated on latest advancements
13. Final words

📌 Principle #1: Think of chatbots as tools for solving problems

Chatbots are just a tool for solving problems. Sometimes it’s an efficient way to solve your problem and sometimes not. And to know if a chatbot is a good solution for your problem, you need to clearly articulate your problem and be very explicit about the business goals and user needs your chatbot needs to address.

Be intentional about the problem you are trying to solve, don’t just say “I want a chatbot”. Ask yourself:

• What is the specific problem I am trying to solve? Is it a low lead conversion rate? A slow response rate from customer support agents? Or something else?

• What do I know about the problem? Try to quanityfy the problem as best as you can and understand how big the problem actually is. For example, if you are not getting enough leads through your website, it's good to check, what is your current website traffic? What is your current lead conversion rate? If you are unhappy with your customer support, check how many support tickets do you get daily? How much money/time does poor customer satisfaction cost you?

• Can a chatbot solve this problem? Or are there more effective ways to solve the problem? For example, if you want to increase lead generation, and there aren’t that many people visiting your website at the first place, having a chatbot won’t help you generate more leads and a more effective solution to your problem might be to focus on your SEO strategy first.

• How do I design my chatbot solution in such a way that it solves my problem? Which type of chatbot is most suitable? Which tech stack should I use to solve my problem effectively?

• How will I track progress on my business goals? How will I measure the success of the project? How do I know the chatbot is solving my problem? Will I be satisfied when the chatbot reduces work load on our customer support by X%? Or generate 10X leads? Or something else?

🎯 Why this matters

Being clear on the problem you are trying to solve will inform your entire approach, including project strategy, technology choices, conversation design, and more. When you clealy understand the problem you are trying to solve, you can:

• Better evaluate if a chatbot might actually help solve it.

• If so, shape the solution, tech stack, and conversation flow based on the specifics of the problem.

• Establish metrics to track whether the project succeeds.

❗Continuously re-examine the problem

It's important to not only clearly define the problem during first stages of chatbot development process but also keep referring back to these problem statements throughout your whole development process. The problem defenitions should be in the core of all the design and development decisions you make.

📌 Principle #2: Do your research

Now, after you defined the problem, you need to analyse how things are currently done and who you will be dealing with. Doing research upfront is key to designing a great chatbot and it will ensure your chatbot solves real problems for real users.

💡 Understand the current processes

Understand the processes you are trying to automate/improve, learn from what has already proven to work, and identify areas for improvement:

• Document the current processes: talk to people in charge, document every step of their current processes and understand the intricacies of how things currently operate. Use tools like draw.io or miro boards to do that.

• Learn from what works and what doesn't: Study both successful and unsuccessful attempts at process automation or improvement. Understanding past successes and failures in your industry can provide valuable insights and help you avoid common pitfalls.

• Analyze existing conversation logs, call transcripts, support tickets, forum posts to see common questions, complaints, behaviors: What are most frequently asked questions based on the data you have? Which of those questions require the most investment from your team? Which of the tasks can you potentially automate with the chatbot?

• See where improvements can be made: Once you have a clear picture of the existing processes, pinpoint areas where improvements can be made.

💡 Get to know your users

It is also important to know your users. If you want to design a successfull chatbot, users should always be the center of your design process:

• Conduct user interviews and surveys: Understand your users' goals, motivations, pain points, and preferences. The insights gained from these interviews will guide your chatbot development strategy.

• Understand the demographics of your users: Gather insights about your users' age, language preferences, and more. You will design your chatbot differently for teenagers as a target group v.s. older people.

• Identify user personas that represent your key user segments: people have different communication styles and different ways to solve problems. Understanding your user personas will help you design your conversational experience to suit different types of users.

🎯 Why this matters

• Documenting current processes will help you later with designing your conversation flows.

• Analysing conversation logs will inform the FAQs you will want to include in your conversation design.

• Knowing your user needs will help you prioritise features and focus on functionalities that directly address the main pain points.

The more deeply you understand current processes and your target users, the better your chatbot will fit into the real world. Do your research and design your chatbot based on real-world data rather than assumptions.

📌 Principle #3: First design, then develop

With no-code platforms that make it easy to prototype, it's very tempting to jump right into implementing a chatbot, but slow down and design first.

When you're creating a chatbot, the decisions you make during development, like the platform you choose and the integrations you include, depend heavily on the design. Your design sets the foundation for how your chatbot will work, so it's crucial to wait until you have a complete design before figuring out how to implement everything.

Think of it this way: a solid design is like a blueprint for a house. You wouldn't start building without a clear plan, right? The same goes for chatbot development. If you start implementing before your design is done, you might end up making mistakes not seeing the whole picture and redoing a lot of work later on because things weren't thought through properly.

So, always wait for designs to be finished before starting the implementation. Use tools like draw.io or miro first, get feedback from your team, itterate and only then start developing.

🎯 Why this matters

• Easier & faster development process: It's much easier to iterate on design documents compared to revising your entire chatbot after you've already started building it based on incomplete or unclear design.

• Smarter choices: The decisions you make during implementation – like which platform to use and how to put everything together – all depend on your design. If you've thought things through properly in the design phase, your choices will be more informed and your chatbot will work better.

❗Design first for every new feature you introduce

Remember, this design-first approach isn't just for the beginning. Each time you add something new to your chatbot, take time to make a design first.

💡 Key lesson

Wait until you have a complete design before you start building. You'll avoid problems later and make better choices. It's like planning your journey before you hit the road – it helps you get where you want to go without getting lost.

📌 Principle #4: Use conversation design principles

No smart technical implementation can make up for a poor user experience. And to design experiences that people will enjoy you need to master the art of conversation design. This involves creating chatbot persona, defining the tone of voice, writing the conversational copy that matches your brand identity and knowing how to design conversations in a way that hels the user navigate through your chatbot features in the most natural and efficient way possible. This is a lot of work, and it's important to do it right for the success of the project.

💡 Improve your conversation design

There are a few things you can try to improve your conversation design:

• Hire an experienced conversation designer (if possible) to design conversations flows. Conversation design is a skill and this expertise is hard to develop internally. Having conversation designer as part of your team will help you design conversational experiences that people will enjoy.

• Get a consultation from a conversation designer and ask for feedback on the conversation flows you designed (if you can't hire them). Having an experienced designer look at your conversation flows and give you suggestions on what to improve will already help you improve user experience drastically.

• Learn conversation design principles yourself. Take a course from conversation design institute, watch relevant YouTube videos or read a few books.

❗Don't underestimate the role of conversation design

It might look like conversational design is something not important that you can save money on and do yourself, but in reality good conversation design is one of the most important parts of a successful chatbot project. Great conversation design makes user experience more enjoyable and improves user satisfaction. No matter how advanced your implementation is, first thing that people will notice is conversation design so take it seriously.

📌 Principle #5: Involve industry experts in conversation design process

When designing a chatbot for an industry where you don't have much experience, it's useful to involve industry experts in making conversation designs and writing conversational copy.

Have industry expert review your design decisions, get their feedback on your conversational copy. If you are building a customer support chatbot, talk to customer support agents from your team, let them test your demo, ask them what they think of your conversation flows. If you are building a mental healthcare chatbot, talk to psychoterapists, ask them to review your conversational flow and give feedback about the content.

💡 Ways to bring in expertise

• Interview people who work in the industry and ask their feedback on your current conversational flows. Start with just a short call if you are not able to make them part of your team.

• Include industry advisors in your project team. Involve them in discussions, ask their input on your ideas, and have them review your conversation flows.

🎯 Why this matters

• Asking industry experts review your conversation design will help you get feedback early on and adjust your implementation accordingly. Their insights might help you save time along the road.

❗Get feedback & stay open-minded

Consulting industry experts provides an invaluable perspective when designing for an unfamiliar domain, ensuring your chatbot conversations resonate with users through industry-specific knowledge and terminology.

Listen to feedback but don't let it restrict your creativity. Keep in mind that industry experts might have a bias for established ways of doing things over new technologies that disrupt the status quo, and that's normal. While valuing their perspective, remain open as the chatbot creator to innovative approaches beyond traditional conventions. Work together and meet each other somewhere in the middle.

📌 Principle #6: Know how to effectively train AI models

Having lots of training data is not enough, you need to know how to prepare this data properly to train AI models. If you are building NLU-powered or Generative AI chatbots, you need to understand the principles of NLP to train models that perform with high accuracy.

• Work with experienced AI Trainers. Hire people who know how to collect and prepare training data, train ML models, do prompt engineering e.t.c.

• It's not always possible to hire someone. Get a consultant to review your training data and train your team to prepare your training data properly to avoid mistakes.

• If you want to build the knowledge in-house, sign up for a course or follow a workshop. Have a look at Conversation Design Institute courses, check Rasa Learning Center or read my tips on how to train NLU-powered chatbots.

• Use tools like HumanFirst that make working with training data intuitive, even if you didn't get your degree in AI or NLP.

🎯 Why this matters

• Even if you have great conversation design, if your models misunderstand what the user is trying to say, it will be a very frustrating experience.

Knowing the basics of how to train AI models will help you avoid common mistakes and improve the language understanding abilities of your chatbot.

📌 Principle #7: Release your first version ASAP, get feedback and iterate

Design first but don’t be a perfectionist. Show your chatbot with just core functionality to your users as soon as possible, get their feedback and then iterate.

No matter how much time you spend talking to industry experts and doing user interviews, the first version of the chatbot you release is never going to be final. Why? Because your initial design is always based on your best assumptions about what your users want. It's only when you release your chatbot to real users when you actually start learning about how your specific target group communicates and what they want.

This is why you should resist overcomplicating early versions and launch with just core functionality, get user feedback ASAP and only then expand features. This is the only way to build a chatbot that your users will like.

💡 Design, Build, Release, Repeat

The real learning comes from users interacting with a live bot. Develop your chatbot based on empirical data vs. assumptions.

• Identify the "minimum viable bot" that solves a core user need.

• Develop initial conversations for only the most common user intents. Reality will show what you need to add later.

• Use a limited set of integrations, channels and technologies.

• Release the simple v1 bot to a small group of users first.

• Monitor conversations and collect feedback to see what works and what doesn't.

• Refine conversations, intents, flows to better match real usage patterns.

• Update your training data using conversation logs.

🎯 Why this matters

If you spend too much time designing and developing based on your assumptions you risk building a chatbot for imaginary users and might have to redo everything from scratch later on.

❗See chatbot development as a continuous iterative process

It usually takes a few iterations until you get the chatbot right. Don’t be afraid to release your chatbot too early. The earlier you release the earlier you learn and the earlier you can improve.

📌 Principle #8: Log all conversations and regularly analyse them

Logging & analyzing conversations that users have with your chatbot is absolutely a neccessity for improving your chatbot over time and if you are not doing that yet this should be the very first thing you fix. This data is gold and provides invaluable insights into how real users interact with the bot.

Rich conversation logs act as a continuous feedback loop to keep improving the chatbot. There are many ways how you can use your conversation logs:

• Conversation logs help you identify unadressed user needs. Look for mismatches between user requests and bot capabilities. When analysing the logs you might find out frequently asked questions that you haven't covered in your conversation design yet, or new requests that are often done by your users that your chatbot can't yet handle. Use those insights to prioritise new features.

• Conversation logs help you identify areas of user dissatisfaction. Identify areas with high abandonment rates or dissatisfaction, analyse why this happens and adjust your conversation flows accordingly.

• Conversation logs can be used as training data. The best training data you can possibly find is real conversations that your users had with your chatbot. The quality of the ML predictions is higher if it is trained on real life data.

• Conversation logs can be used to catch regressions. If you have a habbit of analysing your logs regularly, you have higher chances to catch any regressions in the conversations and fix them quickly.

• Conversation logs is a great source for creating test cases. Using real-life data to test your bot will make sure you optimise for real-world rather than made-up scenarios.

• Conversation logs can be used to automatically monitor if you are hitting your target metrics. Setting up an analytics dashboard will give you insights about how your chatbot is performing and what you should focus your time on next.

Set up your logging processes:

• Store conversations in a searchable transcript database. Include metadata like timestamps, user info, etc.

• Build tools to review, filter and analyze logs to uncover insights. Look for trends, failures, new topics. Or use existing tools like HumanFirst or RasaX.

• When issues arise, review related logs to diagnose root causes. Logs provide traceability.

• Designate someone to manually review samples of conversations weekly. Look for trends, new topics, failures.

• Create automated reports analyzing key metrics over time - number of conversations, intents matched, FAQs, etc.

• Watch for indicators of model degradation like increased unmatched utterances, intents, or entities.

• Generate alerts if certain metrics go beyong the acceptable thresholds.

• Share interesting insights with the team to inform improvements.

❗Don't overestimate the outliers

Regular analysis helps both catch regressions and discover areas of improvement. But be careful not to overreact to one time outliers. Improve the chatbot step-by-step based on clear, ongoing patterns in the data.

📌 Principle #9: Automate testing

As in any other development project, it’s important to have automated testing for your chatbot. Define a list of test cases and automatically test your assistant against it every time you release a new model to production and every now and then when the code is already up and running. Running automatic tests with each release will help you catch regressions when you are updating the model. Running tests routinely will prevent your chatbot quality from degrading.

🎯 Why this matters

• Whenever we make a model update, we want to make sure that not only we improved the cases that were failing before but also that our core functionality stayed intact.

• We often use third-party providers to build chatbots. When our code is already running in production, changes on the side of the third-party providers might occasionally impact our chatbot performance. Automatic testing helps identify any regressions and ensures the stability of your chatbot.

💡 How to set up tests

• Analyse conversation logs and define common conversation scenarios & expected responses, called happy paths.

• Include unusual edge cases beyond happy paths, called unhappy paths.

• Quantify the results of testing. Track metrics like intent prediction score and number of correctly/incorrectly answered questions.

• Integrate automatic testing into CI/CD pipelines. Automatically run tests both on every code change as well as regularly on a stable production version to catch any regressions.

By creating automatic tests, you gain confidence that your new changes won't break existing functionality and that your production version is stable.

📌 Principle #10: Track key performance metrics

Define, monitor and analyse key success metrics such as conversation completion rates, user satisfaction scores, etc., to quantify progress and measure your chatbot success.

💡 What to track

Depending on your business goals, you might want to track some of these metrics:

• Overall conversation volume: how often is the bot used?

• Most frequently asked questions: what are the questions your users ask most?

• Conversation completion rate: how often do conversations reach a resolution / how often do users manage to complete the task?

• Satisfaction scores / chatbot ratings: do users rate your chatbot as helpful?

• Intent recognition accuracy: how well are intents identified?

• Human handover frequency: how often is human agent handoff required?

• Conversation dropout rate: at what points and how often do users abandon the conversation?

• User return rate: (if applicable) how often do users come back to have another conversation?

🎯 Why this matters

• Reviewing your key metrics keeps your focus on the original problem and helps you understand how well your solution is addressing it.

• This also helps determine if any changes need to be made to the solution and provides insight into exactly where and why improvements may be necessary.

Monitoring your key metrics provides crucial insights into your chatbot's real-world performance. This allows you to continually refine your solution to better solve your problem.

📌 Principle #11: Collect user feedback after the release

It's important to gather user feedback and observe how your users interract with the chatbot once it's live. Actively collect user feedback through surveys and user interviews. This qualitative data is crucial for improving the chatbot experience.

• Allow users to give feedback about the chatbot right in the chat.

• Construct feedback flows within the chatbot experience.

• Request feedback after particularly good or bad conversations.

• Send follow-up surveys to chatbot users to rate satisfaction, suggest improvements etc.

• Occasionally invite users to feedback sessions, let them try the chatbot live, note when they strugle and which parts of your chatbot they love.

Continuous qualitative feedback provide insights that conversation logs alone can't reveal. Make sure this is part of your chatbot development process.

📌 Principle #12: Stay updated on the latest advancements

Chatbot technology progresses rapidly and it is useful keep up to date with the resent advancements to adop new best practices and use state-of-the-art technologies as they emerge.

• Attend conferences and events focused on conversational AI/NLP.

• Follow thought leaders/influencers in the chatbot development space. Subscribe to industry blogs, newsletters, communities. My favourites are The Batch, Synthedia and the blog by Cobus Greyling

• Experiment with new frameworks, techniques and chatbot development tools on side projects.

• Foster relationships with vendors building innovative platforms.

The future success of your chatbots relies on your team keeping their skills current as the field progresses rapidly. Make learning an ongoing priority.

Final words

I hope this overview helped you reflect on your current chatbot development practices and inspired you to take some actionable steps to make improvements in your processes.

Remember that building great chatbots is an iterative process. Always keep user needs and the problem you are trying to solve in mind, work closely with domain experts, continually test assumptions, and use real user data to guide improvements.

If you keep these problem-oriented, data-driven, and user-centric principles in mind, your chatbots will better serve customers and your business.

If you'd like extra help developing an effective chatbot strategy, feel free to reach out and start a conversation or book a free call.

And if you enjoyed this post, let me know on LinkedIn and if you like sign up to receive notifications about my future blog posts.

Take care!