I’m really excited to announce that an article about Waterbot has finally made its way to an academic journal! Thanks to IEEE Transactions on Professional Communication for publishing “Demystifying Chatbot Creation: A Comparative Case Study of Available Approaches.” I’m particularly proud of the work that Mayank Muthyala (who led the tests) and Briana Rajan (who created the original bot) did as part of their master’s studies.
AI research moves fast, and so some of the work in here (which we did in 2024) is very point-in-time. But there are also principles about the tradeoffs between different development strategies for bots that I think are sound longterm.
The abstract is here:
Background: Chatbots are generative artificial-intelligence (GenAI) technologies that can deliver information through a conversational interface. This ability is promising to the work of technical and professional communicators (TPCers) who are often tasked with communicating complex information that is accessible and engaging, particularly in public engagement and outreach efforts. This study focuses on demystifying the chatbot creation process to inform future applications of chatbots and GenAI in technical and professional communication (TPC).
Literature review: Research in TPC and associated disciplines has outlined the utility and use cases for chatbots. Specifically, research has focused on how chatbots can intervene in affecting people’s perceptions of global narratives and avoiding Western science bias. Although understanding utility is important, there is an underdeveloped understanding of the affordances and limitations of the emergent technologies that TPCers can utilize to build chatbots.
Research question: What are the affordances and limitations of different chatbot technologies?
Methods: This study reports the experiences of replicating a chatbot on three platforms. First, we built the Arizona Water Chatbot—a custom-built chatbot we coded using the GPT-3.5 Turbo model. Then, we replicated the chatbot using OpenAI’s custom GPT interface and OpenAI’s Assistants API platform. Once built, we compared the development experience as it relates to each technology’s affordances and limitations; namely, we compared the setup experience, customization options, training process, prompt engineering capacity, file management ability, cost, and output quality.
Results: The three chatbots had varying affordances. For instance, the custom-built bot allowed extensive control over data integration and response customization, making them ideal for projects requiring highly accurate and context-sensitive information. In contrast, the chatbots that created on Open AI’s platforms were more cost-effective, faster to implement, and suitable for projects needing rapid deployment.
Conclusion: By describing the affordances and limitations of the chatbot technologies, this article offers academics and practitioners insight into which technology to use given their individual development goals and intended audiences.