Adoption of LLMs in the workplace
A snippet of my MSc dissertation — 'Adoption of LLMs and LLM-powered software in the workplace: A case study using OpenAI's GPT-3.5 model to query an international organisation's database'.
Daniela Valenzuela, MSc Strategic Innovation Management
November 2022, and the world was caught up by storm upon the release of ChatGPT-3.5 — The promise to increase productivity at your fingertips was delivered in a way that seemed quite magical — but could this promise be attained in the workplace? Could it be used rightfully, responsibly and actually productively? In simple terms, this is what my master's dissertation aimed to uncover, building upon a specific LLM-powered tool developed as a use case.
The Rise of Large Language Models
Recent years have seen a rapid evolution in the capabilities of large language models, capturing the attention of organisations eager to harness their potential in various aspects of the workplace. LLMs enable a wide range of applications, capable of accelerating cognitive tasks, improving efficiencies, and ultimately boosting overall productivity. Yet, as with any emerging technology, uncertainty and ambiguity abound, leaving managers wondering how best to integrate LLMs while mitigating potential risks. This research, therefore, aimed to provide insights into the practical application and adoption of LLMs in real-world scenarios, focusing on the perceptions of generative AI tools, their benefits, risks, and barriers to adoption.
A Case Study on LLM-Powered Software
To delve into the multifaceted world of LLMs and their impact on the workplace, this study centred on a case study involving a bespoke tool built on the GPT-3.5 LLM. An application like this is what OpenAI has called 'LLM-powered software'; it performs a combination of tasks identified as searching over an organisation's existing knowledge, data, or documents and retrieving information to answer questions based on this data. The tool was developed through 'Question answering using embeddings-based search'; this means the model does not need to be retrained, but specific library of information is ingested to answer the specific question. if the answer is not contained in the given context, the model responds with 'I don't know'. This tool was designed to query the database of an international organisation and guide analysts by providing answers in natural language. To study the adoption of such a tool, the research adopted a mixed-method approach, combining survey data from over 80 employees within the international organisation with in-depth interviews conducted with five analysts from the same institution.
Perceptions of Benefits and Risks
The study's findings reveal how employees perceive the use of LLM-powered software compared to off-the-shelf tools. Significantly, the data demonstrates that employees perceive greater benefits and lower risks when utilising LLM-powered software. The main advantage of LLMs, in general, noted in the study, has been the quicker completion of tasks. Moreover, as stated by an OpenAI paper, 15% of US worker tasks could be done faster and with equal quality using standard LLMs; however, with LLM-powered software, this rises to nearly 50% of all tasks. In terms, of risks, there was a clear comparative perception for the specific tool as demonstrated by statistical tests; accuracy, transparency and bias became less of a concern for the participants of the study. This is mainly due to the fact that the source of the information is known and well-respected; therefore, fabrication of information is eliminated. The implications are clear: LLM-powered software has the capacity to redefine how we work, making tasks more efficient and reducing the risks associated with off-the-shelf LLMs.
Barriers to Adoption
No innovation journey comes without its challenges. The research identifies a key barrier to the adoption of LLM-powered software, and it's not a technological one. Instead, it's the lack of clear organisational guidelines and use cases. Employees and managers alike expressed a need for guidance on how to responsibly implement and utilise these tools in the workplace. This underscores the importance of not only developing the technology itself but also establishing a framework for its ethical and effective use within organisations.
Enabling Adoption: Key Takeaways
To enable successful adoption, organisations must consider two critical strategic measures: use cases and organisational guidelines. Providing employees with clear use cases demonstrates the practical utility of these tools, reducing uncertainty and promoting their responsible use. Simultaneously, the establishment of organisational guidelines is crucial in navigating the ethical and operational dimensions of LLM-powered software.
This brief article was created to disseminate the findings of my research in the most simple terms, and without delving too deep, however there are many more interesting nuances. The literature behind this research is rooted in Emerging Technology patterns, Innovation Adoption, Technology Acceptance Models and the science behind LLMs. Do request the full copy if you are interested!
