This Is Why Human-Centred AI Design Guidebooks Can Gracefully Fail When Used in Manufacturing
We see there is growing attention to Human-centred AI (HAI) in various communities. The basic idea of HAI is to put humans and humanity at the centre of designing AI-powered applications. The HAI design seeks symbiosis of humans and AI – AI assisting human tasks rather than replacing them and humans improving AI by providing feedback.
Big tech companies like Google, Microsoft, IBM, Apple, and other enterprises value the idea of HAI and have developed their own HAI design methods and shared them in public as guidebooks. For instance, the People+AI guidebook from Google's PAIR research team shows how to organise and facilitate a series of workshops where participants with different domain expertise co-design the functionality and user interface of an AI application. It also provides a set of questions and guides important to be addressed during the HAI design process, such as "What is the user value of the application?" and "How the prediction results should be explained to users?". The guidebook further gives us various example use cases where HAI designs were applied in practice to inspire the design participants. Microsoft shares an HAI method called "HAX toolkits". It offers design guides and workbooks in PowerPoint and Excel formats to serve a similar purpose as the PAIR's guidebook.
The essences of those HAI methods are alike; they allow multi-domain people to participate in the Design process and facilitate capturing and transforming the people's needs into the application design by integrating the theories and practice of User Experience, Design Thinking and Responsible AI into a unified design framework.
Ok, now I am talking about Manufacturing :). You may think manufacturing is a domain that can be easily automatised, but not at all! Even at many modern factories, many skilled persons are working there and play important roles in developing, running, and improving factory operations. It is, therefore, vital to creating harmony between humans and machines.
So why not use HAI methods when integrating AI technology in manufacturing? That's what we – a research team with expertise in AI, manufacturing, and UX – thought. We work with a large multinational manufacturer trying to develop and implement a Machine Learning model for anomaly detection in the manufacturing processes. The model detects anomalous patterns of data from sensory devices installed in critical pieces of manufacturing equipment. A prototype of the model was there. We used the People+AI guidebook to help the company's AI project. This method was chosen because it seemed to be the most comprehensive and well-structured one. We used this HAI method through a one-day workshop with about ten company members with various roles, such as R&D engineers, process engineers, data scientists, technicians, and Lean Six Sigma experts.
So, what is the result of using the method? Well, I would not say it was a complete failure but not particularly successful. Overall, the method did not effectively address the complex and multifaceted challenges when designing an AI-powered application for an industrial process. The workshop facilitators (we) and the participants felt that they had to deal with too many questions from different angles at the same time, causing them cognitive overload and giving them a disorganised and confusing experience.
But we learned a lot from it! We became aware that the method was simply not well fit for the manufacturing context and that a significant reconstruction of the method was necessary to cope with the challenges we experienced. Considering the similarity among the HAI methods, we believe the result would not be much different if we had used other HAI methods.
Let us share our reflection on why the method gracefully failed with cognitive overload and confusion when used in manufacturing. Several factors contributed to the failure, but in this article, I pick up three significant ones. I hope this article is enjoyable for those interested in using AI in industrial settings, whatever your expertise is.
1. Workflow design was not an integral part of the method:
The current HAI methods from the tech companies and other enterprises seem to be primarily geared toward assisting the design of applications used by a single user, such as mobile phone apps for consumers. In such use cases, the interaction between the human and machine typically occurs through the screen, fingers, eyes, and ears of the user. The guidebook seems to support well in designing such interaction by enabling designers to explore different user scenarios and experiences, find the right balance between automation and user control, manage expectations around AI capabilities, and so forth.
On the other hand, the context of AI service in an industrial facility can be quite complex. Let us imagine the case of using anomaly detection in a manufacturing plant. This application shows the health status of the sensory devices through a monitor screen placed on the shop floor and sends an alarm when an anomaly is detected. First-hand users of the application are operators. Of course, interactions between operators and the application are important, but things do not end there. What should operators do or want to do when they receive the alarm? Does this person want to analyse the situation deeper by oneself with the help of the application? Or should he or she consult one's supervisor or technician for further analysis and decision-making? Should the equipment supplier be immediately contacted instead? Does the appropriate action depend on the seriousness of the anomaly? Does the action depend on the skill and knowledge of these persons? How many stakeholders need to be involved in the decision-making? What information should be available for them? How can information be shared among those actors?
As you see, in a manufacturing context, an initial event – raising the alarm in this case – often triggers a complex chain of other actions potentially involving multiple persons within or outside of the organisation. Let's call this chain of actions a workflow. We have learned that finger-eye-screen interactions are hardly designed without the design of the workflows. Thus, it is crucial to consider these designs simultaneously or at least plan the workflow design earlier in the development process, as they are closely interconnected.
Did the HAI method support this? No, not for the workflow design part. During the workshop at the case company, the design participants were glad to create different paper prototypes of how anomaly status and other relevant information should be displayed on the shop floor. They, however, quickly became unsure of which prototypes would be suitable for the actual use, as they had a limited understanding of how the workflows would unfold. An alarm on the shop floor is just one trigger of a workflow. There could be more scenarios triggering other workflows, such as false negatives, false positives, sensor degradation, sensor upgrades, etc. Without proper methodological support, imagining all those scenarios and their corresponding workflows required significant cognitive effort from the participants.
2. The design guides and questions sparked a multitude of additional inquiries:
As we discussed in the introduction of this article, the PAIR guidebook, like other HAI methods, offers a set of questions and guides that need to be considered during the application design process. I can show some more examples here; "How to establish a proper level of trust so that users will not put too much or too little trust in the AI result?", "How can the application accept feedback from the users to improve the application's behaviour?".
These questions or guides are surely beneficial for us to thoroughly tackle key design concerns in the design process. At the same time, addressing these questions requires extensive what-if thinking, especially for AI-driven applications behaving probabilistically. The exact behaviours of the applications are not always clear during development. For simpler interactions, such as of mobile phone apps, the what-if thinking may still be manageable. In the workshop at the company, however, what-if thinking quickly snowballed to a level we could not handle.
At the start of the workshop, little was decided except the participants' will to utilise the anomaly detection model in the operations. We followed the design process that the guidebook suggested, and the design guides and questions seemed to be helpful for the process. The workshop participants, however, became quickly unsure which questions and guides were more important than others and in which depth or detail the questions should be answered. Those questions were also tightly interrelated.
Consequently, answering those questions became a lot of guesswork. Let's take one of the design questions as an example – how to establish the users' trust in the application. Many factors can affect this, but at least it depends on how prediction results are presented to users. The design of the presentation can be affected by the model's performance. The performance is going to be affected by the production-phase data that is not fully known during the development. As we discussed earlier, the result presentation is also dependent on the workflows.
As you see, a single design question causes a chain of other interlinked questions that are hardly answered at once. An answer depends on another answer which also depends on another answer which can be only partially answered…no wonder why the participants quickly got puzzled and overwhelmed. A participant said in the end, "ok, we know now there is a huge mountain ahead of us, but we are still unsure how to climb it."
3. The responsibility of consolidating the collected information was ambiguous:
The HAI methods facilitate design participants to generate a large amount of information necessary to design an AI-powered application. The methods offer various tools, such as ideation cards, design questions, guides, and workbooks, to assist in the generation and documentation of this information.
But who will consolidate all those information? Through the workshop, it became apparent that the method was designed primarily from a UX designer's perspective and that the designer seemed to be the one consolidating the information and transforming it into the design.
Ok, we understand that the phrase "Human-Centred AI" is emphasised in the HAI methods, but they are much biased toward UX. This bias may not confuse people when the methods are used for a simpler interaction, such as a mobile phone app. UX designers have rich experience in designing the functionality and interface of such an application.
But how about when the method is used for industrial processes where the workflow design is a critical and inseparable part of the interaction design? In such a multifaceted use case, should a UX designer still consolidate the information from the workshop? Or would a project leader with a broad and in-depth understanding of the industrial processes be better suited for the task? We started the workshop without a clear understanding of this issue, which further complicated the workshop (that was already a mess!).
Finally, learning from the failure and moving on…
The three factors were already sufficient to overwhelm the participants and create a state of cognitive overload and confusion. We simply entered the workshop with a premature understanding of the limitation of the HAI methods when applied in industrial processes. Although those methods provide us with a solid foundation, we found that a significant modification would be necessary to suit the manufacturing domain.
We are currently developing a new method based on our learning and testing it at companies. We know at least that workflow design should be integrated into the method and that the method should effectively handle the flurry of interrelated what-if questions that arise during the design process. Hopefully, we can report the result in future!! :).
