Engineering Around AI Hallucinations

- Part 2 of an AI article series. See part 1 -

Artificial intelligence has the ability to explain technical concepts, generate code, summarize documentation, diagnose a problem, or help an engineer devise different solutions to a problem in a matter of seconds. And AI can do all this while being remarkably convincing about its answers. The problem is that a convincing answer is not necessarily a correct answer.

For engineers, this divergence is a particularly important (and often overlooked) limitation of AI. Generative AI models can sometimes produce output information that is unsupported, inaccurate, or completely fabricated while presenting it with the same confidence as factual information. These are known as AI hallucinations.

Many engineering decisions depend on specifications, standards, calculations, and other information where being approximately correct is not enough. However, we should not dismiss AI entirely and miss out on productivity gains because it can make mistakes. Rather, we should aim to understand where and how these mistakes can occur and build appropriate controls around them.

Why Does AI Hallucinate?

Large language models (LLMs) are the technology at the core of most of today’s generative AI tools.

LLMs have the ability to interpret input natural language and generate responses by predicting what information comes next based on learned patterns. LLMs excel at producing coherent and contextually relevant language. However, they do not inherently verify every statement before presenting it. Knowing this distinction is very important.

You could test this hypothesis right now. Go ask an AI model for the specifications of a particular PLC, one for which you already have access to the actual datasheet from the OEM. The LLM may correctly describe most of the product, but it may also give you an incorrect fact. Maybe it will give you a specific value from another PLC model that is similar to the one you asked about.

Figure 1. AI can sound convincing even when the answer is wrong. All images used courtesy of the author and generated with ChatGPT for this AI article

What makes hallucinations particularly challenging is that the response may sound completely plausible; it may be nearly all correct but still includes an incorrect statement somewhere. Some hallucinations are obvious, but most of them are hard to identify.

The first step in preventing hallucinations is to learn to differentiate between two types of LLM tasks.

When you use an AI tool to help brainstorm different architectures for an automation system, to improve the structure of a technical document, or to suggest test cases for a software program, you are using that tool to generate ideas. These tasks are more open-ended because there may be many acceptable answers that are ultimately evaluated by the engineer.

On the other hand, when you use AI to do things like asking for a specific torque value, researching safety requirements, or requesting network specifications, you are now asking the model to report facts and specifications. In these cases, the source of the information becomes just as important as the answer itself.

The first step is to recognize when a task requires authoritative information so that you can adjust how you prompt the AI model accordingly.

Grounding AI With Trusted Information

When working with factual tasks, one of the most effective ways to reduce hallucinations is to give the model access to the information you want it to use. For example, if you need the model’s help to find a particular motor drive parameter for your application, you can provide the manufacturer’s user manual and specification docs along with your prompt instructions. This is as simple as adding files as attachments, then asking the model to refer only to what you are providing.

This technique is called grounding, and you can use it with functional specifications, equipment lists, network diagrams, standards, operating procedures, and more.

Figure 2. Grounding AI in trusted sources improves the reliability of its responses.

Grounding exists within what is called retrieval augmented generation (RAG). RAG is an approach that improves a model’s response by supplementing it with relevant information when prompting. Through RAG, you may give the model access to an entire knowledge base containing all the controlled documentation pertaining to your project or task.

RAG can be a very powerful tool in engineering. Imagine that you have an AI assistant that can leverage your approved sources rather than the model’s general training data. The model is still an LLM generating a response, but it now uses a much more relevant foundation for you.

Matching Verification and Risks

Grounding AI with trusted information does not totally eliminate the need for verification. At the same time, not every AI interaction requires the same level of scrutiny. As you gain experience using an AI model, you will begin to discern different subsets of tasks.

Figure 3. The level of verification should match the consequence of errors.

For example, when you ask a model to help you organize information, summarize meeting transcripts, or brainstorm ideas, the consequences of getting an error in the response may be relatively small. However, if you ask the model for safety functions, operating parameters, or to interpret technical standards, an incorrect answer may have serious consequences.

We must always take a risk-based approach when using AI. We do not need to ask ourselves if the model’s answer could be wrong. We already know that it could be wrong, so instead ask, “What happens if this answer is wrong?” As the consequence severity increases, so should our reliance on trusted authoritative sources and independent validation.

Let’s always keep in mind that your own confidence in AI should come from the evidence behind the answer, and should not rely on how confidently the answer is presented.