Extracting Explanatory Rationales of Activity Relationships using LLMs - A Comparative Analysis
Kerstin Andree, Zahi Touqan, Leon Bein, and Luise Pufahl
This study investigates using Large Language Models (LLMs) to automatically extract and classify the reasons (explanatory rationales) behind the ordering of tasks in business processes from text. The authors compare the performance of various LLMs and four different prompting techniques (Vanilla, Few-Shot, Chain-of-Thought, and a combination) to determine the most effective approach for this automation.
Problem
Understanding why business process steps occur in a specific order (due to laws, business rules, or best practices) is crucial for process improvement and redesign. However, this information is typically buried in textual documents and must be extracted manually, which is a very expensive and time-consuming task for organizations.
Outcome
- Few-Shot prompting, where the model is given a few examples, significantly improves classification accuracy compared to basic prompting across almost all tested LLMs. - The combination of Few-Shot learning and Chain-of-Thought reasoning also proved to be a highly effective approach. - Interestingly, smaller and more cost-effective LLMs (like GPT-4o-mini) achieved performance comparable to or even better than larger models when paired with sophisticated prompting techniques. - The findings demonstrate that LLMs can successfully automate the extraction of process knowledge, making advanced process analysis more accessible and affordable for organizations with limited resources.
Host: Welcome to A.I.S. Insights, the podcast where we connect academic innovation with business strategy, powered by Living Knowledge. I'm your host, Anna Ivy Summers. Host: Today, we're diving into a fascinating study titled "Extracting Explanatory Rationales of Activity Relationships using LLMs - A Comparative Analysis." Host: It explores how we can use AI, specifically Large Language Models, to automatically figure out the reasons behind the ordering of tasks in our business processes. With me to break it all down is our expert analyst, Alex Ian Sutherland. Welcome, Alex. Expert: Great to be here, Anna. Host: So, Alex, let's start with the big picture. Why is it so important for a business to know the exact reason a certain task has to happen before another? Expert: It’s a fantastic question, and it gets to the heart of business efficiency and agility. Every company has processes, from onboarding a new client to manufacturing a product. These processes are a series of steps in a specific order. Host: Right, you have to get the contract signed before you start the work. Expert: Exactly. But the *reason* for that order is critical. Is it a legal requirement? An internal company policy? Or is it just a 'best practice' that someone came up with years ago? Host: And I imagine finding that out isn't always easy. Expert: It's incredibly difficult. That information is usually buried in hundreds of pages of process manuals, legal documents, or just exists as unwritten knowledge in employees' heads. Manually digging all of that up is extremely slow and expensive. Host: So that’s the problem this study is trying to solve: automating that "digging" process. How did the researchers approach it? Expert: They turned to Large Language Models, the same technology behind tools like ChatGPT. Their goal was to see if an AI could read a description of a process and accurately classify the reason behind each step's sequence. Expert: But they didn't just ask the AI a simple question. They compared four different methods of "prompting," which is essentially how you ask the AI to perform the task. Host: What were those methods? Expert: They tested a basic 'Vanilla' prompt; then 'Few-Shot' learning, where they gave the AI a few correct examples to learn from; 'Chain-of-Thought', which asks the AI to reason step-by-step; and finally, a combination of the last two. Host: A bit like teaching a new employee. You can just give them a task, or you can show them examples and walk them through the logic. Expert: That's a perfect analogy. And just like with a new employee, the teaching method made a huge difference. Host: So what were the key findings? What worked best? Expert: The results were very clear. The 'Few-Shot' method—giving the AI just a few examples—dramatically improved its accuracy across almost all the different AI models they tested. It was a game-changer. Expert: The combination of giving examples and asking for step-by-step reasoning was also highly effective. Simply asking the question with no context or examples just didn't cut it. Host: But the most surprising finding, for me at least, was about the AIs themselves. It wasn't just the biggest, most expensive model that won, was it? Expert: Not at all. And this is the crucial takeaway for businesses. The study found that smaller, more cost-effective models, like GPT-4o-mini, performed just as well, or in some cases even better, than their larger counterparts, as long as they were guided with these smarter prompting techniques. Host: So it's not just about having the most powerful engine, but about having a skilled driver. Expert: Precisely. The technique is just as important as the tool. Host: This brings us to the most important question, Alex. What does this mean for business leaders? Why does this matter? Expert: It matters for three key reasons. First, cost. It transforms a slow, expensive manual analysis into a fast, automated, and affordable task. This frees up your best people to work on improving the business, not just documenting it. Expert: Second, it enables smarter business process redesign. If you know a process step is based on a flexible 'best practice', you can innovate and change it. If it's a 'governmental law', you know it's non-negotiable. This prevents costly mistakes and focuses your improvement efforts. Host: So you know which walls you can move and which are load-bearing. Expert: Exactly. And third, it democratizes this capability. Because smaller, cheaper models work so well with the right techniques, you don't need a massive R&D budget to do this. Advanced process intelligence is no longer just for the giants; it's accessible to organizations of all sizes. Host: So it’s about making your business more efficient, agile, and compliant, without breaking the bank. Expert: That’s the bottom line. It’s about unlocking the knowledge you already have, but can't easily access. Host: A fantastic summary. It seems the key is not just what you ask your AI, but how you ask it. Host: So, to recap for our listeners: understanding the 'why' behind your business processes is critical for improvement. This has always been a manual, costly effort, but this study shows that LLMs can automate it effectively. The secret sauce is in the prompting, and best of all, this makes powerful process analysis accessible and affordable for more businesses than ever before. Host: Alex Ian Sutherland, thank you so much for your insights today. Expert: My pleasure, Anna. Host: And thank you for listening to A.I.S. Insights — powered by Living Knowledge. Join us next time as we uncover more research that's shaping the future of business.
Activity Relationships Classification, Large Language Models, Explanatory Rationales, Process Context, Business Process Management, Prompt Engineering
