Bachelor's thesis: LLMs × BPMN.

Context

Process modeling is expensive: it takes expert knowledge about the business and about notations like BPMN. LLMs promise to automate both — but can freely available models really turn natural language into valid, usable BPMN 2.0 models? The literature had mostly looked at single models. This thesis compares eight — systematically and measurably, with opinion replaced by measurement.

Solution & highlights

• Eight models in the ring

  o1, GPT-4o, GPT-4o-mini and GPT-4 (OpenAI), Claude (Anthropic), Gemini and Gemini 2.0 (Google), Mistral 7B — from frontier model to open source, all under identical conditions.

• Custom evaluation framework

  Valid BPMN 2.0 XML, completeness, precision, F1 score and logical soundness — quantitative instead of gut feeling. Benchmarked on real datasets from Camunda and hdBPMN plus targeted "pseudo-BPMN" tasks covering gateways, events, loops, subprocesses, lanes and data objects.

• Instruction beats intuition

  Given clear, precise descriptions, the strongest models produced nearly flawless BPMN. The top group: Mistral 7B, Gemini 2.0, Claude and GPT-4o.

• The double load breaks everyone

  With real, unstructured process descriptions, models must first extract the process and then model it — under this double load, all eight models degraded significantly, no exceptions.

• XML is harder than understanding

  Several models extracted processes correctly but failed to produce valid BPMN 2.0 XML — the older Gemini didn't manage a single valid document. Format fidelity is its own problem, not a side effect.

• The human stays in the loop

  Without human correction, no complex process yielded a correct model. That's exactly where my working principle comes from: AI with code review instead of blind trust.