Business Intelligence (BI) is the process of collecting, analyzing, and presenting business data to help organizations make better decisions. It transforms data into meaningful insights enabling companies to understand their performance, market trends, and competitive position. In this blog post I will explain how you can use BI together with a Business Process Management System.
Most BI tools follow the approach to transform raw data from a database centric business application into aggregated data for analysis. Typically those tools are built on SQL based databases which makes it easy to group data or compute totals across different queries. Of course, this approach requires a well-designed data structure and a deep understanding of the technical relationships between different information domains. Thus, it is usually a collaboration between business analysts and developers to collect this kind of data.
But what if you have a business process management system instead of a ERP software? In such a software system, there are floating and volatile workflows that generate the data to be analyzed. For example, a “complaint management” business process is more likely to look at timings, processing periods, and costs over time. The analysis of such processes in a BI tool is often very difficult to realize with conventional SQL queries. So the question is: what could be the right architecture to process this type of data in a BI platform?
The Idea of Metrics
A very powerful solution to analyze business data is the generation of metrics. A metric in this context is a specific, measurable value derived from process data that indicates value, performance or efficiency. For example, “average resolution time” would be a metric for a complaint process, calculated from the time difference between complaint creation and resolution. Another example is the balance of a customer account calculated during a invoicing/payment workflow. In this context metrics transform raw process data into meaningful, comparable measurements that can be tracked and analyzed over time.
In the following I will describe how you can adapt your process-oriented data into metrics for traditional BI approaches.
From Process Data to Business Intelligence
The challenge in analyzing process-oriented data lies in its dynamic nature. Unlike traditional ERP systems, where data is stored in fixed structures, business processes are living entities that evolve over time. Each process instance may follow different paths, involve various stakeholders, and generate unique data points. This is where metrics come into play as a bridge between dynamic process data and structured business intelligence.
The Architecture of a Metric-Based Analysis
To transform process data into actionable business intelligence, we need to establish a clear architectural approach. This approach consists of several key components that work together seamlessly to provide meaningful insights.
1. Metric Definition Layer
The foundation of our solution is the definition of meaningful metrics. These definitions form the core of our analysis capabilities. Each metric must be clearly specified and measurable within the context of your business processes. They need to remain relevant to your business objectives while being consistently calculable across different process instances.
A metric definition should further more be able to scale effectively across different process types. This can be done through appropriate categorization (also called tagging). For example, a metric about for customers’ balances can be tagged with categories such as ‘Currency’, ‘Country’ or ‘Product Category’ . This makes it possible to highlight different aspects of the metric later. E.g. the countries with the highest turnover. In this way the standardization of the metric definitionlayer ensures that your metrics provide meaningful insights regardless of how your processes evolve.
2. Data Collection and Aggregation
The data collection phase focuses on gathering and processing information as your business processes execute. This involves continuous monitoring of process execution and collecting relevant data points as events occur within your processes. The system automatically calculates metric values based on your definitions and aggregates this data over time to enable trend analysis. This ongoing collection and aggregation ensures that your metrics always reflect the current state of your business processes.
An important point to consider here is the granularity of the metric. Metrics should not have unnecessarily high level of detail, as this may lead to performance problems later on. For example, in case of many different customers a metric for the customer balances may generate to many data points. In this case it can be better to cluster them directly, for example by country or region.
3. Storage and Integration
The storage component addresses how we maintain and manage our metric data over time. Metric values are stored in a time-series format, making it simple to track changes and trends. The storage system ensures that your metric data remains easily queryable for analysis purposes while maintaining efficient access patterns. This approach allows for both immediate access to current data and historical analysis of past performance, with efficient archiving capabilities for long-term storage.
There are different types of metrics. The most common formats are:
- Counter: A number that only goes up or resets to zero on restart. The counter is well suited to count events during business process
- Gauge: A number that can go both up and down, like the balance of a customer account
- Histogram: Useful for grouping measurements into ranges or so called buckets
- Summary: Useful for calculating percentiles directly when collecting the data
4. Analysis and Visualization
The final component transforms your metric data into actionable insights. This involves creating comprehensive views of your process performance through interactive dashboards that enable real-time monitoring. The system supports detailed trend analysis and provides custom reporting capabilities to meet specific business needs. When metric values exceed defined thresholds, the system can trigger alerts to ensure timely responses to potential issues.
Benefits of a Metric-Based Approach
This architecture offers several key advantages for organizations. It provides the flexibility needed to adapt to changing business processes while delivering real-time insights into process performance. The approach significantly reduces the complexity of data analysis compared to traditional methods. As your organization grows, the system scales effectively across different process types. Since we’re storing calculated metrics rather than raw data, storage requirements remain manageable. Perhaps most importantly, this approach provides a standardized method for analyzing processes across your entire organization.
Looking Forward
The concept of using metrics as a foundation for business intelligence opens up new possibilities for process-oriented organizations. It bridges the gap between dynamic business processes and structured data analysis. In the next part of this series, we’ll explore a concrete technical implementation of this approach, showing how modern tools and frameworks can bring this concept to life.
By establishing metrics as the cornerstone of our BI strategy, we create a flexible yet powerful foundation for analyzing business processes. This approach allows organizations to maintain the agility of their business processes while gaining the insights traditionally associated with rigid BI systems.