Microservice Saga Pattern with Imixs-Workflow

In my last blog I explained the core concepts behind the Microservice Saga Pattern. In this blog I will address the problem from a more practical perspective by demonstrating how Imixs-Workflow can be used as a Saga Orchestrator within a Microservice architecture. First, I would like to give a brief review of the main concepts of the saga pattern. Later I show some implementation examples.

Domain Driven Design

A core idea of the microservice architecture is to distribute the technical requirements among independent services managed by independent developer teams. One aspect going hand in hand with this idea is the distribution of responsibilities. Each team is responsible for its own service. This makes sense, because it can also reduce the complexity of large organizational structures. Looking at the data of a microservice, the concept of ‘bounded context’ comes into play about which a lot is being written.

The idea behind the term ‘bounded context‘ originally came from the Domain Driven Design (DDD) which was developed in 2003. The idea is, to describe the data objects independent form the technology in a domain model. In this way the separation in smaller blocks is well supported and simplifies the development. The goal for each team is – together with the business departments – to decide which of the data belongs to a microservice and which does not. Each microservice is therefore also responsible for its own data.

Long Running Business Transactions

With the separation of responsibility another problem occurs in the moment when data has to be exchanged between several services during a long running business transaction. The reason for the need to exchange data is the business process behind the scene. You need to deal with different kind of data in different services. How ever you need to guarantee the data consistency over all services. And this problem cannot be easily solved within the a local data model.

In general, a business processes describes the purpose of an application from a business perspective. It is less about the data itself than about describing how a certain business goal can be achieved. For example this can be a sales process within an online shop, the processing of customer services on an internet platform or a production process in a factory.

The Saga Pattern

Now the big question is: Who is responsible for the data consistency across all the service boundaries of the different services? For example, in the case of an online shop, the update of the inventory must be synchronized with the logistic service and the payment service in order not to risk inconsistent data in the entire system.

The “Saga Pattern” is a design pattern to describe and implement such dependencies. A “saga” describes a sequence of local transactions within a microservice architecture with the goal to update data across several independent services. On the one hand a saga describes the order of updates and on the other hand, in case of an error, how data updates can be rolled back.
This means that as soon as a single microservice fails within a business transaction, the saga must explicitly undo the changes to services that have already committed their local data. See the following example:

The business transaction consists of three service calls. If the last service call in this example fails, the previous changes of the service calls 1 and 2 must be rolled back. This is known as transaction compensation and this is an important concept to maintain data consistency within a long-running business process. In this scenario it does not matter whether the “error” occurred for technical or business reasons.

The Saga Orchestrator

To coordinate the saga a so called Saga Orchestrator need to be implemented. The Saga Orchestrator tells each saga participant what to do. The saga orchestrator itself acts as a microservice and communicates with the participants either synchronously or asynchronously. With each phase in a long running business transaction, the orchestrator calls a different service. The advantage: The orchestrator calls the participants directly via an API and does not have to be informed via external events. As a result, the orchestrator depends on the participants and not vice versa. The participants themselves have no knowledge about the saga. As a result there is less coupling between the services and no risk of cyclical dependencies.

Model Driven Design and BPMN 2.0

Now how can the Saga Pattern be implemented? Since a saga is finally a description of a business process, a model-based approach is obvious here. The Business Process Modeling Notation (BPMN) has established as the de facto standard to describe a business process. BPMN can be used to describe a process form the business perspective but also offers a way to store technical details within a model. This makes the BPMN model a perfect choice to describe and to orchestrate a saga.

The Imixs-Workflow Engine

The open source project Imixs-Workflow offers a powerful tool to implement the saga pattern. With its Rest API, the Imixs-Workflow engine can itself be deployed as a microservice. The sub-project “Imixs-Microservice” provides a production ready Docker Container that can be quickly integrated into a container based environment like Kubernetes or Docker-Swarm.

Modelling

After the deployment of the Imixs-Microservice, you can immediately start modeling your saga using the BPMN modelling tool Imixs-BPMN. Imixs-Workflow is an event-oriented workflow engine. This means that the different states of the saga are described as tasks. The transition from one state to the next is described by events. By combining these elements with a gateway element, a business rule can be modeled to handle different situations. The next example illustrates the modeling concept.

As soon as a new Order has been created, the ‘Place Order’ event is fired to update the inventory by an API call to the Order Service.
The saga is now in the status ‘Payment’.
The next API call starts the payment process via a call against the Payment Service API.
The result of the payment call is evaluated.
1. in case the payment fails, a ‘compensation’ task is initiated via a gateway. This initiates the correction of the previously booked order via a new API call against the order service.
2. in case the payment was successful the saga can be completed

This is a simplified example, but it shows how the coordination of the actual business process can be described with a model driven design. Imixs-Workflow automatically persists each process step and creates a process log. In this way each process instance can be traced backwards later. With regard to compliance guidelines, this functionality is a mandatory requirement for complex business processes within an organisation.

Plugins & Adapters

Let’s take a look how the API Call to a Service can be implemented. The Imixs-Workflow engine provides different extension points to implement custom functionality. The Imixs-Workflow Adapter API is an extension mechanism to adapt the processing life cycle of a BPMN event. An adapter class can be implemented to call an external microservice. The result of such a call can be evaluated by the workflow engine for further process processing. The following code example shows the implementation of an adapter calling an external Rest Service:

public class OrderAdapter implements SignalAdapter {
  public ItemCollection execute(
     ItemCollection doc, 
     ItemCollection event) throws AdapterException {
    
  Client client = ClientBuilder.newClient();
  try {
    result=client
      .target(REST_URI)
      .path(doc.getItemValue("productId"))
      .request(MediaType.APPLICATION_XML)
      .post(Order.classEntity.entity(order,
        MediaType.APPLICATION_XML));
     // adapt result
     ...
  } catch (ResponseProcessingException e) {
    throw new AdapterException(
      StockAdapter.class.getSimpleName(),
      ERROR_API_COMMUNICATION, 
      "Failed to call rest api!");
  }
}

The workflow engine provides the adapter with information about the current process instance (doc) and the event. This data (e.g. the productID) can be used to transform data for a POST or GET request against the Order Service.
If the call fails or the data is not valid, an AdapterException is raised. This exception can intercept the further processing by initializing an appropriate compensation via the BPMN model.

The configuration of the API call can be part of the BPMN model. For example a BPMN data object can store a XML or JSON structure with the corresponding information. In this way the adapter can be reused for different service calls.

The Anti-Corruption Layer

In case the adapter class calls the external service directly via an API call we have a tight coupling between the Saga Orchestrator and the service. This should be generally avoided. The so-called Anti Corruption Layer Pattern can be used to resolve this dependency. In this pattern an intermediate layer between the Saga Orchestrator and the external service is implemented to adapt and the data and to decouple the services:

In this solution pattern the API call is encapsulated in a separate microservice. This separates the two layers and the Saga Orchestrator can now be developed independently of other services. If the business process behind it changes, or if new APIs are added, these changes can now be made within the BPMN model without the need to change the Orchestrator Service itself.

Example

The Imixs-Workflow API provides a generic data object called XMLDocument which can be used for a Rest API communication. This XMLDocument depends only on the Jax-B API so it can be used in any framework. The following example shows a Spring implementation of an adapter service:

@Controller
@RequestMapping("/adapter")
public class ControllerExample {
  @PostMapping("data")
  public ResponseEntity<?> getData(@RequestBody XMLDocument requestXML) {
    // consume xml request
    ItemCollection requestData = XMLDocumentAdapter.putDocument(requestXML);
    // consume data....
    @SuppressWarnings("unused")
    String param1 = requestData.getItemValue("param1", String.class);

    // create response data object
    ItemCollection resultData = new ItemCollection();
    resultData.setItemValue("_subject", "some data...");

    // convert resultData into xml
    return ResponseEntity.ok().body(XMLDocumentAdapter.getDocument(resultData));

  }
}

This Spring example consumes an XMLDocument and extracts the data by converting the XML Root element into the convenient class ItemCollection. The response object again is converted into a XMLDocument.

This implementation of an adapter service can be called with an adapter class shown in the example before. The Saga Orchestrator is now independent form the implementation of its Saga Participants and the saga can be changed by changing the model only.

Conclusion

The complexity within a microservice architecture can quickly increase if multiple services with different data (bounded context) have to be coordinated in the course of a business process. The saga pattern offers an elegant way to describe the relationships and processes. Imixs-Workflow provides an elegant way to run a Saga Orchestrator as a microservice. Complex business processes can be modeled using BPMN. The extension points provided by Imixs-Workflow simplifies the integration of services in a generic way.
By using an anti-corruption layer, the Saga Orchestrator can be completely decoupled from the individual services. This avoids dependencies between the orchestrator and the services. The business process can be changed without changing the implementation of the Saga Orchestrator.