The real-time processing of a continuous stream of business data and events is becoming increasingly important in modern IT architectures. This type of architecture, in which events are building the centre of data processing, is also known as a Reactive Streaming Architecture. In the following we will show how to solve some of the related challenges with the help of a workflow technology.
Let’s take a closer look at this type of architecture first. Basically, the event-based processing of data is not new and has actually been developed for decades in various specialized domains such as the financial sector. But since the last few years, new standards for processing data streams have emerged. Technologies like Apache Kafka, Storm, Flink or Spark are gaining popularity and pushing a new hype.
From industrial production systems to multiplayer computer games, so-called Streaming Architectures are used more and more frequently in order to be able to process big data in real time. Streaming architectures have developed into a central architectural element of modern technology companies. In many companies real-time streams have become the core system in their architecture.
The goal is to be able to integrate new system solutions more quickly and to connect any kind of data streams. The streaming architecture is not only found at technology giants such as Ebay, Netflix or Amazon, but today in every modern technology company that is working on the digitization of its business processes. So what are the main challenges in building such an architecture?
For most self managed Kubernetes environments the SQL database is one of the most important infrastructure parts. Typically SQL database servers are not designed to run on distributed nodes in an environment like Kubernetes. One solution is to run a single SQL database in a Kubernetes POD with a distributed filesystem like Longhorn or Ceph. This works well for example with PostgreSQL in most situations. Of course this can have some performance impacts and requires fast SSDs. Another solution is to run a distributed SQL Database like Cockroach. With the latest version of the Imixs-Cloud project we now offer a smart solution to run a SQL Database cluster within a self managed Kubernetes cluster.
Note: CockroachDB does not support the isolation level of transactions required for complex business logic. For that reason the Imixs-Workflow project does NOT recommend the usage of CockroachDB. See also the discussion here.
CockroachDB is a distributed SQL database with a build in replication mechanism. This means that the data is replicated over several nodes in a database cluster. This increases the scalability and resilience in the case that a single node fails. With its Automated-Repair feature the database also detects data inconsistency and automatically fixes faulty data on disks. The project is Open Source and hosted on Github.
CockroachDB supports a lower level of ACID transactions. This means guaranteed atomicity, isolation, consistency, and durability of data is not the same quality as in a PostgreSQL database . However CockroachDB can be used in combination with Jakarta EE and JPA. Supporting the PostgreSQL wire protocol, CockroachDB can be used with the standard PostgresSQL JDBC driver.
In this blog I will explain how to setup and customize Wildfly to run your Jakarta EE application on Kubernetes. We use this setup in our own Open Source project to run modern Jakarata EE applications on Kubernetes. You can find this project on Github.
Wildfly is Jakarta EE 8 compatible and includes the latest Eclipse MicroProfile in version 3.3. It provides a modern application framework out of the box to simplify the development of web applications and microservices. All runtime services minimize the heap allocation and applications are starting very fast with a minimum of memory.
With the new release v5.2.0, the open source workflow engine Imixs-Workflow now supports the asynchronous execution of BPMN events.
This feature is a big step forward especially in a microservice architecture. The new so called AsyncEvents make it much more easier to decouple a Rest API call from the processing life cycle of the workflow engine. In this way the request-response pattern shows better performance and allows a very clear design of complex business processes.
The AsyncEvents were already part of the Imixs-Microservice project in a pre-release and become now a core feature of the Imixs-Workflow engine. Especially in more complex architectures, the use of the so-called SAGA Pattern is an important building block. With asynchronous events Imixs-Workflow is now supporting this design pattern as a core feature. Read also our blog about building powerful microservice solutions with the SAGA Pattern.
In my last blog I explained the core concepts behind the Microservice Saga Pattern. In this blog I will address the problem from a more practical perspective by demonstrating how Imixs-Workflow can be used as a Saga Orchestrator within a Microservice architecture. First, I would like to give a brief review of the main concepts of the saga pattern. Later I show some implementation examples.
Everyone is talking about cloud technologies and of course every modern project relies on a microservice architecture. A variety of technologies and methods contribute to the success of this architecture pattern. But what does cloud native actually mean for the business world? How do companies and organizations implement business processes successfully beyond the big technology promises?
The basic idea of a microservice architecture is to break down the technical requirements of a software system into the smallest possible and therefore manageable services. The advantage: services created in this way can be developed independently of each other with different technologies by different teams. At the same time, we see new methods and technologies to connect, monitor and scale these services.
But just looking at the technology does’t mean that software can be developed faster and better. I would therefore like to compare some of these methods and technologies from the microservice architecture with the requirements for the development of business applications.
Microservices become the new disruptive technology for software development in it’s traditional way. It is the architectural style which brings up a new way to build software systems. The Microservice Architecture evolves very fast and in deed has a lot of success.
But there are also some disadvantages in Microservices. One of the critical parts is the rising complexity within a microservice architecture. It is a problem which is often overlooked in the beginning when all this euphoria lies within a greenfield project. And the problem is often denied for a long time by its advocates. But why does this happen?
The ‘Business Process Model And Notation’ standard is a well designed notation for describing business workflows. BPMN 2.0 becomes the standard for modeling business logic and fits very well the model driven software design in agile software projects.
The BPMN language, which is based on XML, was intended for users at all levels, from the business analysts who create the initial design, to the developers who implement the technical details, and finally, to the business users responsible for managing and monitoring the processes. BPMN 2.0 has evolved to become a complete specification trying to fit the needs to all people involved in the design of a business process. But writing BPMN in XML and visualizing business processes becomes nearly impossible without the use of a graphical tool.
With the latest version of Imixs-Workflow we support now a consistency Lucene search index coupled with the Java EE container based transaction concept.
Writing a Lucene Index during a Java EE transaction, coupled to JPA database operations, it becomes quickly difficult to keep the Lucene index consistent. The reason is that a lucene index can become inconsistency if you write the lucene index within a long running Java transaction. In case the transaction fails lately, there is no way to roll back the already written lucene index automatically. This is different to the build-in roll-back functionality of a SQL database which is only writing new data in case the transaction succeeds. Also clients reading the index before a running transaction is closed will read uncommitted index data which will lead to wrong search results. Continue reading “Lucene Index Consistency”
Why do employees in companies repeatedly use Microsoft Excel or PowerPoint to manage business data and tasks?
Companies today have a variety of business applications in use. Some for legacy reasons, some for strategic reasons and some just because they are hip and make fun. And most of the times, these applications are not enough to cover all aspects of daily working life. A lot of tasks and data that need to be processed on a daily basis are often managed with EXCEL and POWERPOINT. This is not wrong in general and provides employees a way to prepare business figures in tables or display them clearly in a presentation. But often these applications are also abused for a kind of data management. But EXCEL is no database and POWERPOINT is not a messaging tool. The problem is, that these tools often collect data that is needed to implement certain business processes and for which there is no suitable application available. In a short time, these solutions become indispensable and are passed down from generation to generation.