Generating events from a BPMN process

One of the primary considerations for using IBM Business Monitor is to receive events from an instance of IBM BPM. Prior to Monitor v8.5.5 there was an old mechanism and at 8.5.5 and beyond there is a new mechanism. We will discuss the new v8.5.5 mechanism only.

With the arrival of Monitor 8.5.5, the old Common Event Infrastructure (CEI) technology has been deprecated and replaced with a new story called the "Dynamic Event Framework" or DEF. The core concept behind DEF is that Monitor knows what kinds of events it wishes to receive and there is no point in anyone sending in events other than those. From the perspective of a system that is the origination of events, it should not send any events other than those that are desired to be received.

If we think of DEF as two distinct logical parts, we have an event originator part and an event receiver part.

The source of the events (BPM) makes all its events available to the DEF Sender. This "knows" what events the "DEF Receiver" is interested in receiving and filters the available events such that only those of interest are actually transmitted over the network. When the events arrive at the DEF Receiver, they are made available to the consumer (Monitor).

In this model of the world, the DEF Sender and the DEF Receiver are in communication with each other. This communication is now one way. If we follow our story carefully, we see that the DEF Sender needs to know which events the DEF Receiver is interested in receiving. This means that from time to time, the DEF Receiver must tell the DEF Sender what it wants to receive.

IBM BPM 8.5.5 and Business Monitor 8.5.5 must exist in separate WAS cells. As such, we need to configure communication between these two servers for events generated within BPM to make their way to Business Monitor for consumption and processing. There is a recipe that needs to be followed.

We will assume that we have two environments. One is a BPM environment and the other is a Business Monitor environment. Because both environments are WAS based, we can get confused very quickly on which environment a command or step is being performed. We will take great care to call out very explicitly the environment upon which the step is being performed.

  1. We need to import into Business Monitor the SSL certificate that is being used by BPM. This allows Monitor to trust that SSL requests initiated from BPM are valid. This is essential in a secured environment.

We attach a WAS admin console to Monitor and navigate to "Security > SSL certificate and key management > Key stores and certificates > NodeDefaultTrustStore > Signer certificates > Retrieve from port"

Now we enter the details of the location of the BPM server Deployment Manager. In our example it is on localhost at SOAP port 8879. We call our new signer certificate retrieved from BPM "bpm_signer".

We click the "Retrieve signer information" button and then "OK" to commit.

The result is a new signer certificate contained within the Business Monitor definition.

  1. We need to import the signer certificate into BPM that is used by Business Monitor. This is effectively the inverse of the step we just performed where we imported BPM's signer certificate into Monitor.

We connect a WAS admin console to BPM's deployment manager. Next we navigate to "Security > SSL certificate and key management > Key stores and certificates > CellDefaultTrustStore > Signer certificates > Retrieve from port":

Next we enter the details to allow us to connect to the Monitor server to retrieve the certificate. In our example, the Monitor SOAP port is 20027.

Once we have retrieved the data, we can save our changes.

At the conclusion, we will have a new entry in BPM SSL signers table.

  1. Create a J2C security entry for the Cell admin userid/password of BPM within the Business Monitor environment. This entry will be used by Business Monitor to connect with the deployment manager of BPM to install and configure the DEF Sender part to send in events to monitor. Using the WAS Admin console connected to Monitor, open "Security > Global Security".

  1. Determine the SOAP listener port for the Deployment manager hosting BPM. From the WAS Admin console connected to the Deployment Manager, navigate to "System administration > Deployment manager":

From there we can expand the Ports and see the value for "SOAP_CONNECTOR_ADDRESS":

In the example above, the value we need is 8879.

  1. Create a new Event Source definition ….

Click the Add remote event sources... button to add a new event source.

Clicking create will create a new event source on Monitor and cause BPM to be configured to transmit events to Monitor.

If we were to look at the BPM WAS resources, we would now find some additional definitions.

There would be a JDBC Data Souce:

  1. Restart the servers. Restart both BPM and Monitor for all the changes to take. Experience seems to say that this is required and should not be omitted.

Capturing DEF events

Scripts are provided by IBM to configure DEF. These can be found in <Install>/BPM/Lombardi/tools/def. The scripts found there include:

  • EnableBPMAnalytics.py
  • SampleConfigureEventsToJMS.py
  • SampleConfigureJSONEventsToJMS.py
  • SampleReloadDEF.py

SampleConfigureEventsToJMS.py

While this script has the word "sample" in its name, it should be considered far more than just a sample. As best as we can determine, this is how one enables DEF processing in a BPM environment. When executed, it performs low-level magic to switch things on. It runs underlying wasadmin commands that include:

  • AdminConfig.create('DefProducer')
  • AdminConfig.create('DefProperty')

and a bunch of other low level (and as far as I can tell undocumented) commands.

Within the script (and near the top), there are variable definitions that own the primary configuration. These variables are:

  • defListenerId - The identity of the DEF definition. The default is jmsListenerTHREE.
  • eventQueueJndiName - The JNDI name of a JMS queue. The default is jms/monQueue_bpmn.
  • eventQueueCFJndiName - The JNDI name of a JMS queue connection factory. The default is jms/monQCF_bpmn.
  • eventQueueCF_AuthorizationAlias - The identity used to interact with the JMS subsystem. The default is DeAdminAlias.
  • subscriptions - The subscriptions on which the DEF subsystem will capture and publish.

For example:

subscriptions = [
 'HSS/*/BPD/*/PROCESS/*/*',
 'HSS/*/BPD/*/ACTIVITY/*/*',
 'HSS/*/BPD/*/GATEWAY/*/*',
 'HSS/*/BPD/*/EVENT/*/*'
]

The format of the subscription is:

  • Application Name
  • Version
  • Component Type
  • Component Name
  • Element Type
  • Element Name
  • Nature

As we can see there are seven fields. A subscription is the concatenation of these fields separated by a slash (/). A wildcard character of * means that we don't care and everything will match. To subscribe to absolutely everything we would supply:

subscriptions = [
  '*/*/*/*/*/*/*`
];

Before using this script, we must create the JMS resources and, before creating the JMS resources, we must create the SIBus resources.

First launch the WAS Admin console and navigate to Service integration > Buses. From there, we can create ourselves a new bus. I called mine DEFBus and switched off security. With the Bus created, we now create a new Bus Member. With the bus member created, we can now drill into the Bus definitions and create a new destination. I called mine DefQueue. At this point we have a new bus, a new bus member and a new queue hosted on the bus.

Now it is time to create the JMS definitions. First we create the JMS queue connection factory definition. I called mine jms/monQCF_bpmn to match the default in the script. Next we create the JMS queue definition. Again, I called mine the default name of jms/monQueue_bpmn.

With these resources created, we can now run the script to create the DEF resources in the running server. We perform the following command:

$ wsadmin -lang jython -f SampleConfigureEventsToJMS.py

We can now run the SampleReloadDEF.py script to cause a reload.

$ wsadmin -lang jython -f SampleReloadDEF.py

On a restart, we may see the following messages illustrating things are well:

...
ResourceMgrIm I   WSVR0049I: Binding jms_monQCF_bpmn as jms/monQCF_bpmn
...
SibMessage    I   [DEFBus:Node1.server1-DEFBus] CWSID0016I: Messaging engine Node1.server1-DEFBus is in state Started.
...

Consuming the events

At this point, when we run BPM work, the BPM internals will capture events and publish them through DEF. This will result in JMS messages being logged to the desired JMS queue. The format of the messages are text and their content is XML. We can browse those JMS messages using the JMS message tools found in the WAS admin console.

While the mechanical structure is XML, if we devolve to thinking in terms of XML elements, attributes and paths wihin the data we miss the wood for the trees. The power of an event is the information it contains and not its physical structure used to hold that information. If we think of an event as an "object" then, in theory, we should be able to model accessors to that object which would then "own" how to retrieve the information from it. I propose the following logical methods that could be implemented to retrieve the information:

  • getKind() - mon:monitorEvent.mon:eventPointData.mon:kind.content
  • getTime()
  • getJSON()
  • getSequenceId()
  • getProcessId() - mon:monitorEvent.monEventPointData.mon:model.[x].mon:instance.mon:id
  • getProcessName() - mon:monitorEvent.monEventPointData.mon:model.[x].wle:processApplication.mon:name
  • getProcessAppName() - mon:monitorEvent.monEventPointData.mon:model.[x].mon:name
  • getActivityName() - mon:monitorEvent.monEventPointData.mon:model.[x].mon:name
  • getActivityType()
  • getTokenId() - mon:monitorEvent.monEventPointData.mon:model.[x].mon:instance.mon:id
Note: Describing Paths
The events produced by BPM are hierarchical in nature. This means that a field called X could be contained in a container called Y which is itself contained in a container called Z.In addition, the original data was published in XML format. That becomes important to us because the XML data includes the concept of namespaces. For example the XML element called <person:age> might be distinct from <wine:age> even though from an XML perspective, both are the XML element called age. They belong in distinct namespaces. Since we are going to think about our BPM events in terms of JSON rather than XML, we will find that the names of each of the fields in the data may/will be prefixed with their XML namespace separated by a colon. For example, we will find JSON fields that might be called person:age and wine:age. Combining these notions together, we will use a representation of the form Z.Y.X to represent a field called X contained in Y contained in Z and if we include namespaces, this may become owner:Z.type:Y.location:X. This convention follows the style found in MongoDB.

Event sequences

We should now understand that for a given process instance, it will be emitting a sequence of events. The sequence follows a pattern that is loosely:

  • bpmnx:PROCESS_STARTED
  • bpmnx:EVENT_CAUGHT
  • bpmnx:ACTIVITY_READY
  • bpmnx:ACTIVITY_COMPLETED
  • ...
  • bpmnx:ACTIVITY_READY
  • bpmnx:ACTIVITY_COMPLETED
  • bpmnx:EVENT_THROWN
  • bpmnx:PROCESS_COMPLETED

Note that some events have been omitted for clarity.

mon:kind - Event types

Imagine now that we have a collection of event documents warehoused in our database. What now? There is a field in the document called mon:monitorEvent.monEventPointData.mon:kind.content. This field is found in every document. Its value explains why the event was written.

Each event has a:

mon:monitorEvent {
  mon:eventPointData {
    mon:kind {
      content: <Type>
    }
  }
}

The mon:monitorEvent.mon:eventPointData.mon:kind.content is key and has at least the following types:

  • bpmnx:PROCESS_STARTED
  • bpmnx:PROCESS_COMPLETED
  • bpmnx:PROCESS_TERMINATED
  • bpmnx:PROCESS_DELETED
  • bpmnx:PROCESS_FAILED
  • bpmnx:ACTIVITY_READY
  • bpmnx:ACTIVITY_ACTIVE
  • bpmnx:ACTIVITY_COMPLETED
  • bpmnx:ACTIVITY_TERMINATED
  • bpmnx:ACTIVITY_RESOURCE_ASSIGNED
  • bpmnx:ACTIVITY_LOOP_CONDITION_TRUE
  • bpmnx:ACTIVITY_LOOP_CONDITION_FALSE
  • bpmnx:ACTIVITY_PARALLEL_INSTANCES_STARTED
  • bpmnx:EVENT_CAUGHT
  • bpmnx:EVENT_EXPECTED
  • bpmnx:EVENT_THROWN
  • bpmnx:GATEWAY_ACTIVATED
  • bpmnx:GATEWAY_COMPLETED
  • wle:PROCESS_AT_RISK_DATE_ASSIGNED
  • wle:TASK_FIELD_CHANGED

Question: What is it that links records together in a chain to form a flow?

Possible answer: If the record contains a mon:model entry of type bpmn:process then the same mon:model entry will contain mon:instance.mon:id which will be the PIID.

mon:model - Who issued the event?

When an event is published, we find that mon:kind tells us why an event was published but not who published it. Let us start with the notion of what who means in this context. If we limit our discussions to just BPM processes, then we will see that it is not a trivial question. Imagine a step in our process called Step S. If the entry to that step results in an event then we can say that Step S is the publisher. However, a process step doesn't exist in isolation. A process step exists within the context of a process. If Step S is found in Process P then saying that Process P also published the event would also be correct. Further, a process itself doesn't exist out of context. Every process must be contained within a Process App. For example, if Process P is contained in Process App A then that too would be a correct answer for who published the event.

To capture this information, each event contains a field called mon:model. This field is actually an array of all the identies that could be considered a publisher. For a process step, it contains an array member identifying the step, another array member identifying the process and yet another array member identifying the process app.

Note: Complexity
I'm far from being a BPM designer or data scientist but I am finding this array usage to be of the most complex areas to be considered when performing queries. The number of array elements varies and I'm not even sure that we should/could count on the order of the array elements.

Each element in a mon:model thus described a record describing part of the process model that issued the event. This record always contains a field called mon:type which is a discriminator that identifies what portion of the model we are looking at. Values are:

  • bpmn:process
  • bpmn:userTask
  • bpmn:scriptTask
  • bpmn:startEvent
  • bpmn:exclusiveGateway
  • bpmn:inclusiveGateway
  • bpmn:parallelGateway
  • bpmn:endEvent
  • bpmn:intermediateThrowEvent
  • wle:processApplication
  • wle:serviceFlow
  • wle:noTask

The mon:kind description of the event as a whole gives us an indication of what mon:model records we might expect to find.

bpmnx:PROCESS_STARTED

mon:model[0] {
  mon:type: "bpmn:process"
}
mon:model[1] {
  mon:type: "wle:processApplication"
}

bpmnx:PROCESS_COMPLETED

mon:model[0] {
  mon:type: "wle:serviceFlow" | mon:type: "bpmn:process"
}
mon:model[1] {
  mon:type: "wle:processApplication"
}

bpmnx:ACTIVITY_READY

mon:model[0] {
  mon:type: "bpmn:????"
}
mon:model[1] {
  mon:type: "bpmn:process"
}
mon:model[2] {
  mon:type: "wle:processApplication"
}

bpmnx:ACTIVITY_ACTIVE

mon:model[0] {
  mon:type: "bpmn:????"
}
mon:model[1] {
  mon:type: "bpmn:process"
}
mon:model[2] {
  mon:type: "wle:processApplication"
}

bpmnx:ACTIVITY_COMPLETED

mon:model[0] {
  mon:type: "bpmn:????"
}
mon:model[1] {
  mon:type: "bpmn:process"
}
mon:model[2] {
  mon:type: "wle:processApplication"
}

bpmnx:ACTIVITY_RESOURCE_ASSIGNED

mon:model[0] {
  mon:type: "bpmn:????"
}
mon:model[1] {
  mon:type: "bpmn:process"
}
mon:model[2] {
  mon:type: "wle.processApplication"
}

bpmnx:EVENT_CAUGHT

mon:model[0] {
  mon:type: "bpmn:startEvent | ... others"
}
mon:model[1] {
  mon:type: "bpmn:process"
}
mon:model[2] {
  mon:type: "wle:processApplication"
}

bpmnx:EVENT_THROWN

mon:model[0] {
  mon:type: "bpmn:endEvent | bpmn:intermediateThrowEvent | ... others"
}
mon:model[1] {
  mon:type: "bpmn:process"
}
mon:model[2] {
  mon:type: "wle:processApplication"
}

wle:PROCESS_AT_RISK_DATE_ASSIGNED

mon:model[0] {
  mon:type: "bpmn:process"
}
mon:model[1] {
  mon:type: "wle:processApplication"
}

mon:model records

mon:type - bpmn:process

  • mon:type - bpmn:process
  • mon:instance
    • mon:state
      • completed
      • Active
    • mon:id - Process instance ID (PIID)
  • mon:version - 2064.xxx
  • mon:name - Name of the process template
  • mon:id - Unique ID of the step (a UUID)
  • mon:documentation - Unknown

mon:type - bpmn:userTask

  • mon:type - bpmn:userTask
  • mon:instance
    • wle:taskInstanceId - The Task ID (TIID)
    • mon:id - Unknown
    • mon:role
      • mon:resource
        • mon:name - Name of group
        • mon:id - Identity of group
        • mon:documentation - Unknown
      • mon:id
        • POTENTIAL_PERFORMER
  • mon:version - 2064.xxx
  • mon:name - Name of the user task
  • mon:id - Unique ID of the step (a UUID)

mon:type - bpmn:scriptTask

  • mon:type - bpmn:scriptTask
  • mon:instance
    • mon:id - Token
  • mon:version - 2064.xxx
  • mon:name - Name of the step
  • mon:id - Unique ID of the step (a UUID)

mon:type - bpmn:endEvent

  • mon:type - bpmn:endEvent
  • mon:instance
    • mon:id - Token
  • mon:version - 2064.xxx
  • mon:name - Name of the step
  • mon:id - Unknown

mon:type - wle:processApplication

  • mon:type - wle:processApplication
  • mon:version
  • mon:name - Name of the Process App
  • mon:id - Unknown
  • mon:documentation - Unknown

mon:applicationData

For items such as tracking points, we emit a record of type mon:applicationData. Its structure contains:

  • wle:tracking-point
    • wle:groupName - The tracking group name.
    • wle:groupVersion - 2064.xxx
    • wle:id - An id for this event.
    • wle:time - The time stamp when the event was generated.
    • wle:groupId - A UUID
    • wle:version - 2064.xxx
    • wle:name - The name of the step that issued the tracking point.
    • wle:tracked-field - A record(s) indication that we have a tracked field.
      • wle:id - A UUID
      • wle:type - The data type of the field. For example:
        • xs:string - A string representation
        • xs:integer - A numeric type
      • wle:name - The name of the field
      • content - The value of the field

Notes: Take care with the wle:tracked-field. If there is just one tracking point variable, the field will be an object but if there are multiple tracking point variables, the field will be an array of objects.

References

See also:

No Comments
Back to top