JBPM Migrator

Overview

We are using the JBPM workflow library (version 3.2.2) on a project here at Intelliware. After some analysis, we chose JBPM as our process modelling tool because it was open source and it was easy to integrate into our technology stack (Java 4, Hibernate for persistence).

Over time, a process definition needs to change. Usually these changes reflect new business requirements, but they can also be related to a bug fix or an improvement in the existing process. So when we release a new version of a software product, we may need to update the older process instances in the database. Rather than provide a mechanism to migrate process instances, the JBPM library supports multiple versions of a process definition simultaneously:

Process instances always execute to the process definition that they are started in. But JBPM allows for multiple process definitions of the same name to coexist in the database. So typically, a process instance is started in the latest version available at that time and it will keep on executing in that same process definition for its complete lifetime. When a newer version is deployed, newly created instances will be started in the newest version, while older process instances keep on executing in the older process definitions.¹

This wasn’t very appealing to us. Our application has processes that can be resumed at any point in the future (potentially years later) so by following the JBPM prescribed approach our developers would have to support – and the QA folks would have to test – outdated process instances for years to come.

The Migrator

Given an old process instance, the migrator is responsible for transferring data to the latest process instance. The migrator transfers:

1. All of the tokens. This is facilitated through the use of mappings .

2. All persistent and transient variables.
3. It adds a migration memo (a String in the persistent variables map) to the new process instance which records info about the migration (the current date, the old process definition version#, the old process instance id , etc).

The migration is performed recursively, so each sub-process is migrated according to the above steps.

Mapping Token Nodes

One of the key challenges when migrating a process instance is the renaming or removal of wait state nodes. Wait state nodes (the green boxes in the diagram below) are where tokens reside when a process instance is persisted. Determining where a token should be placed is facilitated through a migration. A Migration contains a map that tells the Migrator where to put a token from a deprecated wait state node in the current process. Consider the following three versions of a Process called ‘Application’:

For the Application Process, two migrations would be written²:

1. Migration #1 (maps tokens from version #1 to version #2):
   

   {‘init’ => ‘start’, ‘invalid’ => ‘Requires Review’, ‘end’ => ‘application completed’}

2. Migration #2 (maps tokens from version #2 to version #3):
   

   {‘start’ => ‘application received’}

Our migrator takes these individual migration maps and creates a composite map. With the composite map, the migrator can map a token from any outdated wait-state node directly to the appropriate node in the current process. The composite map of these two migrations would be written as:

{‘init’ => ‘application received’, ‘start’ => ‘application received’, ‘invalid’ => ‘Requires Review’, ‘end’ => ‘application completed’}

Note that the map only needs to explain what to do with tokens on deprecated nodes (e.g. init, start, invalid, and end). No mapping is required for non-deprecated nodes (e.g. managerial audit, application completed, and Requires review). By default, if no mapping exists for a wait state node (i.e. it is not deprecated) the migrator will attempt to move the token to a node with the same name in the new version.

The example I am using includes a discrete migration for each version of the process but this is not always required. Depending on the changes being made to the Process Definition, it is possible that the developer will not be required to include a migration at all. This is a good thing. It means that we don’t have to write a migration for every single process definition change and there is almost no configuration required on behalf of the developer.

But there is a cost. Deprecated nodes can never be used in future definitions of your process. So with our ‘Application‘ Process, the init, start, invalid, and end nodes can never be used again in the process definition (as wait states). Doing so would break the migrator.

Defining a Migration

Migrations are written as Java classes. The class must implement the Migration interface, it must not be abstract, and it must contain a default constructor. The Migration interface declares one method that must be implemented:

public StateNodeMap createNodeMap();

Here is how you would express the first migration for ‘Application’ Process Definition example:

public class ApplicationProcessMigration001 implements Migration{   public StateNodeMap createNodeMap() {       return new StateNodeMap(new String[][]{               {"init", "start"}, {"invalid", "Requires Review"}, {"end", "application completed"}       });   }}

Defining a Migrator

How do we create the migrator and use it to perform a migration? Like this:

Migrator migrator = new Migrator(“ApplicationProcess”, jbpmContext, “com.foobar.ApplicationProcessMigration”);ProcessInstance newProcess = migrator.migrate(oldProcess);

The parameters used to create the Migrator instance are:

1. The name of the Process Definition that it will be migrating.

2. A JbpmContext instance. The migrator requires this to look up the latest Process Definition.

3. The Migration base class name. The migrator assumes that your migrations use the pattern package.ClassName{migration#}. For the base Class name “com.foobar.ApplicationProcessMigration”, the migrator will attempt to load and instantiate classes named “com.foobar.ApplicationProcessMigration001”, “com.foobar.ApplicationProcessMigration002”, etc, until it can’t find any valid classes.

The third point is another example of convention over configuration, resulting in less maintenance and tedium on behalf of the developer.

Unit and Integration Testing

I’m putting this section last, but it was one of our top concerns when considering an approach to migrations. We debated a number of approaches to testing and most of them were deemed to be too complex and error prone.

We already unit tested our JBPM process definitions to make sure that transitions point to valid nodes and that all actions declared in the process were available on the Classpath. With regards to the migrations, we have a base test that asserts that:

1. A developer has not introduced a deprecated node into the current process definition.
2. All current nodes in the composite map exist in the process definition.
3. All current nodes in the composite map are valid wait state nodes.

Our application is deployed several times a week to a test environment where QA folks test functionality and validate stories. This provides a great opportunity to discover problems with our application early, and the migration code is no exception. We write migrations throughout the entire development cycle (not just when we release) to make sure, so if a QA person loads up an outdated process from from last week, the migrator will be run. This gives us a chance to find runtime bugs that the unit tests can’t locate.

1. http://docs.jboss.com/jbpm/v3.2/userguide/html/jpdl.html#processversioning
2. I’m using the Ruby literal syntax for a Hash because there isn’t one in Java.

We wanted to use the java.lang.reflect.Proxy class to make instances of one (very simple) class assignable to different types of interfaces. Everything was going just swimingly and we were close to checking in our code. When it came time to run our integration tests, Weblogic began throwing a ClassNotFoundException whenever it attempted to de-serialize a proxy instance in a MessageDrivenBean. Once we confirmed that the interfaces were indeed on the Classpath, we fired up the debugger and dug a little deeper (but not until we consumed a a copious amount of Coca-Cola and Marlboros… thanks for the tip Mike). We discovered that – in the process of de-serializing our proxy object – the java.io.ObjectInputStream was trying to load our interfaces with the wrong classloader. It was using the parent of the Weblogic application classloader which had no knowledge of the classpath in our EAR file. We were were pretty certain that the reason was due to a bug in the way Weblogic was managing classloaders in it’s JMS implementation (of course, Weblogic is not open source so we can’t be 100% certain of this).

We couldn’t de-serialize the object ourselves inside the message bean and we couldn’t modify the way Weblogic managed classloaders, so we tried using the Cglib library instead. The Cglib proxy worked? The reason is because the ObjectInputStream de-serializes Cglib instances differently than it does a java.lang.reflect.Proxy instance. Why? Because of a key difference between the two types of proxy objects;

• When you create an instance using the JDK Proxy, it generates a subclass of the java.lang.reflect.Proxy class for you (the subclass is assignable to any interfaces you provide). Furthermore, the class generated by the java.lang.reflect.Proxy will have a different class descriptor (in fact, a Proxy class descriptor).

• On the other hand, the Cglib library generates a subclass of an arbitrary class that you provide (as with the JDK proxy, the Cglib generated subclass will also be assignable to any interfaces you choose to provide). The class generated by the Cglib library will have a regular class descriptor.

Whenever the ObjectInputStream de-serializes an object, it peeks at the Class descriptor. If it has a Proxy class descriptor, it invokes the ObjectInputStream.resolveProxyClass() method which creates a new version of the proxy subclass in the JVM (if it didn’t already exist) based on the declared interfaces. Because the Cglib class has a normal class descriptor, the ObjectInputStream.resolveClass() method is invoked instead. For reasons we’re not sure of, the classloader that is retrieved in the ObjectInputStream.resolveClass() method is the correct one and can resolve our interfaces/classes.

Now our Cglib proxy was being de-serialized correctly when sent to a JMS MessageDrivenBean. But we we were also sending these proxies to a rich client, and the Cglib proxies were not being de-serialized there. We were stunned, but only temporarily. This result was to be expected. The Cglib generated objects were instances of a subclass generated dynamically on a different JVM. Our rich client JVM didn’t recognize the class and was quite justified in throwing a ClassNotFoundException.

It turns out there is a pretty simple solution to this problem. Basically, you need to implement the writeReplace() and readResolve() methods on the object that your proxy delegates methods to. The delegate uses these methods to de-proxy and re-proxy itself depending on whether it is being serialized or de-serialized. In our case, our delegate object is the super-class of our Proxy instance, so we implemented the writeReplace() and readResolve() methods on it. Here is an example using a class called DataHandler;

public static class DataHandler implements Serializable {private static final long serialVersionUID = 1L;private final String someData;

public DataHandler(String someData){this.someData = someData;}public String getSomeData() {return someData;}

public boolean equals(Object obj) {return obj instanceof DataHandler ?      ((DataHandler)obj).getSomeData().equals(this.someData): false;}

public Object writeReplace() throws ObjectStreamException{return new DataHandler(this.someData);}public Object readResolve() throws ObjectStreamException{return ProxyFactory.createCglibProxy(DataHandler.class,                   new Class[]{IFoo.class},                   new Object[] {this.someData});}}

public interface IFoo extends java.io.Serializable{}

We create a proxy with the Cglib library like so;

Enhancer enhancer = new Enhancer();
enhancer.setSuperclass(DataHandler.class);
enhancer.setInterfaces(new Class[]{IFoo.class});
enhancer.setCallback(new NoOpCallback());//a NoOp callback. All methods will be invoked on the superclass
IFoo foo = (IFoo) enhancer.create(
new Class[]{String.class},
new Object[] {“someData”});//generates a subclass of DataHandler that assignable to the IFoo interface.

When the foo instance is serialized, the super-class (DataHandler) method writeReplace() is invoked. The writeReplace method returns a new instance of the DataHandler class, thereby discarding the Proxy wrapper subclass. It’s this new instance of DataHandler that gets serialized, not the original subclass. When the DataHandler instance is eventually de-serialized, the readResolve() method is invoked on it. The readResolve() will generate and return a new proxy subclass equivalent to the original . We learned this little trick by studying the code in the Hibernate project which uses Cglib for proxies and is open-source.

I wanted to use Rake as a build tool but there was an obvious hurdle to clear; all those Java specific tasks that we take for granted in Ant aren’t available in the Ruby libraries. For example, let’s say a Rake script needs to kick off a Java process. The core Ruby libraries make this possible:

system("java", "-client -jar lib/foobar.jar")

But the system call is not specific to the Java executable so passing in JVM and Program options can be awkward. In Ant, we would use the java task:

<java classpath="lib/foobar.jar" classname="foo.bar.FooBar" fork="true">

    <jvmarg value="client"/>    <arg value="argOne"/>    <arg value="argTwo"/></java>

This is the equivalent using Antwrap:

@ant = AntProject.new(:ant_home => "/Users/caleb/tools/apache-ant-1.7.0")

@ant.java(:classpath => 'lib/foobar.jar', :classname => 'foo.bar.FooBar',           :fork => 'true'){    jvmarg(:value => 'client')    arg(:value => 'argOne')    arg(:value => 'argTwo')}

The Task attributes are passed to the method as a hash. What would normally be child elements in XML are passed to the method via a block. The AntProject instance doesn’t actually have a java method defined. Rather, its method_missing method is invoked and ends up instantiating the appropriate Ant task for you. This is one example of Ruby’s powerful meta-programming features; they make it easy to provide rich semantics (as opposed to invoking @ant.executeTask(‘java’)).

Here is a more complicated example using the javac task. This simply illustrates that all of the normal Ant tasks are at your disposal, including Ant Properties and Refs:

@ant = AntProject.new(:ant_home => "/Users/caleb/tools/apache-ant-1.7.0")@ant.property(:name => 'common.dir', :value => @current_dir)

@ant.path(:id => "other.class.path"){    pathelement(:location => "classes")    pathelement(:location => "config")}

@ant.path(:id => "common.class.path"){    fileset(:dir => "${common.dir}/lib"){        include(:name => "**/*.jar")    }    pathelement(:location => "${common.classes}")}

@ant.javac(:srcdir => "test", :destdir => "classes"){    classpath(:refid => "common.class.path")    classpath(:refid => "other.class.path")}

The transition from an existing Ant build file to a Rake file is relatively easy. There is a handy conversion script available. So this:

<target name="clean" depends="init">

    <delete dir="classes" failonerror="false"/>    <delete dir="${distro.dir}"/>    <delete file="${outputjar}"/>

    <delete file="${output.dir}"/></target>

Becomes the Rake equivalent:

task :clean => [:init] do    @ant.delete(:dir => "classes", :failonerror => "false")    @ant.delete(:dir => "${distro.dir}")    @ant.delete(:file => "${outputjar}")    @ant.delete(:file => "${output.dir}")end

Antwrap runs on the native Ruby and the JRuby interpreter. If running on the native Ruby interpreter, Antwrap depends on the Ruby Java Bridge (RJB) Gem which invokes Java classes via the Java Native Interface (JNI). Antwrap is currently being used for Ant tasks in the Raven (a.k.a. don’t call me Maven) project. Raven is a JRuby implementation of the Rake tool that provides all kinds of utilities for a Java project.

In the long-term, I see Ruby/Rake scripts complementing a tool like Maven on Java projects. Ruby for fine-grained, project specific tasks and Maven for coarse-grained, boiler-plate tasks (compilation, unit-tests, packaging, documentation) that you need on most Java projects.

Other resources:
Martin Fowler on JRake
Groovy Antbuilder

When possible, you should try to use <macrodef/> instead of the <antcall/> task. By using <antcall/>, you’re invoking your target as if it were a task. This practice can become harder to maintain and result in performance problems in larger build files.

For example, here is a simple project file using the <antcall/> task to invoke the ‘foo’ target :

<project name="antcall">

<property name="A" value="A"/><property name="B" value="B"/>

<property name="C" value="C"/>

<target name="foo"><echo>A: ${A}</echo>

<echo>B: ${B}</echo><echo>C: ${C}</echo></target>

<target name="foo.call.1">

<antcall target="foo"></antcall></target>

<target name="foo.call.2"><antcall target="foo">

 <param name="A" value="eh?" /> <param name="B" value="bee" />

 <param name="C" value="sea" /></antcall></target>

</project>

And here is the same project file rewritten so that ‘foo’ is a task:

<project name="macrodef">

<macrodef name="foo"><attribute name="A" default="A" /><attribute name="B" default="B" />

<attribute name="C" default="C" /><sequential> <echo>A: @{A}</echo>

 <echo>B: @{B}</echo> <echo>C: @{C}</echo></sequential></macrodef>

<target name="foo.call.1"><foo/></target>

<target name="foo.call.2"><foo a="EH?" b="BEE" c="SEA"/>

</target>

</project>

The difference between the approaches is subtle but important;

• The ‘macrodef’ project is going to be more maintainable because the ‘foo’ task attributes are encapsulated (rather than using <property/> elements).
• Eclipse automatically detects the new task and provides the attributes via auto-completion. This saves a person having to scroll down to the target definition to determine what parameters for an <antcall> are required.
• The performance of the ‘macrodef’ project will be better than the ‘antcall’ project because the <antcall/> will actually create an entirely new project instance in memory. This is mentioned on the ant wiki