1.0 Overview

Domain projects are for sharing resources, details of it can be acquired from mule documentation. One thing to note is that domain projects only works for on-prem mule runtimes. It will not work for cloudhub. This article will demonstrate how shared connectors work during runtime.

2.0 Creating the Domain Project

First we need to create a Domain Project, to do this (in Anypoint Studio) you need to go to File > Mule Domain Project (as per Figure 2.0a).

Figure 2.0a

Then you would see a pop up dialog, at the pop up dialog key in the following details to create our first domain project (as per Figure 2.0b), click next and click finish.

Figure 2.0b

Once the domain project is created you will see the following (Figure 2.0c).

Figure 2.0c

Notice that at Figure 2.0c there is a commented area, this is where you put all your global connector settings.

3.0 Creating Mule Apps That Uses Domain Project

Next create a normal mule application project by going to File > New > Mule Project (as per Figure 3.0a).

Figure 3.0a

Name it demodomainapp1 as per the following pop-up screen in Figure 3.0b, click next and click finish.

Figure 3.0b

Once demodomainapp1 is created double click on the mule project XML and select the demodomain at the domain list box as per Figure 3.0c.

Figure 3.0c

Once the action on figure 3.0c is done, create the following message processors in demodomainapp1 (Figure 3.0d).

Figure 3.0d

The app in figure 3.0d would have the underlying XML code.

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd">
   <http:listener-config name="HTTP_Listener_Configuration" host="0.0.0.0" port="8081" basePath="/demodomainapp" doc:name="HTTP Listener Configuration"/>
   <flow name="demodomainapp1Flow">
       <http:listener config-ref="HTTP_Listener_Configuration" path="/app1" doc:name="HTTP"/>
       <logger message="Demo Domain App 1" level="INFO" doc:name="Logger"/>
   </flow>
</mule>

Now we want to share the http:listener connector for both app 1 and app 2, to do this you need to cut out the listener-config from the app 1 xml and paste it in the domain project. You would also need to replicate the xmlns:http xml namespace import to the domain project.

The mule domain xml file (mule-domain-config.xml) would now have the following code.

<?xml version="1.0" encoding="UTF-8"?>
<domain:mule-domain
       xmlns="http://www.mulesoft.org/schema/mule/core"
       xmlns:domain="http://www.mulesoft.org/schema/mule/ee/domain"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:spring="http://www.springframework.org/schema/beans"
       xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
       xmlns:http="http://www.mulesoft.org/schema/mule/http"
       xsi:schemaLocation="
              http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
              http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
              http://www.mulesoft.org/schema/mule/ee/domain http://www.mulesoft.org/schema/mule/ee/domain/current/mule-domain-ee.xsd
              http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd">

       <http:listener-config name="HTTP_Listener_Configuration" host="0.0.0.0" port="8081" basePath="/demodomainapp" doc:name="HTTP Listener Configuration"/>

</domain:mule-domain>

Do the same to create demodomainapp2, except here you need to delete the http:listener connector because it is already declared in the domain project.

Upon finishing creating app 2, you will see the following 3 projects in Anypoint Studio (as per depicted in figure 3.0e).

Figure 3.0e

4.0 Debugging The Apps in Runtime

If you want to run app 1 and app 2 in debug mode at the same time then you will need to run debugging at the domain project, to do this you must go to domain project, right click debug as > mule applicaiton with maven as per Figure 4.0a.

Figure 4.0a

When debug mode is successfully running you will see the following (Figure 4.0b) in your console logs.

Figure 4.0b

If you use postman to trigger app 1 as per Figure 4.0c.

Figure 4.0c

You will see that in anypoint studio your debug mode will stop the the breakpoint in app 1 (as per figure 4.0c)

Figure 4.0d

Now if you do the same for app 2, using postman browse to app 2 (Figure 4.0e).

Figure 4.0e

You will now see that anypoint studio has stop execution at app2 break point (as per Figure 4.0f).

Figure 4.0f

Now if you go to the console log in anypoint studio you will see the following log entries from both app 1 and app 2 (Figure 4.0g).

Figure 4.0g

5.0 Deploying them to On-Prem Runtime

The previous section demonstrated how to run applications via domain project in anypoint studio, here I would go through how to deploy the application into an onprem mule runtime (Mule runtime is available from the mule support site).

Figure 5.0a shows all the folders available in on prem runtime. Two folder to note is the app and the domains folder.

Figure 5.0a

You need to go to the target directory in the domain project and copy and paste the zip file across to the mule runtime’s domain folder as depicted in Figure 5.0b from left to right.

Figure 5.0b

Now do the same for app 1 and app 2, both zip files of app 1 and app 2 are available in their respective target directories (as per Figure 5.0c). After copying it to domain folder rename the domain zip file by deleting the “-1.0.0-SNAPSHOT”.

Figure 5.0c

Now when manual deployment is done, proceed to run the mule runtime. Execute the following command in windows command prompt (figure 5.0d).

Figure 5.0d

Once the on-prem mule runtime is started successfully you will see the following in the same command prompt window.

Figure 5.0e

If you test the app 1 and app 2 using the same postman commands, you will see the necessary log entries in their respective log files (figure 5.0f).

Figure 5.0f

6.0 Conclusion

I have demonstrated how to share the same http connector using domain projects, this is handy if you only want a single port opened for your on prem mule apis that is running on the same mule runtime. You could also share JMS or database connectors using domain projects.

1.0 Overview (the case for flow variables)

I came across a an exam question in Mule and pertaining to the relationship of Flow Variables and Batch Jobs. You need to pay attention to questions like this if you are to pass the mule certificaiton exams. When I try to google for the answer there is no article that has been written about it. Not in the mule documentation no any where on the web. So read on and get an understanding of how each type of mule variables behave in the context of a batch job (mule construct).

The question is something like this, given the following batch job.

Figure 1.0a

And given the following input payload, what would flowVar.aCounter be at the end of On Complete?

[
{
   "ID":"001",
   "OriginNum":"1"
},
{
   "ID":"002",
   "OriginNum":"2"
}
]

This is how your thinking process would be, the payload has an array of two elements, which means one instance of the batch job would run and both batch_step_1 and batch_step_2 would be executed two times (one time for each record).

If at the beginning of the batch flowVars.aCounter starts with one (1). Then theoretically flowVars.aCounter would have a total of 5 at the end of “On Complete” phase of the batch job, Right ? Wrong !!! that notion is totally wrong !!! The following is the XML mule configuration of the batch job.

A batch job is not a flow but it can reference and execute a flow. Each step in the batch job executes as an independant flow, so the scope of the flow variable is really constrained to the individual steps, after execution leaves a batch step the flowVars is gone.

2.0 The Real Answer

The following is logs collected if you run the batch job at Figure 1.0a.

[[batchplayground].HTTP_Listener_Configuration.worker.01] com.mulesoft.module.batch.engine.DefaultBatchEngine: Starting input phase
[[batchplayground].HTTP_Listener_Configuration.worker.01] org.mule.api.processor.LoggerMessageProcessor: Batch Job Input: [{OriginNum=1, ID=001}]
[[batchplayground].HTTP_Listener_Configuration.worker.01] com.mulesoft.module.batch.engine.DefaultBatchEngine: Input phase completed
[[batchplayground].HTTP_Listener_Configuration.worker.01] com.mulesoft.module.batch.engine.queue.BatchQueueLoader: Starting loading phase for instance '67b3a840-974e-11e7-956a-b62d20524153' of job 'batchjob_withnoInput'
[[batchplayground].HTTP_Listener_Configuration.worker.01] com.mulesoft.module.batch.engine.queue.BatchQueueLoader: Finished loading phase for instance 67b3a840-974e-11e7-956a-b62d20524153 of job batchjob_withnoInput. 1 records were loaded
[[batchplayground].HTTP_Listener_Configuration.worker.01] com.mulesoft.module.batch.engine.DefaultBatchEngine: Started execution of instance '67b3a840-974e-11e7-956a-b62d20524153' of job 'batchjob_withnoInput'
[batch-job-batchjob_withnoInput-work-manager.01] org.mule.api.processor.LoggerMessageProcessor: Batch Step 1 flowVars.aCounter: 2
[batch-job-batchjob_withnoInput-work-manager.01] com.mulesoft.module.batch.DefaultBatchStep: Step Batch_Step_1 finished processing all records for instance 67b3a840-974e-11e7-956a-b62d20524153 of job batchjob_withnoInput
[batch-job-batchjob_withnoInput-work-manager.03] org.mule.api.processor.LoggerMessageProcessor: Batch Step 2 flowVars.aCounter: 2
[batch-job-batchjob_withnoInput-work-manager.03] com.mulesoft.module.batch.engine.DefaultBatchEngine: Starting execution of onComplete phase for instance 67b3a840-974e-11e7-956a-b62d20524153 of job batchjob_withnoInput
[batch-job-batchjob_withnoInput-work-manager.03] org.mule.api.processor.LoggerMessageProcessor: oncomplete flowVars.aCounter : 1
[batch-job-batchjob_withnoInput-work-manager.03] com.mulesoft.module.batch.engine.DefaultBatchEngine: Finished execution of onComplete phase for instance 67b3a840-974e-11e7-956a-b62d20524153 of job batchjob_withnoInput
[batch-job-batchjob_withnoInput-work-manager.03] com.mulesoft.module.batch.engine.DefaultBatchEngine: Finished execution for instance '67b3a840-974e-11e7-956a-b62d20524153' of job 'batchjob_withnoInput'.
Total Records processed: 1. Successful records: 1. Failed Records: 0
[batch-job-batchjob_withnoInput-work-manager.03] com.mulesoft.module.batch.engine.DefaultBatchEngine:
[batch-job-batchjob_withnoInput-work-manager.03] com.mulesoft.module.batch.DefaultBatchStep: Step Batch_Step_2 finished processing all records for instance 67b3a840-974e-11e7-956a-b62d20524153 of job batchjob_withnoInput

From the logs you can see that the state of the flow variable flowVars.aCounter is not persisted when it traverse through the batch steps, but the state of flow variables when it was first initialized is being carried through to each phase and batch step. In short the flow variable flowVars.aCounter will always start with 1 in each batch step irregardless if the the batch step try to increment it’s value. And finally the “On Complete” phase still shows a flowVars.aCounter with the value of 1 which was what the variable was initialized with at the Input phase.

So if you ever get a question like this during any of your MuleSoft exams, and if the question ask you to state the value of a flow variable a the “On Complete” phase, then the answer is 1 if the flow variable was initialized with 1 (or zero if the flow variable’s initialized value is zero).

3.0 The Case for Record Variables

Figure 3.0a

Record variable is definitely inaccessible in the oncomplete phase. Users cannot declare record variable in the input phase, the reason behind this is that records only exist after the implicit load and dispatch phase. Figure 3.0a summarizes the availability of record variables, but they are a bit fuzzy about the actual scope/lifetime of the record variable.

To illustrate to you the life time of the record variable I have created the following flow. You can only start setting a record variable in a batch step, I have done it in Batch_step_1 (as per figure 3.0b).

Figure 3.0b

Notice that I have used the expression message processor instead of the groovy message processor to set the recordVars.rACounter, the reason being is that the Groovy shell in mule does not recognize recordVars (take note, so that you don't have to spend hours trying to figure out why).

You cannot access a record variable at the On Complete phase else it will give you a runtime exception. The following is the full batch job XML to show how to correctly use record variables.

   <batch:job name="batchjob_withnoInput">
       <batch:input>
           <logger message="Batch Job Input: #[payload]" level="INFO" doc:name="Logger"/>
           <set-variable variableName="aCounter" value="#[1]" doc:name="flowVars.aCounter = 1"/>
       </batch:input>
       <batch:process-records>
           <batch:step name="Batch_Step_1">
               <batch:set-record-variable variableName="rACounter" value="#[1]" doc:name="recordVars.rACounter = 1"/>
               <scripting:component doc:name="flowVars.aCounter add 1">
                   <scripting:script engine="Groovy"><![CDATA[flowVars.aCounter = flowVars.aCounter + 1]]></scripting:script>
               </scripting:component>
               <expression-component doc:name="recordVars.rACounter add 1"><![CDATA[recordVars.rACounter = recordVars.rACounter + 1]]></expression-component>
               <logger message="Batch Step 1 flowVars.aCounter: #[flowVars.aCounter + "\n"] Batch Step 1 recordVars.rACounter: #[recordVars.rACounter + "\n"]" level="INFO" doc:name="Logger"/>
           </batch:step>
           <batch:step name="Batch_Step_2" >
               <scripting:component doc:name="flowVars.aCounter add 1">
                   <scripting:script engine="Groovy"><![CDATA[flowVars.aCounter = flowVars.aCounter + 1]]></scripting:script>
               </scripting:component>
               <expression-component doc:name="recordVars.rACounter add 1"><![CDATA[recordVars.rACounter = recordVars.rACounter + 1]]></expression-component>
               <logger message="Batch Step 2 flowVars.aCounter: #[flowVars.aCounter + "\n"] Batch Step 2 recordVars.rACounter: #[recordVars.rACounter + "\n"]" level="INFO" doc:name="Logger"/>
           </batch:step>
       </batch:process-records>
       <batch:on-complete>
           <logger message="oncomplete flowVars.aCounter : #[flowVars.aCounter + "\n"]" level="INFO" doc:name="Logger"/>
       </batch:on-complete>
   </batch:job>

Listing 3.0a

The code snippet at listing 3.0a will generate the following output:-

[[batchplayground].HTTP_Listener_Configuration.worker.01] com.mulesoft.module.batch.engine.DefaultBatchEngine: Created instance a3e26950-9758-11e7-956a-b62d20524153 for batch job batchjob_withnoInput
[[batchplayground].HTTP_Listener_Configuration.worker.01] com.mulesoft.module.batch.engine.DefaultBatchEngine: Starting input phase
[[batchplayground].HTTP_Listener_Configuration.worker.01] org.mule.api.processor.LoggerMessageProcessor: Batch Job Input: [{OriginNum=1, ID=001}]
[[batchplayground].HTTP_Listener_Configuration.worker.01] com.mulesoft.module.batch.engine.DefaultBatchEngine: Input phase completed
[[batchplayground].HTTP_Listener_Configuration.worker.01] com.mulesoft.module.batch.engine.queue.BatchQueueLoader: Starting loading phase for instance 'a3e26950-9758-11e7-956a-b62d20524153' of job 'batchjob_withnoInput'
[[batchplayground].HTTP_Listener_Configuration.worker.01] com.mulesoft.module.batch.engine.queue.BatchQueueLoader: Finished loading phase for instance a3e26950-9758-11e7-956a-b62d20524153 of job batchjob_withnoInput. 1 records were loaded
[[batchplayground].HTTP_Listener_Configuration.worker.01] com.mulesoft.module.batch.engine.DefaultBatchEngine: Started execution of instance 'a3e26950-9758-11e7-956a-b62d20524153' of job 'batchjob_withnoInput'
[batch-job-batchjob_withnoInput-work-manager.02] org.mule.api.processor.LoggerMessageProcessor: Batch Step 1 flowVars.aCounter: 2
                                                                                               Batch Step 1 recordVars.rACounter: 2
[batch-job-batchjob_withnoInput-work-manager.02] com.mulesoft.module.batch.DefaultBatchStep: Step Batch_Step_1 finished processing all records for instance a3e26950-9758-11e7-956a-b62d20524153 of job batchjob_withnoInput
[batch-job-batchjob_withnoInput-work-manager.01] org.mule.api.processor.LoggerMessageProcessor: Batch Step 2 flowVars.aCounter: 2
                                                                                               Batch Step 2 recordVars.rACounter: 3
[batch-job-batchjob_withnoInput-work-manager.01] com.mulesoft.module.batch.engine.DefaultBatchEngine: Starting execution of onComplete phase for instance a3e26950-9758-11e7-956a-b62d20524153 of job batchjob_withnoInput
[batch-job-batchjob_withnoInput-work-manager.01] org.mule.api.processor.LoggerMessageProcessor: oncomplete flowVars.aCounter : 1
[batch-job-batchjob_withnoInput-work-manager.01] com.mulesoft.module.batch.engine.DefaultBatchEngine: Finished execution of onComplete phase for instance a3e26950-9758-11e7-956a-b62d20524153 of job batchjob_withnoInput
       _withnoInput-work-manager.01] com.mulesoft.module.batch.engine.DefaultBatchEngine: Finished execution for instance 'a3e26950-9758-11e7-956a-b62d20524153' of job 'batchjob_withnoInput'. Total Records processed: 1. Successful records: 1. Failed Records: 0
[batch-job-batchjob_withnoInput-work-manager.01] com.mulesoft.module.batch.engine.DefaultBatchEngine:
[batch-job-batchjob_withnoInput-work-manager.01] com.mulesoft.module.batch.DefaultBatchStep: Step Batch_Step_2 finished processing all records for instance a3e26950-9758-11e7-956a-b62d20524153 of job batchjob_withnoInput

If you observe both the values of “flowVars.aCounter” and the “recordVars.rACounter”, you will see that only record variable can transfer its state across batch step, but once it is outside of the batch process area (where all the batch step is defined), it is no on longer accessible by the execution context.

4.0 How about Session Variables in Batch Jobs ?

Figure 4.0a

Figure 4.0a shows that I have included the session variable with the fully qualified name of “sessionVars.SaCounter”, its behaviour is the same as a flow variable. In a batch job the notion of the context of scope for both flow variable and session variables does not work as if they are in a flow.

If you observe the log file closely the clues on why it is behaving that way is really obvious.

Figure 4.0b

From the logs you get to know that there is essentially 3 threads that is running. Setting of the flow and session variable happens at a thread named “[batchplayground].HTTP_Listener_Configuration.worker.01”, this is the receiver thread from the flow “batchplaygroundFlow”, because this flow has an inbound endpoint of HTTP it would automatically be an a flow with synchronous processing strategy. The other two threads “batch-job-batchjob_withnoInput-work-manager.01” and “batch-job-batchjob_withnoInput-work-manager.02” are disperae threads allocated by the mule run time to run both batch step 1 and batch step 2. In this multithreading environment both flow variable and session variables are not synchronized only the record variables are synchronized.

5.0 Conclusion

When using batch jobs, it always runs asynchronously and it always executes its batch steps in multithreading mode. Remember batch jobs are built for high throughput processing. If you want the behaviours of your batch job to be predictable always use only record variables because they are synchronized between each batch steps.

Journey to being a IT Polymath

Wednesday, November 15, 2017

Demonstrating Domain Projects in MuleSoft