Hello all,
I have been tinkering with creating a translator/compiler to convert IS flows (XML) to Java code. Does anyone have any interest in this and if so, I am happy to share the tool with you, once it is complete.
If its crazy, let me know, but I think this could useful in various situations and I know webMethods will never build it.

You feedback welcome,
Serge

Serge,

Given the oft-expressed opinion here of “don’t write it in java”, perhaps you could share some of your “various situations” in which generating java from Flow would be valuable?

Mark

Hi,
the first issue about flow is performance, I know many cases, where first we’ve developed a flow service, and than we had to rewrite it to java, because of performance. The java service was sometimes 15 to 20 times faster, and because most of the services that we had rewrite were, let say a bottle neck of our solution, the whole system was much faster.
And for sure that would be good to have a tool, that would automatically create java service from flow.
Unfortunetly, when you use java, you have to use some externall tool (like eclipse) to develop and debug your services and this is paintfull, because using 2 separate IDE for the same project is not easy. That would be good to hava an option to develop your own plugins for Developer, the plugins that I’m missing are:
CVS/SVN integration,
Java ADVANCED:) editor and debugger,
and some tools for refactoring of flow services,

Kasper

Kasper has mentioned one example, and I also agree with Mark. The intent for translating from flow to Java I think is quite narrow. I would only consider if there is legacy code which should not have been written in flow, and there was low skill levels in webMethods or if the entire system was being migrated off webMethods.
The practical difficulties of translating are considerable, depending on the how far the ‘porting’ has to go. A simple translator which gets it mostly right would not be impossible, but migrating a lot of logic (including adaptor logic, use of IS specific functionality) would be challenging.

IMO, it is.

The CVS integration might be nice, though many places get along fine without it–indeed several places don’t use a CVS at all.

I’m conflicted on the Java editor/debugger. On the one hand, it would be nice to have a better editor in Developer. And it would be nice to be able to step through Java services from time to time.

On the other hand, if I want to do Java development, I’ll use a Java development environment. IS is not a Java app server. It’s not a WebSphere or a WebLogic or a JBoss or whatever. IMO, there should not be any significant Java development when creating a solution for IS. The strength of IS is how quickly integration solutions can be built using FLOW. If one wants to use predominantly Java to create integration solutions, then IS is not the tool to use.

I argree with you, but if it’s needed to develop something in java because of performance, then lack of one IDE to develop and debug flow and java services is getting inconvenient.

Dont you think, that a real strength of WM is large number of adapters to diff. systems, and SOA?? not a FLOW by it self??. Maybe I’m wrong, but the most important thing is to design good services, with very well defined inputs and outputs, so that they can be reuse when needed. Than it’s not so important how those services are implemented. FLOW is very good language to do an orchestration of services and for mapping purposes, but if you need to develop a service with more advanced logic than FLOW is not so good any more.

Best regards,
Kasper

Kasper,

I agree with Rob,using Flow is the way to go in IS for faster development/debugging/logical diagam view/maintainence etc…

Ofcourse Javaservice is widely used for advanced logic,utilities (when no built-in flow functions available),So Java/Flow are sticky both have equal importance and if possible Developer plugin’s feature with (Eclipse/Java) most of folks waiting for it.

HTH,
RMG

This is my point of view, as well.
But I dont think that WM will provide anything like that in future, because of company policy:) The FLOW can work only on IS:) but Java services can work on any server:)…and from WM point of view, this is very important reason why they won’t do anything like advanced Java plugin.

BTW, can you list features that, in your opinion, make FLOW much better than Java??

Best regards,
Kasper

Some features are robust UI,debugging(step/trace) makes easier,visual diagram view,flow html view(sample documentation purpose),breakpoints,mapping maintainence
(support,new developers) etc…
Also ultimately JS helps a lot in creating misc utilities,advance errorhandling etc…

HTH,
RMG

As I mentioned, being able to step through a Java service from time to time would be useful. But the amount of Java to debug is usually minimal and relatively easy to deal with.

The number of adapters certainly contributes to the strength of IS. OnRamps, modules, etc. are all very good things to have. Does one choose IS over other solutions because of the adapters? Not anymore. Not often anyway. All the main integration products have plentiful adapters.

Good design is a good thing regardless of the underlying technology. IMO, services with explicitly declared inputs and outputs are a must whether an IS service is written in FLOW or Java. FLOW is easier to work with in almost all cases.

Define “advanced logic.” I truly have not come across an integration case where the logic could not be implemented using FLOW. Yes, sometimes I use Java, for things like containers, linked lists, etc. but those are almost always simply wrappers for core Java objects. And then I use those services within FLOW services that do the logic work.

Don’t get me wrong. Java code has a place in IS. But IMO, it shouldn’t be front and center.

I think save/restorePipeline is the best things about Flow. If you think about it, it’s ultra-cool being able to save and restore the state of your method call between any 2 arbitrary statements.

It’s so useful, I’m surprised mainstream IDEs / languages don’t offer similar functionality in an easy to use manner.

webMethods IS already implements a Flow to Java convertor… so maybe your translator lets them do it faster.

Or…
It would be ultra cool if your translated code could run independent of webMethods – with input/output signatures auto-converted to SOAP WSDLs, calls to pub.* services either replaced with native calls (eg: pub.math:addInts replaced with “+” ) or automatically calling more complex webMethods Fabric services (eg: pub.edi:*) via SOAP. This would let a company use webMethods programmers and the simplicity and debugging power of webMethods developer to implement a unified service development environment.

Rmg, but in my opinion, those are not features of FLOW, but WM Developer.

Best regards,
Kasper

I think that Sonam got to the point. When building solutions using FLOW, unfortunattly You have to stick with Wm IS, you (or company) doesnt have any possibility of switching to some other environment. That is why, a converter to more open language (i.e. Java) can be sometimes usefull.

To be honest, I dont recall a situation that one of my client would like to switch their existing WM solutions to some other platform, but maybe because there is no possibility.

This is interesting. If Java is really 15-20 times faster, then I think a better solution would be to compile your flow service right into a class file, but keep the source in flow for debugging/enhancements/documentation. I know that we don’t currently have the tools for that. It would require a behind the scenes process that would translate the flow to java, then invoke the java compiler. This is similar to some of the old CASE/code generation tools that were so popular in the 80’s and 90’s.

Where is this tool? Can you elaborate?

I think that Sonam thought about that flow is implemented in java. But it’s not the same as flow to java translation.

Best regards,
Kasper

Thanks for discussion Reamon,
From my expierience, there are 2 ways that you can develop services and solutions with WM:
or

1. the “DO NOT USE JAVA” - which means that services are written in FLOW, except those that have to be in Java (utils, performance, …, ) there was a lot about this on wmusers.com. In this case the programmer use WM Developer to develop services

either
2) you “DO IT IN JAVA” - then you write everythink in Java, in this case a programmer DO NOT USE Wm Developer to develop services, but a IDE like Eclipse is used.

The worst case is when one tries to mix those two. Then it’s very hard to maintance/test or debugg the code.

Best regards,
Kasper

This is indeed an interesting discussion. I’m revisiting many long-standing notions I’ve held about “proper” IS development.

Interesting. I’ve never come across an installation that took this approach. Why not use an environment that would more directly support this approach–e.g. WebSphere or WLS? If using IS, wouldn’t one need to do at least some work in Developer?

Hi Rob - no tool as such… just Flow code must, at some level, be ultimately translated to Java bytecodes.

Regards,
Sonam

I see. I have no idea about the underlying code and what it does but I’d venture to say that FLOW is not translated to Java byte code explicitly. Rather, my guess is that FLOW steps are interpreted to run methods that are already in byte code. FLOW is pretty much just about execution path control, copying variables and comparing them from time to time. I wouldn’t be surprised if FLOW is explicitly converted to Java, just betting that it probably isn’t.

Hi,
To be more clear, the second approach (“do it in java”), is quite new, but i’ve seen successful installation of such system.

There are at least few reasons for that:

• the installation was part of a bigger Integration solution, and all of its functions, had to be sean as WM services (some other systems invokes “java wm services”). This is main reason for not using JBoss, WLS, etc…

• the system was written in java because of some FLOW features : ) ( You know this:“it’s not a bug it’s a feature:)”):
  - lack of simple and clear scope handling (I know that there are walkarounds for that, but none of them is perfect)
  - the situations when everything looks fine but something doesnt work (i mean, the situations when you have to remap a step, to make it work:/)
  - it’s much easier to manage Java code (have You ever tried to merge a FLOW code in SVN or CVS?? possible to do , but not so funny:/)

• there is a lack of many refactoring functions like:
  - Developer doesnt give you any warnings about that your code it’s not good, i.e. your are using not “declared” variable; there is a variable on exit which is not declared, etc…
  - it’s not possible to rename one variable in many places, (i.e. You change the name of input var, then you have to change it manually in the flow)
  - lack of many search capabilities, (you can not find a string in a FLOW, (not even saying about many services:( ) )

  • etc…

OK, so that’s it for now:)

Best regards,
Kasper

Hate to keep harping on this in all my posts, but…there are no such things as “WM services”. webMethods is a company, not a product. There are couple of wM products that provide “services” (Servicenet, Glue, IS) so “WM services” is ambiguous.

This would seem to be a poor reason to implement IS services in Java. Why should “other systems” care whether or not a service is written in Java? If those other systems are using the IS API then I can see why they would care that the services were hosted on IS (services could be implemented using FLOW or Java) but if they were doing HTTP posts then they shouldn’t care about that either.

Can you elaborate on the scoping issue? Is this a pipeline management concern?

I’ve never tried to merge package contents at all. That’s because it’s difficult and in my opinion, unnecessary. Having multiple people change a package at the same time is a bad idea. I also feel it’s a bad idea for multiple people to be changing a Java class at the same–people do it because they can but IMO, this is a sign of relatively poor design where the scope of the package/class is too wide. Merging code should be avoided IMO.

Decent things to have for sure. Taken in their entirety, this was enough for your project to avoid using FLOW altogether?

I have to say that the “all Java” approach gives me great concern. My first reaction is that you’re using the wrong platform and should switch to WS or WLS. Writing services primarily in Java in IS is a fundamental mistake IMO. There should be relatively little Java in an IS-based integration solution. To quote Dan Green from a post years ago: “Without sounding melodramatic, Java developers can significant increase a project’s time-to-market and ruin your ROI…It is my belief that a Java developer will never excel with the webMethods Platform until he can let go of the Java mentality.”

First of all, the project and solutions that I’ve described it’s not main! I have not done it, (all solutions that i’ve done were almost totally in FLOW ), but I had opportunity to see it, and I was positively surprised.

You are totaly right, that was a shortcut for me which describes a service that can be INVOKED (using FLOW INVOKE, or Service.doInvoke) within IS.

Other system and solutions hosted on IS doesnt care how services are implemented, they just have to be hosted:) on IS (this is client request).

Yes, exactly.

Merging code should be avoided, but in larger project it’s hardly possible to be done.

As I said at the beginning it’s not main project but I could talk to people that have done it. And there was many small or larger issue (some of them I’ve pointed out in previous post) that made them use Java.

If you would ask me about using java in IS few months ago, I would say the same, but right now I’m a little bit confused. And the idea of my posts was that I wonted to see if there are some others that did/saw something like that. And if yes, what made them do this (switch from FLOW to Java and some other IDE then Developer).

Best regards,
Kasper

Interesting thoughts and interesting discussion. Thanks for sharing your points. A stronger developer environment for FLOW, implementing many of the items you pointed out, would definitely be a welcome change!

I think IS more than make up for this in its ability to validate input and output pipelines, very easily. And “there is a variable on exit which is not declared” is not at all a problem. It’s desirable as a matter of fact from a code-reuse perspective.

Thank ychang for reply,
but can you elaborate more, how leaving not decalred variable on exit can impact reusability of code??
IMO, using not declared variable can have very bad impact on code readability and managment, and on code reusability as well, because only a developer that did a service which returns some unspecified variables can reuse it:)

I know that in some cases it can be desirable to use unspecified variables, but a programmer should be awere of it (he/she is doing this intentionally). That’s why I’m saying only about a WARNINGS that a WM Developer would generate, not a runtime rule .

Best regards,
Kasper

Can you share some of those cases? IMO, it is never “desirable” to use undeclared variables across services. A service should always, IMO, declare its inputs and outputs and should never leave work variables in the pipeline. TN used to be one of the worst offenders of this and still continues today to use undeclared variables, which can be very confusing especially for new developers.

Most of the cases that I know were related to, let say “design mistakes”, and were some kinds of bad patches that had to be done:( i.e: the interface, from some reasons, couldnt be changed but the service had to use some variable that were on pipe but not declared.
I REALLY TRY TO AVOID those situations.

There is another situation which I know some people are using, but I’m not sure should it be done this way. I’m talking about services which are created by Modeler and correspond to process steps. In this case adding “technical variables”(which doesnt have any influence upon process logic) to steps can black out the process flow. In this case(especially if the variables are always on pipe and are not used in conditions or transitions) using not specified variables when implementing those services can be excused.

Best regards,
Kasper

A service should always, IMO, declare its inputs and outputs and should
never leave work variables in the pipeline. TN used to be one of the worst
offenders of this and still continues today to use undeclared variables,
which can be very confusing especially for new developers.

Very true Rob.

Not that this contradicts what you just said, but in rare cases it is necessary to have ‘unknown’ variables in the pipeline - generally when the variable names aren’t known at design time:

One example: a generic protocol handler service receives a document. It then may pass the pipeline to an arbitrary custom handler service (depending on the doc & partner profile ). Only custom services “know about” the custom fields – those fields transparently “pass through” the generic service’s pipeline at runtime.

Another example: in a service to update an array of TN extended fields from a webpage, the cleanest approach was posting in an array of name value pairs that looked like this:
‘TN_field_ID_1’ = ‘TN_field_value_1’.
‘TN_field_ID_2’ = ‘TN_field_value_2’.

However, ‘TN_field_ID’ are internal TN GUIDs which obviously are not known at design time. So I could not access them in FLOW unless I wrote some Java service that stepped through the IData pipeline. I eventually worked around this by passing in duplicate variables that looked like:

name=‘TN_field_ID_1’
value= ‘TN_field_value_1’.
name=‘TN_field_ID_2’
value= ‘TN_field_value_2’.

First, let’s make one thing clear webMethods Integration Platform is based on Java. The Built-in services it provides are java classes. So, if you write a code to add two numbers in a flow service, you would need to invoke a flow step called addInts. This step is actually a java class.

The webMethods Integration Platform is evolved from the early days and its flow language can handle any task you through at it. The only thing is how you approch the problem. Of course if someone is a Java Developer then it is natural to feel more comfortable in writing Java services.

The notion “DON’T CODE IN JAVA” should be followed until there is no choice but to write java services. One of the reasons to not to write java services is that webMethods (organization) does’nt support custom java services. So if your organization prefer to use java in IS then it should be able to support it.

On variables in pipeline is that if you declare a variable in the actual pipeline and not the Input/Output, WM Developer will automatically drop it. Try it yourself just create a variable and dont initialize it.

And anyone who claims that java drops unused variables automatically is mistaken but the matter of the fact is that the garbage collector is responsible for that. But when you are talk about variables in the pipeline, they are a static representation of the variable. They are already destoyed when a service completes execution. But they are visible in the pipeline (a static black box) for debuging purposes.

Agreed. The key word in that phrase is rare.

Kasper,

I’m sorry, I think I totally misread the section regarding undeclared outputs. No, it definitely is not desirable.

Somehow I doubt that having “migratability” in an EAI/B2B environment like wM is even desirable. If the environment is any good (like wM IS is), then obviously it provide services (API, foundation classes, what not) and supporting architecture (security, auditing, error-handling) that’s fairly unique to that environment.

If I were to write a service/code with an eye toward migrating later to say WebLogic, then that means I’d have to avoid using any component(s) specific to wM IS. Then all the reasons for using wM IS in the first place just disappeared – IS becomes just another Java AP server, and a fairly weak one at that.

I was talking about variables that are initialized, then regardless of that if variable is declared on output or not, it’s not dropped at the end of service.

IMO, you are mixing two things, memory management, and pipeline management, those are two separate things!.

Best regards,
Kasper

Kasper,

When you are working in wM Developer that is design time. And when the service is invoked on a IS that is Run-Time. In the run-time enviornment memory management including the data coming from the pipeline is done by a Java Garbage Collector. So in design time the variable you see are local to your pipeline. Pipeline management is closely tied to Memory management. Because what you put in the pipeline will be in the memory during run-time. And as I mentioned earlier the pipeline is static. So you have to populate un-populate(drop) it according to you requirement.

And the one most important reason for not droping variables automatically from the pipeline is that if you want to use the same value again in the flow. Or they are going to be used. As the pipeline is static the developer should know where to use a variable and where not to.

Hi Netcivilian,
unfortunately I cannot agree with you, because:
memory management - specifies how memory is allocated and when and by who is cleared.
The most common ways to do that are: explicit memory management (the developer is responsible for allocating and freeing memory like in : c++, c) and
implicit memory management, the programmer doesnt free memory, but there is something like Garbage Collector which frees it when it’s not used anymore like in Java, C#

pipeline management - specifies who have the access to variable, who and when can modifies it, and when the variable can be marked as unused.

These are the reason why the relation between those 2 things is very week. The pipeline management can be changed without the impact upon memory management and vice versa.

Best regards,
Kasper

For sure, it is not converted to java class. You can try to investigate a stack trace attached to an exception which is thrown from a Flow service - there are no classes there which can indicate that Flow had been precompiled.

Probably one of reasons why this is not precompiled is to make a particular flow service execution more secure and manageable.

Hello,
I like the idea of the translator. I would love to see it as part of a larger toolset to do on the fly java code generation. You could have a pane for the FLOW you were developing and next to it was another pane that showed a read-only view of the java that would generated. This would allow your FLOW developers to quickly come up with solutions and then do a copy and paste to a dedicated java service wrapper or have a “Save As Java Service” option.

Also, it would be nice to have interwoven functionality of java put back into the FLOW line of view. Things like other loop constructs are all that remain in the FLOW that I miss not working with. That is the only reason for some of my java services. Loop and Repeat don’t make the grade in that regard without enough complexity that can have you crying for it to stop.

I guess that the translator is not my main love as it would only pull me back to the dark side. But it could be helping if had things that can work in reverse to reseed the FLOW.

So a FLOW-JAVA-FLOW translator would be really great. It that scenario I could have:

Integer one = new Integer(1);
Integer two = new Integer(2);
Integer three = one.add(two);
IDataCursor idc = pipeline.getCursor();
idc.first("three");
idc.setValue(three);
idc.destroy();

Would translate into a simple service with two assignments in a map and one invoke of addInts. That is a possibility. That could also lead to having a “Save as Flow” option just like mentioned above going in the other direction. This will probably be forgotten before next week, but I just wanted to share for a moment. Good day.

Yemi Bedu

I Serge,
A late entry for this subject, I know, but I would like to contribute with my 5 cents.
About 2 years ago I had the very same idea to build a FLOW-to-Java service translator, for the simple reason of facilitating the job of making the overall performance of a system improve by translating the FLOW Services to Java correspondents, maintaining the original FLOW ones as ‘sources’.
The main intention was to automate the process and cut-short the translation effort.
The first thing I did was a Proof-of-Value to determine if it was worthwhile… and it wasn’t!
Java services effectively perform better in some specific constructs (such as LOOPs) but the translator would have to incredibly smart to go to the extend of replacing some service calls (e.g., math & string manipulation) to Java constructs and still be runtime compatible with the translated FLOW (e.g., you have to be aware that the management of the pipeline is completely different in FLOW services… and that the USE of SEQUENCEs and calls to other services add to the runtime differences).
I first tried to mimic in a Java pseudo-translation the way FLOW handles pipeline and service calls, and came to the conclusion that the FLOW service engine does a good job there and I would end having very little performance improvements. To really have performance improvements, you need intelligence to understand the logic on a FLOW Service (be aware of language side effects and other ‘tricks’) and translate that logic (not the language steps) to efficient Java logic.

Bottom line, I saw no value in the effort and I dropped the project.

Still, being performance improvement a target, I started a new project to help find bottlenecks and pinpoint services that would benefit from refactoring: a Service Profiler.

It is now a commercial tool and I’m not going to advertise it… but you can quickly find it by entering in the Google Search: webMethods “service profiler”

Regards

Hello,
The site is nice and the idea of the profiler is good. I could not find any links for downloading a trial version. Is that even a possibility? Anyway, would you like me to proofread your webpages and send you the changes? I am a stickler for that. Good day.

Yemi Bedu

Awesome. The info you shared on your experience as well as the description of the tool are great. Being able to identify bottlenecks is an essential capability. I’m e-mailing the company to get more info.

Thank you for the comments.
An evaluation version can be requested by e-mail (there’s one specific for the tool, but you can use the one on the contacts page).
The site is under review and some changes & corrections are already underway.

António