Resolving Application Fragmentation Without Soa

SOA is too big and too expensive regardless of what all the vendors tell you. Is it wise to trust the people who created the original mess with fixing it? Isn’t it likely that the typical large corporation will end up with a larger mess than before?

Every IT expert must be aware that developing a software product is an expensive proposition. Thus it is understandable that any software vendor will have to try and expand the lifetime of a product by selling it under a new name. The same is obviously true for an inhouse application.

So how can we utilize those applications and data without too much cost and effort? Using a clever mix of integration, consolidation and federation does the trick without the need for a fully blown SOA project.

Business service consolidation integration and federation is a difficult task, and given the data and applications explosion in most organizations, it will not get easier if you wait. Large organizations typically have large amounts of legacy data and numerous hard-coded processes because they typically buy the so-believed best-of-breed products that create the integration problem. SMB or small to medium businesses have less legacy data and thus focus on integration from a business intelligence perspective.

Five integration towers are to be considered: metadata, content, applications, business process, and user interface. Do not forget SECURITY on all levels. There are many offerings for each tower but it is important to consolidate on all levels of enterprise IT and not just on one of them. Only then a coherent view of the business can be given to the user as well as the customer. I call this approach a Business Service Approach or BSA. The key problem is that in the sense of consolidation ERP, ECM, CRM, and BPM products are legacy systems and have to be considered as expendable.

Integrating on any level lower than the user interface and customer service will come back to haunt the IT organization once again. Yes, I know that IBM and others talk about integrating on a business process level, but that it still not enough and a huge problem if it involves any Java coding at all. Yes, SOA as a concept is good, but it is nowhere near enough. Calling some other approach with business process SOA is just adding to the confusion.

Several vendors now support business process and application integration with a single product set, just business intelligence vendors fail to understand that business data only make sense to anyone from a business process perspective.

A business service approach has to provide users - and/or customers via the web - with a personalized interface to customer focused business services using data, content, business processes, and backend applications. At this point the user interface also has to support collaboration via email and other means. Setting up a portal without taking integration to that level will only highlight the lack of integration and reduce the possible benefits and thus the return of investment. It is at this point where the most agility in terms of solution is needed. The more rigid the user interface, the processes, data and content access is the less it will support corporate agility. Code a single line in Java for any of the five integration levels and there goes your so-wished-for agility!

It is PARAMOUNT that for all levels of integration – data, content, backend apps, business process and user interface – full lifecycle change management is available. All levels of consolidation/integration need integrated SECURITY as otherwise one cannot open up between levels. This is one of the huge issues still hindering consolidation today! What is needed is FULL consolidation and not just interaction that is smugly shown on a dashboard. Any change needed on any level has to be propagated automatically to ALL levels.

A business service approach as suggested here requires a powerful data federation technique, which in turn requires a metadata repository to provide the flexibility to define and use multiple data sources. A business service can then query any data source or business application at any time. Because of the metadata information, synchronized writes are possible when needed, but must be used carefully. Data federation can be slower in accessing than data consolidation or propagation but requires the least changes to current systems and provides the highest flexibility and the most accurate and real-time data access that offsets the possible additional runtime issue.

By means of the metadata the business service has access to semantic and model information and can merge data from different sources into a coherent image for the business user or customer without the huge overhead of consolidation or propagation. Model information can provide data access paths that the user can freely explore given the right authorization.

Data federation is also the best approach for existing content from archives or newly created content from the federated data that is transmitted back to the source application. A simple process oriented changed-data mechanism can ensure that data changes are written back to the data source with full conflict resolution. Time-stamps and versioning that are common in content management are here used for data records.

A solution than can perform data consolidation, propagation and federation from a SINGLE metadata definition is the best choice. A solution also has to perform data replication to remote locations and easily link to the data sources by means such as ODBC, JDBC, SQL or messaging services such as JMS, MQ-series, SOAP or others. Enforcing the use of SOA Webservices at this point will drive integration cost and time sky-high.

As you can see, much of that is not part of the typical SOA project, but these are the problems that have to be and can be solved without SOA overhead. These were the consideration that went into the development of the ISIS Papyrus Process and Content Platform.