Comments on The IT Complexity Crisis: by Roger Sessions

Roger Sessions has written a paper on IT Complexity. When I met with Roger in London earlier this year (at the EAC) we had a short conversation, and it was clear we agreed on numerous things. This paper is useful because it provides a basis for more detailed discussion.

Roger’s paper:

Comments:

1. Cost of complexity is a little suspect to me. Primary issue is basing numbers on government data – government projects have high probability of failure, at least in the UK. More important I think complexity has more impact on full life cycle costs. Direct IT cost is one issue, but lost business opportunity is probably the major cost. Hard to estimate. However we can all agree, it’s a it’s a big number.
   
   
2. I am always wary of “functional decomposition”. It’s notoriously imprecise. At CBDI we recommend a combination of capabilities and business types (data). The outcome of Roger’s example SIP analysis looks similar to what would emerge from structured business type analysis, but in my experience the latter method is more reliable. More importantly I am vitally interested in developing an architecture that is a) demonstrably stable where it needs to be (managing business types such as customer, product and so on) and agile by design where it needs to be (managing process behaviour, transient data and events). I accept this may appear to be a form of religious debate, and we may have to agree to disagree, but I am interested to have the discussion.
   
   
3. But regardless of how you arrive there, Roger and I are completely in agreement on the need for components. The component must be completely encapsulated, own it’s own data store(s) and be the sole provider of owned services. Roger’s components appear to be capabilities. No problem there, because in the CBDI reference architecture there are options. But I would be looking to separate out layered behaviour so that business type components (stable) are separate from process and event components (unstable, subject to change).
   
   
4. The computation of SCUs (standard complexity units) is interesting. Roger’s methodology will lead you towards larger components with more internal dependencies. What’s the internal architecture look like Roger? The dependencies are still there. I am not concerned if the number of dependencies is very high, PROVIDING the dependencies are all well formed, reusable services and operations. Then it’s simply a question of ensuring you have good management (life cycle and run time) systems, which we do know how to do. I am more concerned by poor reference architecture (bad patterns, convergence of behaviors that should really be separate, bad or non existent contracts . . . )
   
   
5. Finally I put more faith in principles, patterns and reference architecture than in numbers. Numbers can lie, and they often do. Whereas patterns and reference architecture are the distillation of good (and bad) experience and provide intelligent people good guidance to become even more intelligent.
   
   
6. At CBDI we have developed methodology and process for modernization. We see real demand for rationalization and modernization of existing systems (full life cycle costs). The essence of the approach is to a) create the SOA façade and b) componentize. You might like to take a look at our method for CA Gen, it’s just one method for one environment, but it illustrates we [practice what we preach. There’s a slideshare there that describes.

Good stuff Roger, fully support the component approach. Happy to dialog in more detail.

SOA Manifesto Marketing Twaddle

Last week a small group of mostly vendor representatives plus a couple of analysts and authors came together last week in a seemingly self important manner to develop an SOA manifesto. Of course the primary reason for the event was marketing, and we shouldn’t be surprised that the result is superficial in the extreme. The primary worry I have is that they came up with a manifesto that actually confuses rather then assists in wider, deeper understanding.

CBDI members will know from our long standing research that the key to architecture, and SOA specifically, is to define principles that a) uniquely differentiate the architectural style and b) enable governance. I wonder how this group managed to omit crucial matters such as separation of provider and consumer, standards based and abstraction?

Meaningless marketing twaddle, but because the vendors sponsored this, political correctness will prevail. Yuck.

The Death of Enterprise Architecture?

- Introducing Smart Ecosystem Architecture (SEA)

In recent months there has been considerable debate about Enterprise Architecture or EA. Practitioners who have embraced EA are now hotly discussing whether EA is a business or IT architecture and the balance between the two domains. Without any doubt most EA today is IT focused even though practitioners know it shouldn't be. And the new EA standard TOGAF 9 merely reinforces this. Although there is a vocal lobby who would like to gain more power and influence in the enterprise by taking the lead in business architecture and design, typically they are unable to achieve this because they can't articulate or demonstrate why the business should let them.

Yet my own observations are that EA has not been an overwhelming success. The tensions between EA and delivery teams remain undiminished. EAs are frequently out of touch with delivery issues and do not command the respect needed to exert strong governance over the delivery work streams. From an SOA perspective, there are few enterprises that have delivered a comprehensive Service Portfolio Plan, let alone the implemented enterprise service portfolio. You may wish to ponder on whether this is a failure of EA or SOA. I couldn’t possibly comment!

Many years ago when we published the first CBDI Maturity Model we recognized that SOA maturity would be defined by the ecosystem. This maturity stage followed the enterprise stage. I will readily admit that in the early days the ecosystem stage was ill defined; in fact no one was interested. Apart from a small minority of enterprises that have always operated an ecosystem business model, the focus of attention was always heavily on the enterprise. Today we can see things have moved on apace. Various influences particularly Complex Event Driven Architecture and Smart Business and IT are strongly predicated on optimizing business design and processes involving all the ecosystem stakeholders. Examples:

- Airlines, airports, airport concessions, airport services, hotels and rental car companies form an ecosystem that optimize their resources and prices on a dynamic basis dependent upon external events, raw demand and actual traffic. Actually we wrote a research report around this model years ago – see reference.

- Smart power grids with tighter linkage between customers, suppliers, generators, assets and operations manage power supply to optimize use of green energy, price to consumers and profitability.

- Also see IBM's Smart Planet site. Reference below.

The emphasis on business optimization is very interesting. This means that conventional architecture domains of operational systems, business intelligence (BI) and management information (MI) become much more inter dependent and dynamic. New types of information, perhaps conventionally classed as operational, now become critical MI. Real time feedback loops require derived information to be available in the operational timeframe. Conventional (sic) SOA layered architecture policy requires extension to manage new patterns.

In these examples of the new enterprise each participant collaborates to optimize the overall ecosystem and in the process optimize their own position vs their collaborators. In this world the architecture is absolutely not enterprise wide. It is goal driven, focusing on a scope that is business results driven.

Does this mean that EA is dead? As discussed, in its current form EA has actually been less than successful. The current level of debate is merely confirmation of the patently obvious. Personally I recommend a more practical approach to architecture that is more grounded with stronger relationships and shorter feedback loops to business value and delivery projects. Given the current issues with EA it would be expedient to have unambiguous naming without the baggage of EA. I am temped to say that Business Architecture works because most grownups understand that Business and IT are indistinguishable at this stage of the game. But I see a lot of immature debate from EAs that want to elevate EA away from IT, which is plainly wrong. Also I don’t have a strong opinion on the "role" of EA. Rather I worry about the nature of the architecture. I want to get everyone thinking about a new perspective and scope.

On balance therefore I prefer a naming that emphasizes the architecture scope and deliverables, the differences from what we have been doing and the need for change. I suggest Smart Ecosystem Architecture (SEA).

References:

IBM Smart Planet

CBDI Report: Modeling for SOA - Worked Example - The passenger departure process - from arrival at the airport to boarding the plane.

CBDI Report: A Web Services Maturity Model

The Failure of SOA Standards!

You will have heard the joke before: “The nice thing about standards is that there are so many to choose from.” Sadly in SOA land this is reality.

The need for standards in SOA are multivarious including standard concepts so we all speak a common language, standard meta models that underpin profiles, repositories, deliverables, specifications etc. standard protocols that allow implementation and platform neutrality for service publication and consumption. In this note I am interested in concept standards that support architecture and the full life cycle.

In this area we all know we have three “horizontal standards” organizations (OASIS, The Open Group and OMG) that have each followed their own path, ignoring the blindingly obvious need to have consistency across their work. The situation is so ridiculous that the three organizations resorted recently to publishing a document to explain how to navigate around the SOA architecture standards. Curiously although the document is branded by the three organizations and acknowledges participation and contribution from all three organizations, the document is formally a joint OASIS and Open Group document. One might be forgiven for assuming they couldn’t agree the legalities of the collaboration. Also the document doesn’t address inconsistencies between standards and guidance produced by the same organization.

The document declares two goals – 1) to help users to understand the strengths of each body of work and select products appropriate for their needs. 2) to encourage consistency. But the latter is declared only as a secondary goal!

But inconsistency is only part of the problem. We must judge any standard in its support for the user community and crucially user adoption. The primary purpose of these standards under discussion is achieving common definitions of important concepts and simply put they have failed to achieve widespread adoption because of the competing efforts have confused the user community. Further individual efforts have in my opinion failed to meet user needs.

• While TOGAF is getting considerable traction our experience is that the parts of TOGAF that get used are the ADM and the contracts. Not the concepts and meta model, because they are patently not based on real world experience. They are both too verbose for some purposes and insufficiently detailed for others. And the guidance for use is non existent.
• Similarly the OASIS model is widely regarded as academic – not useful in practical application.
• I have more respect for the OMG’s work in SoaML, because it is usable in support of code generation, but it suffers from over generalization, which makes it very hard to use by mere mortals.

I observe that in the referenced “navigating” document the nearest the authors come to agreements on core concepts is at the most superficial level. I see no evidence that there is any real intent to deliver real convergence at a meaningful (i.e detailed, defined) level of abstraction.

Over the years CBDI has encouraged standards efforts; we have consistently made the case that common concept standards are essential at all levels of abstraction. We didn’t just sit by; we developed our own meta model which actually predates most of the work referred to above, and made it available to standards bodies. Frankly we fully expected that our work would be superseded by the standards work and that we would progressively align our SAE models to a converged standard.

Well you won’t be surprised to hear that the complete opposite has happened. This month we are publishing V3 of the CBDI-SAE Meta Model for SOA. We have actively participated and contributed to the OMG SoaML work and we are pleased to announce a high level of alignment. This is possible because of the highly generic nature of the SoaML work.

The CBDI-SAE meta model has always been very different to the various standards bodies work. We have focused on what may be termed a detailed Business Type Model (DBTM). The goals of the DBTM are to provide consistency of concept for use across the life cycle at a level that can be implemented in tools and inform practical architecture and full life cycle deliverables. And of course users of the CBDI-SAE meta model tell us that is exactly what they use it for – populating architecture and service repository schemas, underpinning UML profiles and defining detailed of deliverables that can provide full life cycle traceability and governance.

What we didn’t anticipate at the outset was that our meta model would actually form the basis for a further purpose – to provide a mapping between the other concept standards such that users forced to work with multiple conceptual schemas (few of us live in a homogeneous world) would have a basis for coordination and possibly transformation.

So as a user what do you do? Currently we see minimal use of these concept standards because they are inadequate. We recommend that you continue (or commence if you are a late adopter) using the CBDI-SAE meta model. We will commit to map to the other standards and, if and when they ever get to a credible position of convergence, we will certainly plan model convergence also. But we will need more than alignment of the concepts, we will need to see practical abstraction and detail required for real work.

We will be announcing a user review of the V3 model very shortly and welcome all input.

---

Navigating the SOA Standards Landscape Around Architecture, Copyright The Open Group 2009, OASIS 2009,

V3 SAE Meta Model note this is currently for CBDI subscribers only. Public domain version follows shortly for the review.

Don't Believe the Hype!

I note Gartner has published its 2009 Hype Cycle Report^[1]. I must admit I find it a very strange beast. It claims to be an evaluation of technologies. But it is plainly a mix of technology maturity AND technology adoption maturity. For example SOA is classified as being half way up the “slope of enlightenment” and 2 to 5 years to mainstream adoption. Yet it is very clear that SOA technology is reasonably mature at this stage, but it has equally obviously run way ahead of users’ ability to deploy it, because it is an architectural issue facilitated by technology.

The report also includes a technology referred to as Context Delivery Architecture (CoDA) as being in the Technology Trigger stage and less than 2 years to mainstream adoption. Two issues here. First so called CoDA is simply part of SOA. By Gartner’s own definition it’s about architectural response to the end user’s context such as location, preferences, identity, etc. and delivering the information that is most suited for it. Second the Differentiated Service pattern^[2] defined by CBDI in 2000 has high levels of support in many tools and platforms that are fully mainstream in terms of technology maturity. Yet Gartner believe CoDA will become mainstream in less than 2 years, even though they advise SOA will only become mainstream in the 2 to 5 year timeframe. Doh!

And I wonder how Location Aware Applications (slope of enlightenment) differs to Context Delivery Architecture?

And then there’s Cloud Computing. Currently shown by Gartner as being at the top of the “peak of inflated expectations” with 2 to 5 years to mainstream. Let’s examine this more closely. SOA is central to Cloud Computing. I note various commentators saying that Cloud simply enforces the encapsulation and virtualization principles of SOA. Of course there are new technologies in Cloud. Infrastructure as a Service (IaaS) in particular is key; yet this is still just SOA for infrastructure, which many vendors have already done a great job of cracking, and have many user deployments.

Web 2.0 is also on the list. Whilst Web 2.0 is a very ill-defined trend spanning many disparate technologies, SOA is clearly central to most of the use cases. Yet it’s shown in the Gartner report as towards the end of the “trough of disillusionment” and less than 2 years to mainstream adoption, ahead of mainstream SOA!

And then there’s the heterogeneous nature of the Hype Cycle Report. Because it includes such a wide and disparate range of technologies, it’s really more confusing than informing. Further it ignores one of the most important influences on maturity – standardization. Reading the report this year I was tempted to simply ignore it. But the inconsistencies highlighted above persuaded me that this report is actually quite dangerous guiding decision makers in highly undesirable ways. Listen to Gartner and you are encouraged to think that SOA is a technology. But the A is not just important, it's the architectural foundation to many of the evolving trends. Yet Gartner seem to place them all on the same level as individual technologies. I suppose the real reason why is because it plays to vendors marketing programs, of which Gartner is an intrinsic part. However I see it as distinctly unhelpful to enterprise customers.

My recommendations on hype cycle analysis are as follows:

- Decide which classes of technology you need to track in terms of technology maturity. Create a grid for this. Use this to inform R&D and Discovery projects.

- Decide which classes of trend you need to track in terms of adoption maturity (include architecture, practice, technology, products). Create a separate grid for this. Use this to inform project chartering decisions and governance review criteria.

- Change both grids to include an assessment of when important and relevant standards will be stabilized.

- Create dependency models of the trend areas, so you can understand the relationships between such clusters as SOA comprising Cloud, IaaS, EDA etc).

Clustering and dependency techniques are important. Not only do they allow you to plan sensibly, they also facilitate completeness reviews. And strangely I observe Gartner omit the area of semantic integration, a technology domain that I see as starting to become very important for many of our customers. Of course it's an integral part of the SOA cluster, and it's strongly driven by SOA maturity and the requirement to deliver on consistent information services. By chance I am publishing a report on the Progress Data Exchange Semantic Integrator this month, and I also note that last month Oracle acquired a provider of a similar product - Golden Gate. But the Gartner methodology apparently doesn't seem to surface such an interesting area.

The basic idea of a hype curve has been around for decades. I recall doing something very similar back in the mid ‘80s with an oil major client. But like all models, the deliverable is highly dependent upon some clear definitions and rigour that make the opinion based report credible and comprehensive. Sadly Gartner seems to have been caught up in it’s own hype.

---

^[1] Gartner's 2009 Hype Cycle Special Report Evaluates Maturity of 1,650 Technologies
http://www.gartner.com/it/page.jsp?id=1124212

^[2] Design Pattern: Differentiated Service, Fewer Interfaces than Components

http://www.cbdiforum.com/secure/interact/2000-12/design_pattern.php3

Repeatable, Reusable, Rapid?

I am intrigued by the amount of attention and support being given to “In Praise of Slow”. Originally published in 2004, this book by Carl Honoré is an entertaining and thought provoking commentary on our “culture of speed”. Honoré grabs our attention in the opening pages by describing how his lifestyle has led him to optimize his time with his son, searching out the shortest books for the bed time story, and pondering on why Snow White couldn’t have made do with 3 dwarves! His thesis is that we should pay attention to detail and do important things right first time. Like the slow food movement, set up originally to combat fast food, it’s about preserving culture, heritage, localization and small scale.

I am instinctively supportive of this idea of “doing things right”. In our own industry I worry about unfettered offshoring, agile development adopted purely for speed, the compromise of architectural principles trading short term gain for life time cost.

I note Ron Tolido’s blog develops this theme and extends the idea to Slow IT. Ron suggests “It is about using the principles of Enterprise Architecture to create a platform for continuous business change. This is not a paradox: only on top of a simplified, secure and flexible foundation of building blocks we can orchestrate and change solutions on a daily basis.” He calls it “Slow IT, the art of careful technology”.

I am completely with Ron in rejecting the superficial - Web 2.0, panic package acquisitions and the like for use in serious enterprise business processes. Yes we need to transition enterprise systems to modern componentized architecture that permits continuous upgrade of smaller moving parts.

However for all that I do believe that Slow IT is not going to go anywhere fast! We already have Slow IT today and the opportunities for misunderstanding are legion.

Last week I attended a presentation from Nick Cheetham at the Department of Work and Pensions in the UK. He commented that his organization is one of the few that right now is experiencing growth – because the unemployment rate is set to treble. But for most of us the imperative is to do more with less. And it’s interesting to observe different responses to this pressure.

I see tangible evidence that companies are increasing the rate of offshoring, in order to cut costs. I see others slashing the number of projects and programs, and focusing on the core business. But the primary observable effect is redundancy – reduction in head count, which is driven simply by the numbers. Then it’s up to the retained staff to figure out how to do more with less.

Maybe “slow” is an unintended but inevitable consequence of the current situation the archetypal enterprise is in, but I suggest a more appropriate focus is along the lines of “repeatable, reusable, rapid”. And this applies to everything – processes, services, components, infrastructure, skills, etc.

In this month’s CBDI Journal we publish a report on Implementation Architecture and Automation Unit Specification. This is an area in which we have been teaching and advising for some time, but we realized we hadn’t documented the guidance. It’s a critically important area – you have the business designs, the service specifications, so how do you deliver an effective software design and implementation, and demonstrate to governance reviewers that you are complying with SOA and EA principles and policy? And it’s a classic opportunity area for practicing repeatable, reusable, rapid techniques.

Like many folk, CBDI has been talking and advising on matters relating to repeatable, reusable, rapid for years. I seem to recall the phrase “reuse before you buy before you build” was coined by a colleague in TI around 1994 when we were developing the ideas around Component Based Development. It seems what goes around comes around, which perhaps says “good things come to those that wait”. But that’s different to advocating “slow”, which seems a bit like turkeys voting for Christmas.

Modernizing Legacy – Thoughts on Analyzing and Classifying Existing Application Landscape

The last few weeks I have been working on strategic direction for a government’s applications in a particular area of citizen support. It has prompted me to challenge some of our widely held assumptions about Legacy and how we manage it.

First what do we really mean by Legacy? The general understanding of legacy means something handed down from the past, an inheritance, gift or donation. Clearly in computing generally the term legacy has become synonymous with obsolete – still functional to some extent but does not work optimally with modern systems.

But this really isn’t adequate. Old doesn’t automatically mean obsolete – to be superseded. Old fashioned shouldn’t automatically mean redundant. What’s needed is a better taxonomy that allows us to communicate more meaning about the nature of the application portfolio.

I recall the presentation last year by Colin Smart of HBOS on their work in classifying their application landscape. Colin suggested an expansion of legacy to the following:

·         Strategic – everyone should use this

·         Retiring – no one should use this (but should use this instead)

·         Contained – no one should use this (but we haven’t an alternative)

which I like a lot. It immediately provides an answer to my basic concern regarding supersession and or redundancy. Incidentally the Contained class relates to our Chernobyl pattern – which we describe as “wrap in concrete, don’t expect to replace any time soon”.

But I feel we could go further in the classification particularly to integrate with the emerging SOA.

A significant part of the application area we were analyzing had been developed using the Information Engineering methodology and presumably at that time the IEF (Information Engineering Facility), now renamed CA Gen. A particular feature of this delivery approach was to establish an integrated, business driven data model which transformed directly to an integrated database. Now regardless of the issues surrounding a tightly integrated database, it was interesting to see how easily the aging application portfolio could map to the Core Business Service layer because the integrated, business context database effectively inherited an implied Business Type model (BTM). Rather than publishing Underlying Services, the CA Gen application would be easily able to publish Core Business Services that would be directly called by the Business Process layer.

I just happen to have been looking at a CA Gen application, but of course back in the 80s and 90s Information Engineering was widely used, and I wouldn’t be surprised if many legacy applications exhibited similar characteristics. However it must be said that many older delivery technologies wouldn’t have the same internal integrity as the IEF, and so any original quality may have deteriorated with time.

But if you do discover this, the alignment of the (very large) application area with the BTM also provides more confidence that the application area could be componentized – to break up peer to peer calls (between modules) and to turn those into service calls, thereby reducing the unit of change, deployment and release. And believe me, the example I am advising on is a BIG application area, so there is real value in doing this. The same action opens up the possibility of replacing or reengineering selected modules if appropriate – simply giving the customer more options on how to manage the application area.

So to return to my taxonomy question, I recommend that there are more useful classifications that would really help to better communicate what’s really going on in the application estate – Usage (as per Colin’s suggestion), Service Layer and Separation.

Usage	Service Layer	Implementation
Strategic	Core Business	Components
Retiring	Underlying	Componentized modules
Contained	Underlying	Monolithic structure

 

In most cases I would imagine Strategic applications would be supporting Core Business Services directly. Sometimes there will be situations where the API architecture is inadequate, and where a Strategic application is shown as supporting the Underlying Service layer, it serves to highlight the potential conflict. Similarly if a Retiring application is supporting the Core Business layer, then some urgent action is needed.

It might be expected that Strategic applications would have some level of componentized implementation. Conversely if a Strategic application has monolithic implementation architecture it highlights the risk, likely cost overhead and reduced options.

I would be very interested in hearing about other legacy classification systems that members have used.

---