Wednesday, March 19, 2014

Agile is not Dead, it's Morphing

I note healthy discussion around whether Agile is Dead [ref 1]. And while I may sympathize (sic) with many of the comments, particularly the commercial trivialization of education, the core issue must surely be the difficulty of adopting de facto Agile practices to support real world enterprise programs and projects. My experience is most of the advice and guidance out there is predicated on scaling the de facto Agile development methods. And this isn’t the best place to start.

An exception is Dean Leffingwell’s SAFe, [ref 2] which does introduce the idea of portfolio, program and project perspectives and intentional architecture. I recommend this framework as an intelligent set of practices, but for me it doesn't go far enough because it is still primarily about development practices. This is the core problem - that Agile is development specific and practices only. In the enterprise, Agile development needs to be an integral part of a bigger ecosystem spanning business design, architecture, requirements, modernization and operational transformation practices plus architecture, delivery and modernization disciplines.

I am indebted to Dave Thomas whose recent blog, [ref 1] includes a worthy successor to the Agile Manifesto, reducing the original, development specific values and principles to a minimalist, more generic set. He says; “here is how to do something in an agile fashion
- Find out where you are
- Take a small step towards your goal
- Adjust your understanding based on what you learned
- Repeat
And this is more useful in the enterprise context because it is relevant to a broader set of activities than purely software development.

The diagram below is an outline maturity model template for Agile in the enterprise. It suggests there are four key views that need to be part of the transformation.  In addition to agile practices we need to be equally focused on what elements of agile architecture are required for an enterprise. What the agile delivery framework is and how the existing application portfolio will be modernized to progressively eliminate the duplication and complexity present in every enterprise on the planet.

People & Process. Much of the dissatisfaction with Agile arises from the limitations of the basic Agile practices, and the need to compromise these in an enterprise context. Both DAD and RUP (yes it’s an iterative method) are examples of extended or hybrid practices that introduce coordination, phasing and other disciplines that are more acceptable in enterprises that require traceability, governance and compliance with pre-existing life cycle practices. Enterprise frameworks such as Leffingwell’s SAFe as discussed and Everware-CBDI’s SOAM are examples of frameworks that adhere more closely to the purity of Agile principles while addressing enterprise specific needs. SAFe provides a framework which is more strongly Lean, coordinating portfolio, program and project activity to meet agile release train demand. SOAM provides a complementary, full life cycle process framework for software service modernization and delivery.

Agile Architecture. There is a requirement to articulate the enterprise requirements for agility as a reference architecture for business agility. In today’s fast moving world core architecture for the business, services, implementations, technology and deployments needs to be:
- under continuous development using Agile principles
- derived from the assessment of business needs for response to change, and constantly updated to reflect competitive and technology opportunities and threats.
- mapped to service architectures, patterns, policies and modernization strategies
- modeled using MDA/MDD to allow delivery as consistent architecture runways for portfolio and demand management, programs and projects.

Agile Delivery Framework. Most enterprises have a well-defined delivery framework of tools, repositories, templates etc that are designed to support well established QA and delivery policies. This is one of the most common inhibitors that Agile projects in the enterprise have to  overcome. In an enterprise Agile context that framework must be realigned to provide maximum automation of life cycle management and governance so that key enterprise requirements for integrity can be met without loss of productivity. Similarly the development  activity must be structured so that developers can extend the architecture runway with business solution specific rules and behaviors in a managed fashion which preserves the integrity of the architecture. Everware-CBDI has pioneered this runway extension capability implemented as a model driven (MDA/MDD) capability in which the runway code is generated and provided to developers enabling very significant productivity gains in both forward engineering and even more so in iteration.

Agile Modernization.  Finally in an enterprise context the elephant in the room is the existing or legacy portfolio. Unless this elephant is addressed, the enterprise will continue to create more and more complexity, increase costs and reduce response times to change. What’s required is a discipline of continuous, Agile modernization. That means, using Dave Thomas’s minimalist manifesto [above] every portfolio item, program and project must include steps to find out the current situation and address minimum goals that reduce complexity and support a progressive modernization strategy. Without that, all enterprise Agile projects will remain narrow focus and simply add technical debt.

I suggest that while Agile is not dead in the enterprise it is certainly struggling to survive. This is because Agile practices alone will be suffocated at birth by enterprise realities of consistency and integrity; or turned into narrow focus, standalone projects; or morphed into BAU. I really don’t want to enter into a debate about nouns or verbs, life is too short.  IMO Agile has considerable momentum and it can be morphed into a ground breaking concept that delivers enterprise business agility.

1 Agile is Dead (Long Live Agility)
2 Dean Leffingwell SAFe

Thursday, January 16, 2014

Architecture for the Digital Business

There’s another metaphorical asteroid approaching the world. The last one (the Web) hit around 1991 and we are still struggling to come to terms with it. But the next one is an order of magnitude bigger - it’s often referred to as Digital Business. It’s not a bad name, but it doesn’t immediately signal some of the awesome impacts that will result. Essentially Digital Business is about Web enabling everything we do. Don’t be fooled into thinking this is about Google Glass, or the Smart Watch. It’s about Web enabling “everything”, from vehicles, power and water meters, appliances, houses, offices, driverless cars to traffic management systems, personal healthcare and anything you can think about. All these nodes become devices using sensors to communicate on a peer to peer basis via Machine to Machine (M2M) networks, and generate vast amounts of data that will revolutionize the way the world works! Like the Web, adoption will be slow at first, but the initial stages are already happening and we can expect it will be another 15 to 20 year change cycle which follows an exponential growth curve.

Don’t believe me? A really good example I just happened to see as I wrote this blog, is the smart bike lock from Lock8. The lock is paired with a mobile app and, lo and behold it’s paired with a peer-to-peer marketplace that a) alerts the owner if someone tries to steal the bike and b) tracks it if the thief manages to take it away, and c) even allows owners to loan or rent their bikes to friends or others registered on the service, and make a little cash on the side. Of course Lock8 also works in lots of other use cases. And just to reinforce the message, Lock8 have been hugely successful in raising significant funding through crowdsourcing!

It’s interesting to observe the media debate about Google Glass, which is strongly focused on privacy issues. Yet there is little or no debate yet about the Web enablement of almost everything, which has far more implications, not just for privacy, but also security, reliability and risk, as well as impacts on employment and changes in long established cost assumptions. It’s likely that Digital Business will trigger huge cultural changes. And that’s before you get to talk about humanoid robots to do our housework, look after children and older citizens. And yes they are coming also.

So what’s happening to the archetypal enterprise in this process of change. And what does enterprise architecture look in this changing environment? Of course technology is the primary stimulus, but we need to look at the social and economic effects of new technology enabled capabilities to understand how the enterprise may respond.

By coincidence I noted the Drucker Form met last month in Vienna and one of their major topics was Complexity. And there was fascinating debate about what complexity is, and whether it is increasing over time. And surely the Digital Business with the expected exponential growth in nodes, information and dependencies is going to be a major driver of complexity increase.

Roger Martin, of the University of Toronto, told the forum that one reason we feel overwhelmed by complexity . . . is because of a growing tendency to see and study problems within silos. The complexity of our world has actually not increased. It is just the unaddressed “inter-domain complexity” which makes us feel like everything has become more complex.

Don Tapscott, (of New Paradigm) was clearly responding to this issue of inter-domain complexity when he told the  conference - first, our institutions need to radically decentralize. Further, the future is not to be predicted, but to be achieved through a new dynamic paradigm including self-organizing, emergent and sometimes resilient networks involving millions of stakeholders. These networks embrace, active participation, uncertainty and constantly changing conditions, and they show great promise for solving global problems and governing an volatile and complex planet in the future. Command and control being replaced with self-organizing networks. Linear and non-linear organizations.

I particularly liked Tapscott’s analogy of murmurations of flocking starlings. Starlings somehow organizing themselves en masse to see off predators are at the opposite end of the (command and control) leadership spectrum from the arch bureaucrat. It reminds me also of the March of the Penguins. Did you ever see the huddle of thousands of penguins, their backs to the ice cold wind, continuously moving inwards and then outwards to allow all the group members to share the extreme exposure and protect the inner circle.

And of course this is what’s happening today. Self-organizing networks are forming everywhere, facilitated by the Web, and enabling collaboration and information sharing and transactions on a local and global scale without any central command or control.

So what does the Enterprise model of the future look like? In the diagram below I suggest the enterprise is a network of networks, a series of Capabilities that are supporting many Networks. The scope of the enterprise is the extent of interest in the Networks and Capabilities, not the boundaries of the enterprise as a legal entity. Participants are of course any form of Node, Device, Person, Entity that has a relationship with the Capability through some form of Experience and /or Channel. That Experience is provided by one or more Solutions, but we should plan that distributed Capabilities collaborate to share common assets on many levels. Note this is a very high level diagram, and you can safely assume that all the relationships are many to many. Further there is much more detail that is required particularly to make this a business AND IT model. But it is sufficient for my purpose in showing a core enterprise.

So what does the Enterprise Architecture look like in the fully distributed, self-organizing enterprise. The diagram below shows that the new EA needs to establish the basis for the federated business with architectures that are about managing the interconnections and dependencies, while allowing the distributed business to have as much freedom of action as is possible. The Reference Architecture , therefore defines the minimum necessary alignment. The Platform Architecture manages the essential business and IT interoperability and consistency that ensures the federation has minimum necessary cohesion. The Common Asset Architecture is then a portfolio of pluggable service components – that comply with the platform and facilitate sharing of common behaviors where appropriate, and enable solutions that have the necessary level of enterprise consistency together with required level of localization. Each of these architectures of course has the five views – Business, Service and Application, Implementation, Technology and Deployment. But this is a Federated Enterprise Architecture, that’s purpose is not to exert command and control over the leaf nodes, rather to facilitate the minimum necessary consistency to ensure the enterprise is a) in regulatory compliance and b) able to optimize cross enterprise dependencies, information and customer experience.

 In the Digital Business world Enterprise Architecture is vital, but it needs to be tightly focused on optimizing the distributed, federated business work, which means delegation. And here we have another cultural problem, because we have to embrace subsidiarity, the notion that all matters ought to be handled by the smallest, lowest or least centralized authority capable of addressing that matter effectively. And in general we are very bad at subsidiarity. Enterprises tend to prefer command and control as the default modus operandi.

The diagram below suggests that EA needs to undergo a significant transformation, moving from the As-Is in which the objective is frequently to seek the highest level of commonality to a To-Be model in which the aim is to organize the business as independent capabilities. Clearly this is much more than an architecture issue. It must be reflected in the way the business and IT are organized, transferring and delegating responsibility to leaf nodes, while establishing and governing the key policies that make the enterprise a coherent whole.

In my experience many enterprises have frequently responded to Web based business opportunities in a tactical, technology led manner. This has led to duplication of core business capabilities and, I observe, critical weaknesses in customer experience and lost opportunities. Many enterprises particularly telecos, publishers, video stores, music and media firms, have through competitive pressure come to recognize the strategic importance of the Web. In embracing the Digital Business, enterprises may be tempted to go down the tactical route, but this is likely to be a big mistake, as the network effect will become an integral part of the enterprise’s product and services.

It’s true that Asteroids don’t hit the earth very often, and when they do the impact is immediate and very dramatic. Digital Business isn’t going to be instantaneous, but the impact is likely to be colossal.  So the key message is - figure out your new reference model, get rid of command and control, but manage the cross network dependencies at all levels of the reference architecture - that's how you deliver real agility in the new world.

Drucker Forum
Blogpost: Agile Architecture
CBDI Journal Report: Capability Planning and Analysis

Tuesday, December 3, 2013

Systems Failure at RBS - Again

Subhead: Modernization is NOT Optional!

Once again customers of The Royal Bank of Scotland (RBS) and its subsidiaries Ulster, Natwest etc are in trouble! Unable to use their debit cards, or access their funds! And the new Chief Executive Ross McEwan admitted today that RBS has failed to invest properly in its systems for decades! RBS has given no explanation of the crash, except that it was not related to the high volumes of transactions on Cyber Monday.

Actually the root problem for RBS, and many other banks and other very large organizations, is NOT that they haven’t invested in their systems for decades. The problem is that they have allowed the systems to evolve without good architecture and governance. Banking systems in particular used to be leading edge. But over the years the core systems that actually run the bank, have become badly out of date. Instead of modernizing the core systems, many banks have responded to demand for new products and services by surrounding the old legacy systems with new bolt on systems. As a result the old systems are held together with sticky tape and sealing wax, and modified piecemeal to adjust to new requirements, and the new systems create more and more interfaces and dependencies (including duplicated, hard coded business rules), so that eventually the systems resemble a hair ball. And as you know, a hairball is a tightly packed cylinder of fur, which also includes all manner of foreign particles, which can be quite hazardous in humans and animals. In systems terms we would more usually refer to the resulting mess as a monolithic systems portfolio.

The root problem therefore is twofold:
1. The increasing complexity and risk inherent in making change.
2. Increasing horizon of any change.

And the issues become apparent either when extremely complex operational processes execute incorrectly, or when changes in process, software or environment  are made, often unavoidably without full knowledge of all the implications due to lack of experience, documentation or even code!

Although I have no doubt that in all the large banks there are fierce debates about the IT budget, strangely the solution to this problem isn’t about the amount of money that needs to be spent. Rather it’s about how the organization embraces a real modernization program, which is led by good architecture and managed with strong governance.

Sadly in the IT industry the term modernization has come to mean something quite specific – the re-platforming or technology upgrade of some application. Whereas modernization should really encompass a much broader remit including:
1. Establishing a lingua franca between business and IT so that modernization issues and implications are genuinely understood by both business and IT management.
2. Defining a business systems architecture that a) facilitates transformation with low risk and b) progressively develops an inherently agile architecture, that can evolve continuously.
3. A roadmap for progressively rationalizing the business systems portfolio to comply with the architecture.
4. Implementing an integrated business and IT organization that owns the business systems.
5. Implementing knowledge management around business systems that ensures integrity and consistency and ownership of processes, information and rules.
6. Implementing a continuous Agile modernization and ongoing evolution process in both business and IT.
7. Establishing coordination, governance and risk management that ensures architectural integrity at all stages of the life cycle.

It would be easy for business management to think that the sort of problems experienced by RBS and other banks as IT problems. The above list suggests this is nonsense. The only way to properly resolve the issues is for business management to accept co-responsibility for IT. And it’s no accident that the first point in the list above is to ensure that business management understand the issues they are dealing with, and can communicate effectively with the IT organization. This isn’t just attending an education class; it’s embracing a common language that allows critical decisions to be taken on a properly balanced understanding of business and IT assumptions, costs and risks.

Retailers, manufacturers, power companies, logistics companies, government departments etc etc should not be complacent or smug when they see the problems at the banks. In fact almost all very large organizations have these problems. And the message for all enterprises is Modernization is not Optional!

RBS Crash - Management Prefer Offshoring to Modernization?
CBDI Journal Reports on Modernization

Tuesday, November 12, 2013

Are Service Architects and Designers like Cobbler’s Children?

During October I ran a survey asking Architects, Designers and Project Managers about Service Specification practices. I have just completed the analysis and report, see link below. There are some interesting conclusions:
1. DIY is the dominant service specification approach.
Respondents report there’s a wide range of approaches to service specification in use, and the most common approach by far (50%) is “do it yourself”. Sure, DIY will mean that practitioners are like magpies, they pick up ideas from a variety of sources, and create their own capability. But this, by any measure, must represent a failure of industry standards. While consistency of specification meta data may not be a primary goal for designers and developers, architects should consider a range of risks including use of development automation support tools, standardization of technical components of outsourcing contracts, opportunity to find/reuse pre-existing services, and crucially standardization of tooling. Which takes me onto the next point . . .

2. Documents and Spread-sheets still dominate tool usage for service planning, reporting and management. 
In the early stages of service adoption almost everyone simply uses the tools that are available to hand. But as the service portfolio evolves and matures these tools become a strategic liability in areas of planning, reuse management, portfolio and asset management, governance and reporting. I note not one respondent mentioned the EA Repository in context with service support tools. And again this is closely connected with the third interesting conclusion . . .

3.   64% dissatisfied with tool support. 
There was a very high level of dissatisfaction with tools, particularly in the context of communicating service specifications between the different roles and groups across the organization. We all appreciate high quality service specifications are the result of multi-disciplinary efforts, spanning Business Analysis, Architecture, Design, Developers, Product Management, IT Service Management, Outsourcing, Procurement,  Governance etc . So this is much more than reporting, there’s an evident need for modern tools that facilitate the collaboration process.

See Service Specification Practices Survey 2013 - Report

Wednesday, October 23, 2013

Service Factory 2.0

In July 1989 the Harvard Business Review published a seminal article by Chase and Garvin titled The Service Factory [1]. They argued that “The factory of the future is not a place where computers, robots, and flexible machines do the drudge work. . . the next generation, then, will compete by bundling services with products, anticipating and responding to a truly comprehensive range of customer needs.”

Since the publication some 25 years ago, much has happened to validate Messrs Chase and Garvin’s thesis. The world has manifestly transformed into a “service based world”. By some accounts the service sector grew to over 80% of the US economy by 2000. There are various implementations of the Service Factory concept; we might observe that the ubiquitous Call Centers represent a scalable, repeatable factory model. Similarly in the technology world services, or APIs, are a central part of many IT architectures, and some firms have adopted a service factory model by separating service and solution delivery.

But it must be said that for most enterprises there remains a significant gap between the business and software services. Over 25 years ago Chase and Garvin recognized this as an issue. They use the example of how 200 years ago horse-drawn carriages were mostly made by craftsmen, and the most successful carriage maker was invariably the most accommodating for the customer. But as mass production overtook craftsmanship customers came to value standardized, lower price more than higher price, personalized products. Gradually manufacturing became separated from pre and post-sale customer facing activities. In recent years in some industry sectors, there has been a noticeable reversal of this trend, and mass customization is widely practiced. However in many industries, notwithstanding Agile methods focus on customer involvement, the design of business services remains a discrete activity from the architecture and design of IT services.

This is starting to change. The Mobile revolution and the Internet of Things will inevitably cause a convergence of business and software services. I have described this as “turning the business inside out”. In future it will be a very rare business service that isn’t delivered by a software service, at least as an option or complement. It’s a racing certainty that Call Centers and other conventional service delivery models are going to go the same way as Telephone Exchanges, Typing Pools and Airline Reservations Departments! In this fast emerging world, “everything is a service”. And this will have a profound impact on the shape of the Service Factory of tomorrow.

The Service Factory 2.0 will be a software factory separate from solution delivery projects. As discussed some firms already employ this organizing model in order to architect and design services to standardize core business information and rules for use in many processes.

The software factory concept has been in use for some time, particularly in conjunction with the Product Line concept, where common components are delivered using a framework of tools, repeatable processes and patterns, that are reused in solutions supporting the Product Line. The Service Factory 2.0 is a specialization of the software factory, with a framework that is specific to services. However the most effective service factory will also be a “business service” delivery model in which the life cycle of the business, software and IT service perspectives are integrated.

We can anticipate a number of critical innovations that will enable the Service Factory 2.0:

1. Everything is a Service All business capabilities are delivered as modular software services which are integral to the holistic business service offering.
2. Automated Requirements Capture. The requirements activity is automated in a manner that facilitates the involvement of the real stakeholders in the specification of the business, software and IT service, in a language that is entirely natural to them and to engage meaningfully in distributed Agile processes.
3. Automation of Common Service Patterns. The Service Factory frameworks that bootstrap service implementations will increasingly automate many of the common architectural and business model styles. Yet unlike the de facto application package products, the Service Factory will facilitate the extension and customization of the common pattern to allow competitive differentiation for the customer organization, as well as ongoing flexibility.
4. Service Portfolio Management. The Service Factory must support the management of the “business service” and provide tools that integrate demand management, architecture and governance over a composable and constantly evolving business and software service portfolio.
6. Pattern R&D. An important role of the service factory will be to continuously evolve patterns to enable and support innovation in styles of business service together with increased scope of automation coverage
7. Architecture Managed Iteration. Pattern based development delivers code that is always compliant with the architecture, but also with meta data that allows full automation of configuration management. This facilitates continuous, low effort iteration in the delivery, test, DevOps cycle and evolution in production.

In their paper, Chase and Garvin quote Drucker who noted that the imperatives of information-based competition are breaking down barriers within businesses and making functional divisions obsolete. To my mind, the primary characteristic of Service Factory 2.0 will be the realization of the business, software and IT service as a composite offering, architected, designed and delivered as a business asset.

Tuesday, October 1, 2013

SURVEY: Service Specification Usage

Question: What’s the difference between a Web Service, an API and an SOA Software Service?

Answer: The Web Service and the API are technical interfaces, that may or may not be well-formed services that comply with SOA principles of loose coupling, autonomy, encapsulation (of the service), reusability and composability. A well-formed Software Service will usually have some form of Service Specification that defines and governs the compliance with SOA principles.

From observation, the use of Service Specifications by architects and designers is highly variable. In our work at Everware-CBDI we have encouraged more formality of specification over many years in order to increase the quality of delivered services. We have done this by making templates,UML profiles and methodology guidance freely available. Today we are actively engaged in delivering automation methodology and tools at all stages of the delivery life cycle. We plan to deliver some specification capabilities on the same freely available basis. We are therefore interested to learn what others are doing and we are inviting participation in a survey to establish some independent data around how services are specified.

The survey is intended for architects, designers and project managers.
-  It should be very quick to complete; about 10 minutes.
 Participation is anonymous. Just click here to commence
If you would like to get more involved, please:
     a.      Join the CBDI Forum LinkedIn Group and or
     b.      Register with the Everware-CBDI site
We will keep these groups up to date and of course publish results of the survey in due course.
Everware-CBDI Rich Service Specification:
Download the CBDI-SAE Rich Service Specification Template and view the Service Specification Reference Framework

Tuesday, September 24, 2013

Agile SOA in the Digital Economy

Are you and your enterprise a prisoner of the past? I don’t mean legacy applications and technologies, I mean today’s business processes and applications. I work with many different enterprises and what’s common to the great majority is the centrality of business processes and applications, and the difficulty in evolving these existing solutions.

Actually I am frequently amazed at the understanding of many business managers. I marvel at how they use the lingua franca of their applications to describe their business. I will readily admit that when I first meet someone like this it’s a bit scary, because their vocabulary is like a foreign language. But frequently I find that it’s also a foreign language to their colleagues and it represents a rather primitive form of power play! Believe me. And that vocabulary of course often pervades the business process also. But the even scarier thing is that these organizations don’t realize they are locked into yesterday; or looking in the rear view mirror is you prefer. And just like Fred Brooks mythical beasts, struggling against the grip of the tar pits [1], they will eventually be overwhelmed by the complexity and their inability to change. Yes they may be delivering Cloud based Web and mobile applications to their customers, but are they just adding to the inherent business complexity?

I observe smart, successful companies making major mistakes as they enter the digital economy. First they set up an eServices project or division. This is treated as an innovation center and separated from the core business, in order to get to market quickly. But of course when they get to market the new products don’t integrate with the core business. Sure there’s application and service integration, anyone can patch old and new together at that level; but what about the way the business works; the business model and the vocabulary used, the opportunities for channel switching, and the development of distinctive sales and customer support systems and internal and external company culture that transcends the technology channels. And the ability to evolve the old and new in a way that they complement each other?