Monday, January 5, 2015

Understanding Agile Adoption Failure

The most common concern our customers voiced in 2014 was the unexpected outcomes of Agile projects. They don’t talk about failure as such. But they do talk about loss of consistency; inability to govern; lack of coordination AND THE INCREASING TIME TO MARKET caused by these precise issues.

I was struck by the results of the Agile Adoption Experiences Survey 2014 published by Scott Ambler. The really significant result to me is that 40% of respondents rates their organizations adoption of Agile as neither a success or a failure. Add to this the categories of Failure, Great Failure and Too early to tell and you have 58% that are not successful! This synchs with my customer feedback referred to above.

The advice my colleagues and I give when customers approach us looking for answers to these questions, is to look at how architecture is integrated into Agile projects. And there are some key areas that we look for in our assessments:
1. Is there a good reference architecture and associated contextual patterns?
2. Are there clear policies attached to work products together with the rationale?
3. Are developers and architects working as a community of interest to evolve the reference architecture, patterns and policies?
4. Are the reference architecture, patterns and policies integrated into the tooling and the architecture runway?
5. Is the architecture runway model based – allowing it to provide a reusable design time platform to be evolved by projects?

Agile projects can be successful in an enterprise situation. But architecture and governance need to be coordinated for consistency and mechanisms (automation) enforced to ensure consistency.

I wonder why the Agile Adoption survey didn’t ask any questions along these lines?

Thursday, September 25, 2014

Agile Self Governance

I guess most readers of this blog know that Ireland went through a massive bust in 2008/9. The primary cause was a massive building bubble. And because the economy was dependent upon building related taxes, the crash was brutal. One of the side effects of the bubble was that building standards suffered. In one extreme case a block of apartments was constructed with no fire safety protection! There were many, many more prosaic examples. In my own case I had building works completed that had to be completely reworked within a couple of years! The reasons for the low quality were complete lack of governance. In the boom times many people set-up without adequate skills and many developers over-committed to deliver more work than they could manage competently. In Ireland, unlike say the UK, there was no independent building inspection regime. Architects merely had to give approval for work to proceed at the major stages. There was no independent oversight.

Today in the Agile project world the idea of self-governance is pervasive. But the parallels with the Irish governance regime in the noughties is too close for comfort. The Agile principles guide that projects should be built around motivated individuals, given the environment and support needed and trust them to get the job done. Further valuing working software over comprehensive documentation is effectively encouraging teams to dispense with transparency and traceability. While this may work in small scale environments, in a large enterprise the idea that all teams will be highly skilled, properly resourced and motivated contradicts general experience. So in many large organizations conventional governance is still a requirement that can be highly detrimental to the efficient Agile process.

In my last blog post I described a new Agile governance model that addresses the empowerment question with:
a) A defined Agile governance model
b) Defined principles and reference architecture that establish ways of working together with articulation of business value
c) Automation systems that progressively incorporate the principles and reference architecture into frameworks, tooling, design time platforms, deliverable profiles and knowledge management systems.
d) A Community of Interest (CoI) responsible for the governance system and communications
e) A communication system that ensures Agile projects are fully implementing the governance system, and providing at least retrospective feedback to the CoI that contributes to a common asset base as well as practice maturity.

And in this post I want to explore further the organizational issues. In the image I show there are three primary stakeholders in the Agile process. A Design Authority (DA), the Community of Interest (CoI) and everyone else; let’s call them the Crowd. And what’s really important is this is not a conventional hierarchy. The Crowd, developers, architects, product owners and business analysts are the drivers of the process, engaged on business improvement delivery projects. The CoI are also part of the Crowd insofar as they are also engaged on delivery projects, but they are individuals that have a further role of taking responsibility for coordinating consistency of the approach taken by delivery teams. As shown the CoI will be responsible for reference and enterprise/domain architecture and have an approving role for solution architecture and design, platform components and factory configuration. And the key point here is that all of these key deliverables start by being crowdsourced. The delivery teams are in the best position to establish solution architecture and design. The delivery team member with a CoI role is responsible for identifying and promoting candidate components of reference architecture, the design platform etc to the collective CoI.

The DA is different. It’s still not hierarchic, but it does approve demand shaping decisions, reference and domain architecture. This generally happens in response to delivery project concerns, for example when there are affordability or timescale issues with “doing the right thing”. You know the deal, if we don’t worry about technical debt, we can deliver on time and budget. But if we address the bigger architectural questions, and deliver solutions that will reduce complexity, reduce operating and maintenance costs etc, then the delivery date will be pushed out and additional resources required. Frankly this is the right time to address the value of architecture, because the questions are real.

So what I have described here, and in the last post, is an evolving governance model in which the definition of “self” is stretched a little to encompass the role. The Crowd has a major role in informing the requirements for policy and architecture as required for specific projects.  The CoI are responsible for synthesizing these requirements to meet some broader, longer running remit and the DA guides on bigger questions of architectural compliance from a strictly business value perspective. The Crowd and CoI are also responsible for automating policy wherever possible; selecting exemplar solution components for platform and factory implementation, so that the very best solution components become reusable as patterns, configuration items, coding standards, services etc.

Whether it’s in a building or software development context, simplistic self-governance doesn’t work in complex enterprise situations. However understand the roles and responsibilities, work bottom up with Crowd sourcing and you may just motivate a very large team by enabling them all to contribute to the greater good.

See Also: Agile Governance

Wednesday, September 3, 2014

Agile Governance

Last year the decision was finally made to mandate Agile across our enterprise. The decision was taken, even though there were many unanswered questions. The assumption was that forcing the migration, along with adoption of popular “enterprise Agile methods” would ensure resolution of the outstanding questions. In practice, Agile methods have been very effective in delivering specific digital business initiatives. But almost inevitably the distribution and delegation of architecture has resulted in duplication, inconsistency and increased complexity, across all project types including legacy and new projects. We are now concerned that we no longer have an effective governance capability. The question is how do we fix this without losing the undoubted benefits of Agile methods?“ Enterprise Architect, F2000 company.

Over the past few months I have heard this message over and over again. While Agile is being successful, it is increasingly in conflict with broader goals. And this is clearly becoming a major issue, manifest in increased complexity, horizon of change and coordination issues as well as inconsistent customer experience.  I am now regularly advising a practical approach to resolution by addressing from the governance perspective.

In many organizations governance is still practiced by phase or stage gate peer review, and Agile projects are forced to accommodate, which leads to WaterScrumFall or worse. But governance criteria and policies are often very weak anyway, out of date or non-existent. Consequently governance is frequently a matter of opinion and experience, highly dependent upon the experience of individual reviewers. As we all know, a basic principle of Agile methods is delegation of responsibility, and ideally we need to delegate governance to the Agile practitioners and teams. So the question is how to implement self-governance and ensure quality and consistency of governance?

I think it was an old John Cleese training film in which Cleese himself plays the part of a manager telling a subordinate that he is now empowered, and he scatters magic dust over him and shouts some magic words saying, “You are now empowered!” Clearly this isn’t any more useful than telling an Agile team that they are now self-governing! Rather we need to go back to basics and define and communicate what governance is required and provide Agile teams with guidance on what is expected.

That sounds good in theory, except that in practice no one is going to be able to accurately define all the governance requirements; certainly not in a fast changing, Agile business and technology environment; nor will Agile development teams be able to keep up with a bureaucratic regime that continually issues edicts that everyone is expected to adopt.

What’s required is a governance system that works in an Agile environment. The parameters of the Agile governance system comprise:
a) A defined Agile governance model
b) Defined principles and reference architecture that establish ways of working together with articulation of business value
c) Automation systems that progressively incorporate the principles and reference architecture into frameworks, tooling, design time platforms, deliverable profiles and knowledge management systems.
d) A Community of Interest (CoI) responsible for the governance system and communications
e) A communication system that ensures Agile projects are fully implementing the governance system, and providing at least retrospective feedback to the CoI that contributes to a common asset base as well as practice maturity.

Agile Governance Model

This is a much simplified Agile governance model. Key points to note are:
1. the centricity of the architecture runway, and the tight relationships between policy, reference architecture, reusable assets and the automation platform. 
2. variants guided by various dimensions of scope, which may include applicability, business or technology domain, or even the maturity of the business or technology domain to achieve compliance.

Principles  are a great place to start. Self-governance is going to be a key part of Agile governance, and if we can’t articulate and communicate what’s important then we are dead in the water. Some examples shown here:

Reference Architecture is a critical capability, defining the architecture styles and patterns and applicability. But reference architecture shouldn’t stop at models, it must be realized in code in the Design Platform – which progressively realizes the reference architecture as application level infrastructure reusable across multiple projects. The design platform is typically managed by the CoI, a collaboration of senior developers and architects that decide what should be in the platform and develop the models and code as exemplars that succeeding projects will be happy to use, customize and or extend. The design platform is therefore a critical governance tool that actively evolves, managed by the CoI and constantly challenged by project developers  to provide optimal solutions to delivering on the principles and reference architecture. As a by-product the mature design platform will also be a major productivity tool; for example in the Everware-CBDI Agile Service Factory, over 80% of the code for typical large projects will be automatically inherited from the platform.

As shown Principles are generic patterns or techniques that guide strong solutions to business problems. The application of Principles is achieved by Policies. But not your father’s policies! In many (most?) organizations Policies are outdated lists of standards. In Agile governance, policies should be a context based record of how the principles and reference architecture have been realized. Like principles, policies are not mandates from senior management, they are transparent  communications of pragmatic decisions made by the CoI on the best way of delivering an optimal result in a particular context, reusing tried and tested methods supported by existing architecture and design assets. This is therefore a continuously evolving body of knowledge, specifically tailored for one enterprise’s needs. Examples below. Note in particular the Policy Context that highlights applicability and exemption.

Many Agile teams are now using the Scrum of Scrums approach to coordination of multiple projects. This is a highly effective mechanism to manage the pan-Scrum backlog. However this coordination must not be confused with architecture realization. The Community of Interest is not a Scrum of Scrums, it is a group of the most respected architects and developers who will be active practitioners in architecture and development projects, who coordinate the realization of the architecture, the models and implementing code, typically in direct response to project demand, but involving CoI members as appropriate to review, refine and contribute to improve the solution, to be optimal, generic, principle and policy compliant. The Architecture Scrum may therefore on occasions be a series of architecture specific sprints, perhaps at commencement of new programs, or in response to significant new areas of reference architecture or design platform requirement. But in momentum situations the Architecture Scrum is more likely to be integral to multiple development Scrums.  

In a generic sense, governance is concerned with ensuring the integrity of the delivered product. This requires a strong focus on the architecture and how it is realized. As many organizations are now realizing, delegation of architecture in an uncontrolled manner is high risk. The approach outlined actually encourages delegation of architecture but to a coordinating body, the CoI, which itself is charged with supporting project demands and broader organizational objectives. But the approach outlined also recognizes that there needs to be explicit documentation of architecture principles and policies and their application in order to allow communication and review, and justification of business value. This is a necessary level of documentation needed to communicate to the many stakeholders involved. 


Reliance on opinions expressed on a case by case basis, or architect resource involvement in projects without the backing of strong, defined reference architecture, gives programs or projects far too much discretion. Whilst we may laugh at John Cleese’s magic dust, in practice the embedding of key architectural code into the platform layer actually does make governance considerably more effective. But even if it’s incredibly effective, it’s not magic. An effective CoI comprising the very best architect and developer skills available means all projects have access to optimal solutions as well as automatic compliance. Agile governance as described in this post is therefore not an extension of Agile methods per se, rather it is a bridge between Agile methods and agile architecture that defines and ensures desired outcomes, without compromising the integrity of the Agile process. 

Monday, July 21, 2014

Agile and the Fairy Godmother

Once upon a time, a long, long time ago, well some 20 years actually, some clever folk figured out a way of structuring work that was quite revolutionary. So revolutionary in fact that most people didn’t understand it. Now a few folk in a parallel world worked hard over the years to make this method work, and they invented more ideas, frameworks and tools. And every day they are constantly improving their productivity and quality; and many of them also use Agile methods, as they are entirely complementary to the revolutionary method. The method – Design by Contract.

However the mainstream of the market seems to be fixated on Agile methods, as being the metaphorical silver bullet. Yet we all can’t help noticing that Agile Land is not such a happy place. There is continuing debate about the difficulties of making the methods work in the enterprise; the fragmentation of the original Agile principles and the outbreak of religious wars. And I am minded to comment, again, that the root problem with Agile methods is that they are one-dimensional - solely focused on people and process to the exclusion of all the other opportunities to create agile businesses.

At the heart of the conundrum is the need to focus on some different questions. Such as:
- How can we reduce the amount of work that has to be done?
- How can we structure the outcomes (deliverables) so they are inherently agile?
- How can we structure the work so that there is real traceability between the intrinsic business model and the delivered systems and services?
- How can we ensure that overall technical debt is always reducing faster than new functionality is being delivered?
- . . .
You get the idea.

So back to my parallel world. In Design by Contract the business problem, typically the Use Case, Services and Operations are attributed with Pre-conditions, Post-conditions and Invariants (rules that must remain constant). These artifacts provide us with functional and design level specifications that can be produced in an Agile, iterative manner, that
a) Deliver implementation independent service specifications (descriptions if you prefer)
b) And therefore also for publishing API specifications
c) Form the basis for structuring the code that is fully traceable to the business model
d) Create inherently agile systems and service structures
e) Create the structure (stubs) for Unit, Integration, Functional and Regression testing
(e.g all conditions and rules within a given test scope must be tested)
f) Depending on the technology employed to define the conditions and invariants, (rules engines, pseudo languages etc) both code and test cases can be produced automatically.

I was prompted to write this blog because I happened to read Rob Marvin’s useful blog on testing, and his very interesting ideas for improving test productivity to keep up with Agile projects. In his piece he talks about automation, but actually seems to miss the opportunity to auto-generate the test cases. Even more important, he seems to believe that the current state of Agile projects is his benchmark that he has to keep up with. I would comment that state of the art Design by Contract projects together with model driven frameworks are delivering order of magnitude greater productivity with exceptional quality, and this ought to be the where testers should set the bar.

Sometimes it seems to me that the Agile methods community is rather in the situation of hoping a Fairy Godmother will appear, wave a magic wand and all will be well. Everyone will use Scrum like they ought to; enterprises will waive their awkward little requirements for inter-project coordination and all will be well. But while the Agile community continue to ignore the broader scope of agile enablers this day won’t come.

If you are interested in an example of Design by Contract at work take a look at the Agile Service Factory.

Wednesday, June 11, 2014

Mapping Agile Architecture

Jason Bloomberg recently published a mind map for Agile Architecture. It’s a nice map that sketches top level thinking and I welcome that. It prompted me to do a drill down.
Mind maps are useful in that they are, by definition free form and intended to support brain storming. The downside is obvious – they are generally inconsistent and cause modelers’ intense frustration! Caveat emptor over, I fully agree with Jason that we need a dual interpretation of Agile  - that is Agile practices and Agile Architecture, and I have written about this on many occasions. Also that the entire motivation is about business agility. On this last point my mind map is clearly a little more technical than Jason’s, and on reflection I think that is because it’s essential to converge the business and technology concerns.

For example, the map suggests a strong capability centric approach to interpret the business morphology. However this is insufficient; the technology must also establish appropriate levels of implementation independence that will facilitate the pluggability of business capabilities. Similarly you might think that considerations regarding the platform and delivery technology (such as MDA/MDD) are irrelevant to business concerns. However the platform and platform delivery technology are potentially massive drivers of rapid iteration and ongoing change, because they encapsulate common application level infrastructure and common services, so understanding the “business” standardization and localization model is crucial to delivering agility through this structure.

     Jason's Mind Map
Related posts: 

Tuesday, May 27, 2014

Open Architecture for the Public Sector

In his blog Richard Veryard relates how the public sector, in the UK at least, is moving slowly towards understanding the impact of Digital Government. He reports on a workshop he attended recently to discuss some of the architectural aspects of Digital Government, hosted by Skyscape. One purpose of this discussion was to feed into the Labour Party Digital Government Review, and possibly into the Labour manifesto for the next election.

I very much agree with Richard that the key to success in this area is multi-agency collaboration that delivers joined up government. Sadly the rate of progress is evidently very slow. Richard and I responded to a UK Government call for information on Shared Services way back in 2005. See below for link. Much of what we said at that time, based on work for the Danish Government, is still highly relevant - the need to develop joined up government architecture based on business driven SOA.

Today we might add:
1. While information sharing is clearly needed between silos, an (implemented) business capability architecture would be a better way to rationalize inter-government complexities, and lay a componentized, federated foundation for collaborations. The business capability based architecture also defines units of integrity that are ideal units of provisioning or acquisition which is essential in today's Cloud based, multi-source environment.

2. In his post Richard mentioned that at the workshop there was "considerable discussion about the role of Government in providing a platform, and whether the platform should be a Minimum Viable Platform (similar to the Internet) or provide added value." The idea of an MVP is very interesting as it parallels work that we (Everware-CBDI) are doing. In our Agile Service Factory we guide organizations to develop what we now call a Common Core, which handles all the standardized services and all non-functional behaviors. We are finding this is 80 - 95% of the code/effort/cost. This Common Core becomes the enterprise (or domain or capability) platform, which is managed as one or more product lines concurrently evolving with solutions, and providing continuous change with massive reduction in cost. But the primary value is the ability to exert standardization and governance over parts of federated solutions while actively facilitating localization that does not compromise the standardized components.

3. By focusing multi-agency inter-working on capability, we have a business driven approach that is inherently service oriented, and clarity of understanding around what aspects of the common platform should be shared both within and across agency.

Richard Veryard: Towards Open Architecture for the Public Sector
2005 CBDI Report Shared Services for the UK Public Sector

Thursday, May 8, 2014

Not Another Framework? Part 2

In my last post Oh No! We need another Practice Framework,  I developed the earlier theme commenced in “Beware the New Silos”. I argued that the widely used frameworks are narrowly discipline centric and actually inhibit cross discipline working. I described how my own firm’s experiences have led to the development of a de facto framework, (we call it SOAM) and illustrated how this is essentially a value chain commencing with customer demand and finishing with value add to some enterprise.

I ended by sketching some basic principles concluding that we need a new framework that is goal driven and incorporates the entire value chain of capabilities, which of course may selectively reuse some parts of existing frameworks. In this post, I suggest a strawman that covers a) principles and b) capability model.

Before diving into principles, it will be useful to declare some scope. Our framework has developed from working with larger enterprises, both commercial and government in the area of business service and solution delivery. All of these enterprises share common issues that they have extensive legacy application assets that act as a serious inhibitor to business change, and successive, narrowly scoped solution projects over many years have often resulted in great complexity and technical debt. It is also common in my experience that enterprise architecture functions are routinely bypassed or ignored; that Agile methods have been attempted and found useful on narrow focused projects, but because of the constrained focus, tend to increase overall complexity of the ongoing application asset base; that consistent customer experiences are commonly compromised by narrow focused projects; and line of business managers in large enterprises are frequently dissatisfied with IT application service support.

The objectives of the framework are to:

- describe practices relevant to service and solution delivery in the digital business environment
- achieve a balance between short term goals and longer term objectives
- support progressive transformation to an enterprise comprised of independent business capabilities
- facilitate continuous, short cycle time evolution of business capability
- progressively and continuously resolve legacy portfolio complexities
- enable rapid delivery at low cost without compromise in quality

Principles are foundational for any framework. 

Principles should be enduring and lead to both excellent policy communication and policy interpretation in everyday situations. I also find it useful to classify principles by subject.

Capability Model

In business architecture the capability model has become ubiquitous. And in thinking organizations I observe delivery of highly independent service and solution components that reduce dependencies and the impact of change, as well as mirroring the IT architecture on the business organization. Why wouldn't we use the same approach in defining a set of activities to deliver services and solutions?

If you are uncertain about the capability concept, it’s important to appreciate that the optimum business capability is one that enables:
- maximum cohesion of internal functional capability, plus consistency of life cycle, strategic class (core, context, innovating . . . ), business partition (global, local, LoB . . ), standardization, customizability, stability, metrics and drivers
- defined, stable dependencies that are implemented as services
[Further reading on capability optimization ]

In the capability dependency model below, the arrows are dependencies. For example, Demand Shaping is dependent upon Conceptual Business Modeling and Portfolio Management.  So this is not a flow diagram, rather all the capabilities should be regarded as iterative, I will come back and discuss how Lean principles operate in a framework like this, and as discussed above, highly independent.

Most of the capabilities in the model are self-explanatory. However some need explanation:
1. The Conceptual Business Modeling capability is the ability for business stakeholders to describe business improvement in conceptual terms. Many business people speak in solution terms. Most business requirements therefore surface as solutions, some more baked than others. Because the business stakeholder generally has the budget, the solution vision frequently drives and shapes the project with outcomes that frequently compromise the existing and planned portfolio. By educating business stakeholders to communicate in concepts, the opportunity is created to develop the business improvement idea without preconceptions of implementation or product, and to optimize architectural and portfolio integration. 
2. Demand Management is reasonably well understood. Demand Shaping is best regarded as a complementary capability that takes raw customer demand and decomposes into components such as pre-existing or planned services/APIs, considers opportunities for modernization and provisioning, and reassembles as a set of projects or project components that optimize the progressive development of the portfolio. Demand Shaping is primarily an architectural task, but should be run by a cross functional team including architect, product management, business design and technical expert roles. 
3. The Architecture Capability is shown as a decomposition of sub-capabilities, essentially one for each View, plus modernization. Whilst modernization is not classically an architecture view, there is commonly a specialist requirement for modernization architecture that will include identification of appropriate transformation and transitional architecture patterns. The primary objective of all of the architecture sub-capabilities is to define realizable structure to meet the demand and, as discussed above, to optimize opportunities for modernization and provisioning. While there is no explicit enterprise architecture View called out, each architecture capability should be executed separately and iteratively for reference, portfolio, program, project and module, thereby defining progressive layers of standard functionality that will be common to the defined scope, as well as situation specific business functionality. 
I will detail all the capabilities in a subsequent post.

Final remarks. 

This high level view of the framework has attempted to list a set of principles and associated capabilities required to support the value chain illustrated in Part 1 of this extended blog post. What will hopefully have become clear is the need for architecture capabilities particularly to be involved throughout the value chain. This approach integrates all types of architecture (enterprise, service, solution, deployment  . . . ) into the business improvement value chain and creates better opportunity to demonstrate the ROI on architecture. Further the approach prevents enterprise architecture particularly becoming divorced from mainstream business improvement and encourages a better balance of short term and strategic goals. What will not yet be fully clarified is how the framework is very strongly focused on realizing architecture in delivered services and solutions, as a series of successive collaborations. I will describe how this is done using a Lean approach in a subsequent post. 

                  Beware the New Silos