Wednesday, November 14, 2012

Agile Architecture

The English language is well known for its subtlety. Sometimes it’s a delight, but on other occasions it can be very frustrating. If I use the term Gothic Architecture you will immediately understand I am describing a style of architecture that flourished in medieval times. And if like me you are interested in ecclesiastical architecture you will know that this style was used in many of the great cathedrals and churches across Europe, which were distinctive because of key architectural patterns that enabled great increases in height and internal light of the buildings without increasing the size of supporting pillars.

Now if I use the term Agile Architecture, what am I referring to? In today’s Agile world I would hazard a guess that most readers will think I am referring to the architecture techniques and tasks undertaken in the context of an Agile software development project, not the collection of patterns and practices that enable agile business systems. That is, an architecture that enables agility.

This potential for miscommunication is a core issue for enterprises. There is ample evidence that Agile Architecture is a primary contributor to business agility, yet we do not have a well understood architecture management system that integrates with Agile methods.

 Let’s use an example readers may be familiar with. Amazon CEO Jeff Bezos famously[1] issued an edict that laid down some key architecture principles to Amazon development teams that I will summarize as:
·       All teams will henceforth expose their data and functionality through service interfaces.
·       Teams must communicate with each other through these interfaces. There will be no other form of interprocess communication allowed.
·       It doesn't matter what technology they use.
·       All service interfaces, without exception, must be designed from the ground up to be externalizable.
·       No exceptions.

What Bezos did here was to lay down key business and technology architecture principles that you might reasonably conclude were central to the extraordinary level of business agility that we have seen demonstrated by, Inc. That widely circulated edict contained the foundations of the Amazon reference architecture.  

In the October 2004 CBDI Journal[2] we commented, “Two of the most successful and enduring dotcom start-ups, Amazon and eBay, now expose their core applications as Web Services. In doing so they have created a new class of platform that could have a profound impact on end-user organizations and IT vendors alike.”

And so the reference architecture became the enabler of growth and agility for the Amazon business, not we understand[3] as a grand plan, but through natural technological evolution. The services formed the platform that allowed the extraordinary expansion of the Amazon business that I would be certain not even Jeff Bezos imagined, back then in 2004. That is real business agility, and it was delivered by smart architecture backed up by clear policies and realized by agile processes.

Although Amazon has clearly evolved in pursuit of solutions to specific business opportunities and challenges, it’s also clear they have established a de facto architecture and architecture management system that guides the work of the many product delivery teams and ensures consistency of approach where it’s required. Let’s consider how an enterprise might establish a similar agile architecture management system.

A reference architecture articulates primary principles that are typically central to an entire enterprise. Principles should be focused on establishing the product and solution independent environment in which agility can be delivered and maintained, so they would be stable over time. We might refer to reference architecture as a Level 1 architecture perspective (L1) that exists purely as a set of models and guidelines.

Larger enterprises should explore the business value potential of platform based architecture as a mechanism to deliver cross enterprise consistency of core reference architecture behaviors and to enable closer integration with the wider ecosystem including customers, suppliers, end consumers etc. This is an extended management services platform which encapsulates the technology infrastructure and enables rapid delivery of business services.

The platform architecture defines common services that manage business delivery including security, life cycle management, change management, release management and operations, as well as catalogs, eCommerce, B2B, regulatory control and risk management, standardizing these key capabilities and reducing the footprint of business domain services. The platform will also manage important behaviors that deliver on specific business goals such as scalability and availability. For example, Amazon services are usually very fine grained, specifically to reduce the scope of each service in order to facilitate narrow focus SLAs and maximize scalability by reducing individual service complexity. We might refer to platform architecture as a Level 2 architecture perspective, engineered to be relatively stable in support of  large numbers of business services and consumers, but also engineered to evolve and respond rapidly to business and technology change. Not all enterprises will see business value in making their platform and business services available to their ecosystem, but some will.

Enterprises clearly vary considerably in their make up in terms of geographic and organizational, product and process standardization and differentiation, but typically there will be considerable potential for an inventory of shared assets that leverage agile architecture to support business agility. The assets may include:
·       Common services, frameworks and components that are designed to deliver common behaviors to all parts of the enterprise. For example core services that establish genuinely enterprise wide services such as Customer, Ticket, eCommerce etc; services that deliver business value by standardizing common business services and processes.

·       Configurable services, frameworks and components that are designed to provide common behaviors but are engineered to be customizable in local situations to accommodate many aspects of localization ranging from the simple – taxation, geography etc, to the complex – variant ordering patterns, variations in event and process sequence dictated by local de facto business practices. Configurable services may provide business value simply by providing reusable components, or they may establish a common core of business process and information that establishes common reporting and regulatory control in a local context, or both. Configurable services may also be an important time to market strategy for service providers who customize their services for each client or customer group.

·       Information architecture and services. Establishing a coherent approach to information is commonly a major issue for large enterprises and this architecture level defines an integrated approach for structured and unstructured (big) data, transactional and reference, enterprise reporting and regulatory control and so on.

Common and Configurable assets together with the Information Architecture might form a Level 3 architecture perspective and be widely applicable across a large, distributed enterprise.  

We then have two further levels which are closely related, Family Architecture and Product Line Architecture. Whilst many architects chose to view Family and Product Line as synonyms, I recommend that they are kept separate. A Family architecture is a domain framework that is much more specialized that L3 assets that would be applicable on a broader basis. The Family architecture establishes core business (domain) services and possibly other artifacts specific to the domain, where the domain is likely to be a subject area or a cluster of major types. For example Customer, Supply Chain, Manufacturing, Risk etc. Families are also commonly acquired products.

In contrast Product Line architecture is what it says – it’s the architecture for a product offering. The product is an offering that has direct relationship to end customer revenue and usually continuity of purpose over multiple releases. Although from a narrow technical perspective the Product and Family architectures might be similar, the way a product is managed must mirror the business product life cycle. Family architectures may therefore be engineered for stability, whereas, depending on the industry sector, product line architectures may be engineered for maximum agility and minimum response time.  

Finally we have the Solution architecture level, the architecture specific to solution project delivery, where the focus is on feature architecture and integrating solution architecture with the Level 1 to 5 architecture perspectives. It’s important to note that where product line architecture is used, then this may subsume the Solution architecture.

These six architecture levels provide us with a nomenclature for agile architecture that will be central to managing agility into the delivered product/solution. The architecture perspective guides the structure of programs and projects and the incorporation of architecture and reuse goals into delivery charters. The architecture also provides traceability and governance over realization of core architecture principles.

The question of how Agile Architecture integrates with Agile delivery is likely to prove contentious because architecture introduces a form of direction that contradicts Agile concepts. Yet the lessons from Amazon are insightful. The most senior business management need to be fully engaged and actively leading the development of architectural direction. Further in large enterprises customer project demand needs to be managed and aligned with business strategy and architectural direction.

There’s no reason why these Demand and Definition processes shouldn’t adopt Agile concepts, notably cross functional teams, time boxes and backlogs. The outcomes should be excellent visibility and traceability of key strategies and policies that provide real clarity of purpose for projects, that will increase the probability of success. In a typical large enterprise use of existing (or well understood) organizational concepts, adjusted to use aspects of Agile methods as discussed, will meet less organizational resistance. For example:  

1.     Architecture Review Board (ARB) or equivalent, a cross functional team (senior representatives of business, product management, architecture and delivery), that provide direction and funding to all architecture development.
2.     Design Authority (DA), also a cross functional team (domain specific expert level representatives of business, product management, architecture and delivery), that transform raw customer demand stream into project charters and manage the portfolio view. It is the DA that takes responsibility for aggregating and decomposing customer and strategic demand, chartering Common, Product Line and Family architecture, typically as integral elements of delivery projects, which can demonstrate business value.
3.     Investigatory architecture projects – short duration projects that validate assumptions prior to chartering composite architecture/delivery projects. Sometimes carried out as part of a Definition Phase activity concurrent with outline requirements and knowledge discovery. Using patterns as a mechanism to increase consistency of architecture decisions and communicate them to delivery projects at sensible level of detail that is useful to delivery teams.  Recommend includes delivery team members as appropriate.
Note this is a recursive model, and the process may executed at enterprise and program level.

You may ask where Enterprise Architecture is in this. The answer is that enterprise architecture is a role and responsibility that must coordinate and govern all levels of architecture. Enterprise Architects are most likely to be assigned to a specific architecture perspective level. The notion of, “one architecture to rule them all” really doesn’t exist.
Each enterprise should develop its own architecture management approach, and integrate this into an end to end architecture, delivery and governance process. The term Agile Architecture should be used to describe and deliver architecture that facilitates the agile business by compliance with reference, platform and other architectures that facilitate evolution, customization and plug and play. Faster cycle time and quality outcomes are then a function of both the reusable patterns and parts available for assembly and the Agile delivery process.  

In medieval times the builders of the Gothic cathedrals didn’t start their designs from scratch. But equally they didn’t have finely detailed (ivory tower) plans – the technology didn’t exist to support that. Master builders moved from city to city bringing their proven architecture in their heads, often together with experienced craftsmen, to new projects. Craftsmen and master builders together tried out new designs and gradually evolved core patterns such as the flying buttress, which became standard components in cathedrals across Europe. Sometimes the great buildings fell down during construction and the builders had to adapt the architecture and try again. They were truly early adopters of Agile methods as they combined architecture and build in what clearly was from time to time an empirical delivery approach, but they also had their equivalent of a reference architecture and patterns that enabled systematic reuse of proven designs. Of course their delivery cycle time was a little longer than today’s Agile project!

Talk to Everware-CBDIabout the Agile Enterprise Workshop. This is currently available as an in-house, intensive workshop. Public scheduled classes will hopefully follow next year.

[1] Amazon and eBay Web Services, The New Enterprise Applications? By Lawrence Wilkes, CBDI Journal October 2004

[2] Inadvertently published by Steve Yegge, 2011, in a comparison of Google and Amazon practices.

[3] Werner Vogels, 2006, SOA creates order out of chaos @ Amazon, Rich Seeley, Search SOA

Wednesday, November 7, 2012

The Agile Enterprise Value Chain

Agile methods have not been widely adopted by enterprises. Agile projects remain, for the most part, independent software development activities, and often by design focused on key areas of enterprise innovation. The latter makes sense, but we should question why Agile concepts should not be rolled out more broadly, because there are considerable opportunities for process improvement across wider range of project classes as well as greater coverage of the end to end life cycle.

If we take this broader, multidimensional view, it should also help enterprises to take a more mature position on agile and agility. Agile methods are primarily guiding management and to an extent project management practices. The business value focus is therefore not surprisingly on “project” quality, cycle time and cost. If we take a broader view we can also focus on enterprise level business improvement, governance and end to end process optimization.

Nobody wants to overload an Agile delivery process unnecessarily. But there are key enterprise perspectives that need to be addressed, and good way to figure out which contribute to the overall delivered agility is to model business value. The business value model allows us to a) develop and refine the solution delivery value chain required for varying enterprise and project contexts and b) charter (structure, manage, govern) architecture and delivery projects with greater probability of achieving optimal outcomes. 

Naturally all enterprises and projects have varying needs for business value. Yes, fastest cycle time and lowest cost are always important, but we can imagine that these will be reasonably compromised for the right business improvement, or reduced risk. A good place to start therefore is by considering the agility related business value required for a project, scenario or enterprise in its broadest sense and relate this to delivery life cycle outcomes. In the simple model below I have listed some practice domains and potential outcomes and then mapped these to candidate business benefits.
Agile Outcomes Mapped to Business Value (Example Fragment)
I have focused Agile practices on Lean process values because these seem to encapsulate all the various Agile methods. In addition I have included disciplines that focus on typical enterprise activities including architecture, asset management, application lifecycle management and automation. I don’t pretend this list is exhaustive, it’s merely illustrative. I am sure readers will have many ideas for practice domains and relevant outcomes. I then mapped this starter list against business benefits using the very effective approach that I cribbed from COBIT5 when I was developing extensions of same. FYI P: Primary, S: Secondary.

This analysis then provides structured data on which to develop an agility value chain (diagram below). I’m sure readers will be very familiar with this technique, first described by Michael Porter[1].  For further explanation see my introduction in Realizing the Agile Enterprise.

Agile Enterprise Value Chain
There are some key points to make about the agile value chain:
1. The primary activities are a cohesive set of activities, and it is important to optimize value across the entire life cycle. For example:
- Addressing software development alone is likely to be suboptimal.
- Making sure that demand is understood, grounded in business strategy, aggregated across lines of business and geographies where appropriate, decomposed into optimal units of work, consolidated into units of release and so on is key.
- Establishing clarity of purpose and matching with an optimal delivery approach.
- Integrating the activities of architecture, definition and delivery in a continuous value chain that minimizes architecture and definition efforts based on value creation. 

2. The value of primary activities can be dramatically enhanced with good supporting activity.

3. That supporting capabilities may be delivered using primary activities which either have qualified goals and objectives, or that the outcomes of primary activities are harvested to create supporting capabilities. For example, in the typical enterprise there are frequently considerable benefits to be gained from reusing many types of asset such as  services, components, schema,  platforms, patterns etc. but it is relatively unusual for enterprises to capitalize on these opportunities for a multitude of reasons including politics, budgets, ownership and support. However if the potential value can be demonstrated and quantified in terms of reduced delivery times and costs, then a business case can be made to put effective systems put in place. 
4. Agile concepts do not just relate to software development! There is great opportunity to adopt key Agile concepts including particularly Lean, Kanban and Scrum, across the entire delivery value chain, particularly for primary activities such as demand and define, and supporting activities such as governance, architecture and delivery infrastructure.

5. That few enterprises are independent, and collaborations are part of business as usual. Further, innovative forms of collaboration may be actively pursued relative to the enterprise’s goals, which might result in widespread use of a common platform, business or technology services, or involvement of unconventional partners such as brokers or social networks.

The Value Chain provides a framework for analyzing the relative business value of the capabilities involved in product delivery in terms of agility outcomes.  In the table below I have shown just a small fragment of what this might look like. I have decomposed each Value Chain Activity into capabilities and assessed potential agility outcomes. Some very obvious extensions would be to include scoring (weighted support to business level benefits) plus inter capability dependencies. A logical conclusion might be to quantify value in terms of cycle time hours or cost reduction, but this seems unnecessary for our purpose here.

Capabilities Mapped to
Agility Outcomes  (Example Fragment)
The detailed Value Chain provides a structured basis for creating and communicating delivery life cycle templates. And it occurs to me this could be just the way to address the elephant in the room for many enterprises – the SDLC standard, commonly a formally mandated standard that is all but ignored by most projects. For most enterprises I believe there are just three basic delivery patterns which provide three template choices, and I will expand on these shortly. I will also be discussing all of the value chain activities in some detail.

Talk to Everware-CBDIabout the Agile Enterprise Workshop. This is currently available as an in-house, intensive workshop. Public scheduled classes will hopefully follow next year.

[1] Porter, M.E. (1985) Competitive Advantage, Free Press, New York, 1985.

Friday, November 2, 2012

Realizing the Agile Enterprise

Have you noticed? Organizations have become initiative driven. Ten years ago enterprise architecture was topic de jour precisely because of initiative fatigue. But today there’s a huge focus again on narrow focus strategic projects and programs, because they are (perceived to be) the only way to deliver business change fast.

Architecture, especially enterprise architecture, has become yesterday’s issue – primarily, many would argue because it failed to deliver the promised business agility. Challenging for the same mindshare but at the other end of the spectrum are Agile software development methods that bring an almost religious zeal to rapid delivery by concentrating responsibility on self-managing, cross functional teams. Don’t get me wrong, narrow focus teams are highly effective in solving complex problems that are intrinsically narrow in scope. The problem is that many enterprises are inherently complex, and well executed architecture is the only way in which complex problems can be broken down and structured in order to establish appropriately independent units of work that can be addressed in an effective manner using Agile methods.

There have been several well intentioned attempts to evolve Agile methods to be effective in an enterprise context, to deal with the inevitable complexity that goes with very large scale operations that demand high levels of regulation, governance, scalability, standard services and business processes. But these efforts are unlikely to succeed because they approach the problem through the narrow lens of the Agile methods.

At the same time we should observe that use of UML based model driven methods and tooling has not become widespread, as was once anticipated. Agile methods provide no guidance on “how” to undertake tasks and Agile practitioners by my own observation commonly reject the rigor of formal methods and tools. This single action without question limits the scalability of Agile projects making continuous change and iteration an effort intensive and lower quality activity.

Many enterprises have voted with their feet and in their use of Agile methods have adopted a hybrid approach commonly referred to as Water-Scrum-Fall. In other words, architecture, planning and requirements are undertaken in the time honored fashion, and development is executed using Agile methods, typically Scrum. In truth, Water-Scrum-Fall should be designated an Anti-Pattern because it perpetuates the inefficiencies of early phases and renders the Agile development process sub-optimal because conventional levels of requirements errors and development and test driven rework persist.

What’s happened is that Agile methods have in the main been adopted in a relatively uncritical and immature manner. It’s very noticeable that proponents of Agile methods strongly advocate adherence to the core concepts and methods, citing the transformational nature of the approach and the inherent dangers of compromise. Yet there are examples of Agile being used effectively in large scale, but these are the exception, and have usually been achieved with either, exceptional levels of skilled resources, or more probably considerable customization of method, together with high levels of structure and tooling.

One must conclude that adoption of Agile methods remains at an early stage of maturity, and that like many new ideas in many domains, will be evolved by convergence with depending and dependent practices, which themselves must also evolve.

A practical way to manage this maturing process is with a value chain of the business change delivery cycle.
Whilst architecture and structured methods and tooling are important as discussed, it’s clear there’s a larger ecosystem that Agile methods must collaborate with. This collaboration cannot be approached in a casual manner, it needs to be specified in detailed processes, practices and deliverables with appropriate automation to bring high levels of discipline to the end to end delivery process. It’s time enterprises applied the same level of process change effort to the IT activity that it does to the broader business. In this business improvement process it’s also important to note that Agile concepts can be productively applied to a broader range of activity than purely software development. And as with any value chain, there’s great opportunity to organize the supporting activities and leverage common practices, methods, resources and assets.

The very pace of technology change means today’s enterprise is inevitably going to be initiative driven, but this doesn't mean initiatives should be isolated in order to be successful - this is a path to delivering instant legacy. Rather Agile methods and concepts are effective Organizing and Management approaches, and they need to be integrated into the broader value chain, particularly architecture and life cycle automation, that delivers rapid business change. 

Talk to Everware-CBDI about the Agile Enterprise Workshop. This is currently available as an in-house, intensive workshop. Public scheduled classes will hopefully follow next year.