In the eighties IT had 'work study' and in the nineties methodology became fashionable.  If methodology was so important a few years ago, why is it less important or non-existent now?  Surely all the reasons for wanting to have the methodologies of the nineties still exist – it couldn't have just been a passing fad where everyone was sold snake oil, could it?

The short answer is that methodology was, and still remains an important part of delivering IT systems.  The need for methodology has not decreased; it is just that methodology is very unfashionable at the moment.  The familiar pattern of hype cycles in IT, such as data or centralized processing, go through cycles of being fashionable or not - methodology is in a slump and if it does come back into fashion it will look very different.

What is a methodology?

The idea of this post is not to define methodology and those definitions that do exist are a bit iffy. Basically methodology in an IT context is about the tools and techniques that we use as well as the processes around how we use them.  A methodology would, for example, guide how we take users' (data) requirements, turn them into a logical model and ultimately to a physical database.  All of the checking, feedback, versioning, rework and how to use the techniques themselves are rolled up into the methodology.

The Downfall of Formal Methodologies

There are many places to point a finger at what caused the lack of popularity of methodologies, but three stand out as having the greatest impact.

The Hyperlinked Generation

The last time I tried to get a bunch of users in a room for a series of formal sessions I spent most of the time running around the office, making phonecalls, re-scheduling and generally herding the group into a room – it would be easier to bath five cats at once.  The last time I distributed a specification document for comment nobody even read it properly.  Sound familiar?  If your methodology says that you need all of the users in the same room or that documents need to be formally reviewed then you are going to struggle to get your methodology implemented.

In today's business environment people are doing many things at once and don't really have the attention span that may be required by formal processes and techniques. Most people will have a few browser windows open, email, IM a media player and any number of things on their desk vying for attention – never mind mobile calls, text messages and other interruptions.  These days, a specification document is more of a media mashup from documents, emails and IM transcripts than an engineered document.

The methodologies of the nineties (and most heavyweight methodologies today) demanded attention that people are simply unable or unwilling to give.

UML

In the early nineties large organizations such as IBM and JD Edwards had huge teams working on methodologies and their related tools.  Although they kept an eye on object oriented techniques and considered them important, they were only a small part of the bigger picture.  As these heavyweight methodologies were maturing, three object guys shook hands and came up with The Unified modelling language – implying only one and that it was unified - modelling is a big part of any methodology.

The first, and loudest, criticism against UML was that it sucked as a methodology and while the proponents exclaimed "It's not a methodology, it's a notation!" the guys back at the office hastily assembled RUP (Rational Unified Process).  RUP was (is?) implemented as a heavyweight, document-centric, lethargic methodology that was rejected by developers because more people were required to maintain the documentation than actually do development.

UML was also rejected by business because the stick-man diagrams and 'useless cases' meant little to them – they preferred the good ol' flowcharts, data flow diagrams, ERD's and even IDEF diagrams.  After all, this so-called Unified methodology did not even have a way for business to represent process flow – business simply could not relate to the 'everything is an object' paradigm.

Waterfall is 'Evil'

In the mid-nineties the bulk of the methodologies being used were waterfall or BDUF (Big Design Up Front) oriented although they were not necessarily named as such. The movement from plan-driven to agile methodologies (Fowler) resulted in a development meme that the waterfall approaches were 'evil' and to be avoided at any cost. The reality is that few development teams actually took the time to understand the new methodologies and a long period of no usage of methodologies took place.

The software development teams had turned their backs on existing methods but were unable to phase in new ones – particularly with finding a way for organizations to painlessly move from document-centric to people-centric methods. I think that most organizations that are struggling to implement agile methodologies today did in fact have workable methodologies ten years ago that were abandoned with nothing to fill the vacuum in the interim.

Is Methodology Dead?

Methodology is certainly not dead – it has simply become unpopular and gone underground.  These are some of the underground groups.

The Silent Groups

There are a lot of people out there producing software and building big systems.  They use methodologies – they just don't call them methodologies and they don't talk about them.  They are not churning out daily builds with Ruby and have big waterfalls, long processes with lots of people that are doing something other than coding.  The reason that they keep quiet about it is because the market, competitors and even drinking buddies poke fun at them.  Fashion dictates that we should not wear comfortable shoes in public even if they are, well, comfortable.

Microsoft definitely uses methodologies but they don't talk about it much, it doesn't make marketing sense and they are not really in the business of selling development processes.  Imagine telling everyone how great your development processes are and then releasing a product like Vista late – everyone will be saying 'Microsoft development processes suck and they are not agile enough!'.  Okay, so in that particular example people are saying it anyway – but you should get the point.

The Germans, always known as being good engineers, definitely use methodologies and they don't talk about them much either.  SAP calls their methodology ASAP (AcceleratedSAP) and even use the word 'methodology' in their definition.

The Agilists

Some of the definitive works on Agile and Scrum made explicit use of the term 'methodology' to describe the approaches but they were written before methodology had the negative connotations that it is currently saddled with.  But very few agile users would acknowledge that they are using a methodology – it has too many negative connotations and is automatically associated with waterfall (which is evil).  Agile teams have approaches, principles and manifestos – not methods or methodologies.

I think that the problem with putting agile and methodologies in the same sentence is that agile methods give rise to multiple methodologies which are not formally described and documented for use by a particular development teams.  The agilists sell lots of books and seminars and hang out in private corners of the Internet, but seldom actually proclaim their use of formal methods and techniques – nor do the development teams understand that agile methods expect a team to choose its own process (Fowler).  Unfortunately this lack of formalism and secret-handshaking results in many development teams following some sort of Scrumbut approach ('We're doing scrum, but...').

Perhaps the authors and the leaders in the agile domain are trying hard to reinforce the correct use of methods but the users are not reading the books and articles – it is far easier to download a 'Scrum in 5 Minutes' document and implement it without even bothering with the detail.

The Architects

So if you are not part of 'The Silent Groups' or 'The Agilists' does that mean that you have a team of Cowboy Coders?  Many development teams produce the same sort of software as teams using a more recognisable (or even agile) methodology without officially following a method at all – but at the same time not making it up as they go along.  I believe that this is in fact the biggest group but how do they get it right?

The answer is that any successful development team does use methods and they configure those methods into their own proprietary or configured methodology – even if it is not recognised.  The distinction is that the methods are far more technical and the documentation is mostly in the most definitive requirement document that there is – the actual code.  Due to the low level technical nature of the methods the architects are generally the ones that are driving and implementing the methodology – mostly unknowingly to themselves and definitely unrecognised by business or even project managers.

Some examples are obvious – I, for example, use the built-in diagrams of SQL Server to represent, describe, document and even model my (MS-SQL) physical database.  There is no need for some external ERD tool that offers few benefits and is always out of sync.  Interfaces are another area where specific methods come into play – consider IDL and it's more modern counterpart WSDL – if software is going to interface correctly, across say a web service, then in order to define the interface correctly various processes have to be followed.  XML Schemas are another similar example where there is little space for cowboy programmers – your data will simply bounce back.

Good architects are aware that the technical choices that they make have a huge impact on how the system is developed.  An architect that takes a more object-centric approach (as opposed to data-centric; read DataSets) automatically has an environment where methods such as TDD (Test Driven Development) or even Domain Driven Design become attractive and value-add methods and techniques.

Even simple responsibility allocation can lead to different methods being used. An architect or technical lead may say particular developers work on the front-end, others on the back-end and so on. Such segregation requires that developers communicate, through agreed structures or interfaces, with one another. 

This concept of architects selecting the methods and directing the overall methodology is not lost on methodologists or product vendors. The increased awareness around DSL's (Domain Specific Languages) is evidence of the understanding that architecture and engineering types are moving away from loads of documents with stick-man drawings to something that is more useful to them. 

This is not lost on product vendors such as Microsoft who are trying to productize Software Factories in Visual Studio Team System. While Jack Greenfield from Microsoft is struggling to get buy-in to the Microsoft version of the Software Factory, the fact that the ideas are more aligned with software architects and engineers as implementers of the teams' methodology to me indicates a better chance of success than the MDA (Model Driven Architecture) nirvana - which is still too document centric and has a waterfall (evil) feeling.

The Future of Methodology

On a daily basis when working in corporate IT environments you get the feeling that there is no basis, no method behind the chaos in projects.  However, if you are prepared to stand back and analyse the processes at work you will see that the disciplines are often there and development do try and do things properly, as an engineering team would.  The methods that they use may not be bound up in a huge book or corporate 'Development Standards' file given to new recruits – but in many cases they do exist.

The challenge is to understand more clearly what we are doing and to embrace the changes and new methods that will no doubt be possible with new tools and influenced by inevitable technological advancement in our field.  The biggest problem is getting the non technical influencers, such as business and project management to understand that we are neither selling them the latest fad nor behaving like cowboys.

Simon Munro

The tale of the Pied Piper refers to all of the towns' rats being led away, followed by the children after he wasn't paid. I see the same thing happening with capable IT architects - the analogy works even if you can't decide if architects behave like rats or children. While corporate IT departments are busy with other things, the real architects are slowly disappearing.

Definitions of the architectural role aside, let’s for a moment assume that an IT architect is a technical person with significant experience making a technical contribution to a project. Look around within your own organization and count how many people who are actively working as architects that are above thirty five, in their forties and their fifties – I don’t count very many. It seems strange that with IT systems being so darn hard to build successfully every time, that there is a lack of receding hairlines on our technical teams. This is not necessarily the case in other industries – most surgeons are considered to get better with age, right up until they are too shaky to hold a scalpel. My grandfather finally stopped working when he was in his seventies, spending every day on civil engineering sites – he had built virtually every major bridge and road in the city and was useful to have around.

Where the IT Architects are going

If as an architect you can coerce a multi million dollar system into existence, chances are you have a pretty broad range of skills. IT architects, in addition to technical knowledge, can project manage, lead teams, handle politics, understand business and so on – all of this in a fast-paced, highly stressful environment. It is pretty easy to handle a career change.

Some architects have moved out of IT completely – I know some who have become pretty good property developers, while telling me how great it is because building technology is fairly static. Some stay in corporate environments, even consulting, doing things such as project management – it is far easier to brand yourself as a senior project manager than an architect. Some move onto the supply side of IT, using their experience to sell products, services and resources to corporate buyers.

Most do less and less architectural work on a daily basis, lose touch with the current technological hype and feel that it is not worth the effort to re-skill again (intentional tautology).

Why the IT Architects are leaving

The reasons should really be handled by a more substantive survey, but there are some things that really annoy architects and make them throw up their hands and think "I'm going to become a project manager!" Here are some of my theories.

Fifteen years of experience overridden by technical developments in the last year

There is a perception in IT that the more you know about the latest technology, the better you are. While this may be relevant for part of the project team, much of the work that needs to be done is not latest-framework specific. Imagine an architect with fifteen years of experience focusing briefly on non-technical aspects such as quality for three years – by the time he tries to become involved on the latest project he is considered out of the loop and drops a couple of rungs in terms of technical seniority. Architecture is a lot about abstractions and as much as developers will say how radically different v3 of the framework is, the abstractions still remain the same. At some point good architects say "This is the last framework/language/methodology/platform that I am going to learn if we buy into the 'next big thing' I'm out of here!"

Seen this movie before

Do you ever get the feeling that in IT we are continuously re-solving problems that were solved a long time ago? I'm not even talking about that latest language, framework or other technologies that could arguably be evolving. I'm talking about things like handling temporal data, long running transactions, concurrency and contention and so on. Often architects find themselves mentoring, arguing or persuading others that if they choose a particular approach they will run into well known problems that may have been solved thirty years ago. Obviously this is the architects role – imparting their knowledge and experience, but sometimes they are flat-out ignored by people who think they know better because the technology has changed. In these cases I often start my counter with "I've seen this movie before and it has an ending like Titanic" – by now, people on my team know that it is a prompt to pay attention to avoid pain.

He doesn't code

Amongst architects the discussion as to whether or not an architect codes is unnecessary. The problem is that the technical members on a project feel that the architect should be coding and their reasoning is sometimes valid. A good architect is stuck in the middle – too technical to be valued for business knowledge and not a good enough coder to earn street cred from developers. Even coding architects find it difficult to keep up because they are not coding all day every day.

Salesperson Competition

Pre-sales techies are the bane of virtually every architect – energetic semi-technical salespeople that have all the latest brochures, white-papers, presentations and anything else to get business excited. Sometimes by the time the architect becomes involved, business is so sold on the particular product that the train simply cannot be stopped. Instead of inviting architects soon enough, and face it they don't get as excited and can ruin a 'fun' demo, business holds back on architectural input until they say "We have bought this product and it is a strategic fit – please get it integrated with all the other products that we have bought"

False Credentials

I was involved in a project with an awesome architecture assembled and delivered by a brilliant architect. When the final testing was going on external auditors rocked up to do an assessment – fair enough, it was a bank after all. They conducted a few days' worth of interviews and inspections and in their auditors report proudly announced that the architecture was bad (it should have been in a cube and not a relational database). How on earth would an auditor know, where are his qualifications, what is the basis for his argument, can we see the report, when can we have a meeting? Nothing… no answers… nada! "The auditors have spoken and it is so recorded, entered and agreed." Architects, without recognised qualifications and the rest of business not really understanding what they do are always at the whim of roles that traditionally have more clout and credibility.

Make him a manager

Sometimes, the best way to keep experienced architects around is to give them an office and a larger credenza. That way they can justify the cost and still have someone who knows how all their systems hang together. The problem is that then as a technical manager the architect does less and less architectural work and does administration, putting out fires and ultimately more golf. The result is an architect that is totally out of touch with what he or she really enjoys doing.

How to stop the Architectural Pied Piper

I don't consider myself one of those people-people and can't really provide input that is of a general nature – you can go and speak to your own people-people about that.

The first thing to decide is if you want the people with experience around – sometimes you may want to cut out the dead wood so that you can finally replace the token ring network. This can often be the case with architects that are simply looking backwards instead of forwards – the kind who look back fondly on SmallTalk and print out their emails.

Also you need to establish why you want to retain the experience – it could be that you need someone who knows all the history of the legacy systems so that they can be used as a reference guide for all the hacks and workarounds that have been put in over the years. That kind of person may be quite happy being a manager of a maintenance team and does not want any architectural responsibilities.

If you want to keep good, experienced architects, you need to create an environment where you can realise their benefit. A good place to start is to define the architectural role carefully – not simply employee specific roles and responsibilities. What is needed is a clear understanding of what architects do and how they fit into the IT department and it needs to be communicated more broadly. Although good architects are able to explain their functions to whoever they interface with, it would be a great help if all participants, from developers to business are clearer on the functions. Understanding functions within an organization is usually general knowledge – most people know the difference between say a sales manager and a financial controller – why should it be any different with IT.

For other ideas, look at the reasons for leaving above and find ways to counter them, such as:

• Insist that architects have relevant current knowledge of implementation issues, but don't insist that they code it up themselves or patch it into the network.
• Bounce proposed approaches off architects and ask for their input. Most good architects will be able to provide useful pointers, anecdotes and long stories of things that you have not considered yet.
• Involve your architects throughout the entire procurement process. If they rough up the sales person a little – let them. The worst that can happen is that they uncover some cracks. If the product is worthwhile and the salesperson any good they will be back for another round – hopefully having addressed some of the concerns.
• Back up your architects and present them as key participants in decision making – if they disagree, ask them to present their reasons formally, logically, concisely and clearly. Good architects revel in such an approach.

In case you were wondering, I'm not some washed-up techie who is going through a mid-life crisis. I have been in software development for about fifteen years and been doing architecture for at least eight. When I need to, I can out-code and out-deliver most developers on my team on the latest technology. I'm not leaving, although I if my current framework/platform (.NET/MS) becomes marginalised, I probably won't learn a new one in as much detail.

To the architects out there – see what you need to do to stay in the game. To business – keep your great architects so that systems can be delivered.

Simon Munro

If you are considering a career as an IT Architect you need to pause for a moment and wonder if you want to label yourself as an 'Architect'.  While there is a general trend at the moment to clarify the term more carefully and formally, most of the people you run into will have there own preconception of what an architect is.  Lots of people run around IT giving themselves the architect title and, whether doing architecture or not, have given all aspiring architects a bad name.

So when positioning yourself as an architect, consider that the following types of architects have already set the perceptions of what an architect is.

The PowerPoint Architect

By far the most common type of architect is The PowerPoint Architect, these kinds of architects produce the best looking architectures on paper... I mean PowerPoint.  Great colours, no crossing lines and reasonably straightforward to implement... apparently.  The problem with PowerPoint architects is that they are so far removed from real implementation that architectures that they propose simply won't work.  The PowerPoint Architect is generally a consultant who, just before implementation is about to start, picks up their slides and moves to the next project - leaving everyone else to implement their pretty diagrams.  The PowerPoint Architect believes that software development is similar to doing animations in PowerPoint and infrastructure is about how to get your notebook connected to a data projector.

How to spot The PowerPoint Architect

The PowerPoint Architect gives him/herself away by scheduling presentations in meeting rooms and having so many slides that there is no time to go into the detail.  If the meeting has more business and project representatives than technical staff, it was probably organized by The PowerPoint Architect so that technical questions seem out of place and should be 'taken off-line'.  The PowerPoint Architect has also been known to use Visio.

The Matrix Architect

Named after 'The Architect' in the Matrix movie series, The Matrix Architect has been there so long that he/she doesn't know any other way.  Matrix Architects leaves no room for improvement, discussion or negotiation as the architecture was written by them eons ago and has worked fine, thank you very much.  Much like the scene in The Matrix Reloaded, The Matrix Architect has a personalised, well defended office and if you manage to get in, you simply have to leave by one of two doors - without getting a chance to explain yourself.

How to spot The Matrix Architect

The Matrix Architect normally has their own office and is well settled.  Technical books on CORBA, Betamax and other has-been technologies are proudly displayed on the shelves.  The Matrix Architect can also be spotted by their uncanny ability to work their way into meetings and throw curveball comments like "That's just like the SGML interface that we used on DECT and in my day..."

The Embedded Architect

The Embedded Architect creates architectures that are so huge and complex that removing them is similar to taking out your own liver.  Most of the time they do this for career stability or, if they come from an external organization are there to milk as much future profit out of projects as possible.

How to spot The Embedded Architect

The Embedded Architect is very difficult so spot during the embryonic stage when they are infecting the existing architecture and often once spotted it is too late.  The Embedded Architect often has a team of disciples that as a group understand the entire architecture, but individually know very little.  A requirement that new team members go on an induction course on the architecture is a sign that there may be an Embedded Architect somewhere within the organization.

The Hardware Vendor Architect

The Hardware Vendor Architect is actually a salesman with a reworked title.  The Hardware Architect's role is to point out the flaws in everyone else's architecture so that they can justify why the extra hardware expense is not their fault.  At Hardware Architect School, The Hardware Architect is trained in creating proprietary hardware platforms that create vendor lock-in.

How to spot The Hardware Vendor Architect

The Hardware Vendor Architect normally has a car full of pens, mouse mats and notepads emblazoned with some well-known brand which they use to assimilate the weak.  They also have huge expense accounts where they can take the entire data centre to lunch occasionally.  They are often heard saying things like 'You need a 24x7 99.999999% disaster recovery site'

The Auditor Architect

We are not sure of the origins of The Auditor Architect, because they are supposed to be auditing things, not creating architectures.  The Auditor Architect will always propose an architecture that uses spreadsheets for every possible system interface that requires each user to be a CA so that they can review the transactions before they are submitted (not to be confused with The Auditor Project Manager who uses spreadsheets for all documentation).  Since most organizations don't have that many CA's, The Auditor Architect represents a firm that can provide as many CA's as may be necessary.

How to spot The Auditor Architect

The Auditor Architect always wears a black suit, white shirt and an expensive tie in the latest fashionable colour and style.  The Auditor Architect will often go to great lengths to express that they are unbiased and just want to make sure that things are done correctly.  Most emails received from The Auditor Architect have spreadsheet attachments.

The Gartner Architect

The Gartner Architect has knows all the buzzwords and has all the supporting documentation.  They never actually put together a workable architecture but run ongoing workshops on the likelihood of the architecture looking a particular way at some point in the next six months to five years.  As soon as an architecture is established, The Gartner Architect uncovers some 'new research' that requires a suspension of the project while the architecture is re-evaluated.  Incidentally, sometimes The Gartner Architect is known as The Meta Architect.

How to spot The Gartner Architect

The Gartner Architect always does presentations with references to some research noted on every slide and the true test of The Gartner Architect is asking for the document that is being referred to - it won't materialize.  The Gartner Architect is often accompanied by a harem of PowerPoint Architects eager to get their hands on the material.  The Gartner Architect is often entertained by The Hardware Architect, provided that they represent products that are in 'The Magic Quadrant'.

The ERP Vendor Architect

True Architects for ERP systems do exist - but they hang out somewhere else, like in Germany, and not on your particular project.  There is no need for an architect on a system that if changed, self destructs within thirty seconds.  The ERP Vendor Architect is actually an implementation project assistant that is billed at a high rate.

How to spot The ERP Vendor Architect

The ERP Vendor Architect almost always has a branded leather folder of some really fun training conference that they went to in some exotic location with thousands of other ERP Vendor Architects.  A dead giveaway is if The ERP Vendor Architect and The Hardware Architect are exchanging corporate gift goodies - a sure sign that they are colluding do blame legacy systems for the poor performance.

The UML Architect

The UML Architect is not interested in any architecture that cannot be depicted using UML diagrams and spend a considerable amount of effort making sure that this happens.  The UML Architect lives in an object bubble and has no consideration that their intended audience never learned SmallTalk.

How to spot The UML Architect

The UML Architect is easy to spot from the documents that they produce.  All documents have a lot of stick-men, hang-men and and cartoon characters pointing at bubbles.  The UML Architect will always be able to describe the architecture by <<stereotyping>> it as something that you will understand.

The Beta Architect

The Beta Architect insists that the current version of whatever software you are using is going to be ridiculously out of date by the time the system goes live.  For that reason it is important that the development be done with the beta framework, operating system or development environment and not to worry, the product will be probably released before the system needs to go into production.

How to spot The Beta Architect

The Beta Architect normally wears a golf short with a large software vendors logo embroidered on the front and walks around with a conference bag suitably branded.  The Beta Architect normally comes from an external organization that has a partnership with a large vendor indicated by some metal, but always gold or platinum - bronze and silver partners are not worthy.

Simon Munro