In the eighties IT had 'work study' and in the nineties methodology became fashionable. If methodology was so important a few years ago, why is it less important or non-existent now? Surely all the reasons for wanting to have the methodologies of the nineties still exist – it couldn't have just been a passing fad where everyone was sold snake oil, could it?
The short answer is that methodology was, and still remains an important part of delivering IT systems. The need for methodology has not decreased; it is just that methodology is very unfashionable at the moment. The familiar pattern of hype cycles in IT, such as data or centralized processing, go through cycles of being fashionable or not - methodology is in a slump and if it does come back into fashion it will look very different.
What is a methodology?
The idea of this post is not to define methodology and those definitions that do exist are a bit iffy. Basically methodology in an IT context is about the tools and techniques that we use as well as the processes around how we use them. A methodology would, for example, guide how we take users' (data) requirements, turn them into a logical model and ultimately to a physical database. All of the checking, feedback, versioning, rework and how to use the techniques themselves are rolled up into the methodology.
The Downfall of Formal Methodologies
There are many places to point a finger at what caused the lack of popularity of methodologies, but three stand out as having the greatest impact.
The Hyperlinked Generation
The last time I tried to get a bunch of users in a room for a series of formal sessions I spent most of the time running around the office, making phonecalls, re-scheduling and generally herding the group into a room – it would be easier to bath five cats at once. The last time I distributed a specification document for comment nobody even read it properly. Sound familiar? If your methodology says that you need all of the users in the same room or that documents need to be formally reviewed then you are going to struggle to get your methodology implemented.
In today's business environment people are doing many things at once and don't really have the attention span that may be required by formal processes and techniques. Most people will have a few browser windows open, email, IM a media player and any number of things on their desk vying for attention – never mind mobile calls, text messages and other interruptions. These days, a specification document is more of a media mashup from documents, emails and IM transcripts than an engineered document.
The methodologies of the nineties (and most heavyweight methodologies today) demanded attention that people are simply unable or unwilling to give.
UML
In the early nineties large organizations such as IBM and JD Edwards had huge teams working on methodologies and their related tools. Although they kept an eye on object oriented techniques and considered them important, they were only a small part of the bigger picture. As these heavyweight methodologies were maturing, three object guys shook hands and came up with The Unified modelling language – implying only one and that it was unified - modelling is a big part of any methodology.
The first, and loudest, criticism against UML was that it sucked as a methodology and while the proponents exclaimed "It's not a methodology, it's a notation!" the guys back at the office hastily assembled RUP (Rational Unified Process). RUP was (is?) implemented as a heavyweight, document-centric, lethargic methodology that was rejected by developers because more people were required to maintain the documentation than actually do development.
UML was also rejected by business because the stick-man diagrams and 'useless cases' meant little to them – they preferred the good ol' flowcharts, data flow diagrams, ERD's and even IDEF diagrams. After all, this so-called Unified methodology did not even have a way for business to represent process flow – business simply could not relate to the 'everything is an object' paradigm.
Waterfall is 'Evil'
In the mid-nineties the bulk of the methodologies being used were waterfall or BDUF (Big Design Up Front) oriented although they were not necessarily named as such. The movement from plan-driven to agile methodologies (Fowler) resulted in a development meme that the waterfall approaches were 'evil' and to be avoided at any cost. The reality is that few development teams actually took the time to understand the new methodologies and a long period of no usage of methodologies took place.
The software development teams had turned their backs on existing methods but were unable to phase in new ones – particularly with finding a way for organizations to painlessly move from document-centric to people-centric methods. I think that most organizations that are struggling to implement agile methodologies today did in fact have workable methodologies ten years ago that were abandoned with nothing to fill the vacuum in the interim.
Is Methodology Dead?
Methodology is certainly not dead – it has simply become unpopular and gone underground. These are some of the underground groups.
The Silent Groups
There are a lot of people out there producing software and building big systems. They use methodologies – they just don't call them methodologies and they don't talk about them. They are not churning out daily builds with Ruby and have big waterfalls, long processes with lots of people that are doing something other than coding. The reason that they keep quiet about it is because the market, competitors and even drinking buddies poke fun at them. Fashion dictates that we should not wear comfortable shoes in public even if they are, well, comfortable.
Microsoft definitely uses methodologies but they don't talk about it much, it doesn't make marketing sense and they are not really in the business of selling development processes. Imagine telling everyone how great your development processes are and then releasing a product like Vista late – everyone will be saying 'Microsoft development processes suck and they are not agile enough!'. Okay, so in that particular example people are saying it anyway – but you should get the point.
The Germans, always known as being good engineers, definitely use methodologies and they don't talk about them much either. SAP calls their methodology ASAP (AcceleratedSAP) and even use the word 'methodology' in their definition.
The Agilists
Some of the definitive works on Agile and Scrum made explicit use of the term 'methodology' to describe the approaches but they were written before methodology had the negative connotations that it is currently saddled with. But very few agile users would acknowledge that they are using a methodology – it has too many negative connotations and is automatically associated with waterfall (which is evil). Agile teams have approaches, principles and manifestos – not methods or methodologies.
I think that the problem with putting agile and methodologies in the same sentence is that agile methods give rise to multiple methodologies which are not formally described and documented for use by a particular development teams. The agilists sell lots of books and seminars and hang out in private corners of the Internet, but seldom actually proclaim their use of formal methods and techniques – nor do the development teams understand that agile methods expect a team to choose its own process (Fowler). Unfortunately this lack of formalism and secret-handshaking results in many development teams following some sort of Scrumbut approach ('We're doing scrum, but...').
Perhaps the authors and the leaders in the agile domain are trying hard to reinforce the correct use of methods but the users are not reading the books and articles – it is far easier to download a 'Scrum in 5 Minutes' document and implement it without even bothering with the detail.
The Architects
So if you are not part of 'The Silent Groups' or 'The Agilists' does that mean that you have a team of Cowboy Coders? Many development teams produce the same sort of software as teams using a more recognisable (or even agile) methodology without officially following a method at all – but at the same time not making it up as they go along. I believe that this is in fact the biggest group but how do they get it right?
The answer is that any successful development team does use methods and they configure those methods into their own proprietary or configured methodology – even if it is not recognised. The distinction is that the methods are far more technical and the documentation is mostly in the most definitive requirement document that there is – the actual code. Due to the low level technical nature of the methods the architects are generally the ones that are driving and implementing the methodology – mostly unknowingly to themselves and definitely unrecognised by business or even project managers.
Some examples are obvious – I, for example, use the built-in diagrams of SQL Server to represent, describe, document and even model my (MS-SQL) physical database. There is no need for some external ERD tool that offers few benefits and is always out of sync. Interfaces are another area where specific methods come into play – consider IDL and it's more modern counterpart WSDL – if software is going to interface correctly, across say a web service, then in order to define the interface correctly various processes have to be followed. XML Schemas are another similar example where there is little space for cowboy programmers – your data will simply bounce back.
Good architects are aware that the technical choices that they make have a huge impact on how the system is developed. An architect that takes a more object-centric approach (as opposed to data-centric; read DataSets) automatically has an environment where methods such as TDD (Test Driven Development) or even Domain Driven Design become attractive and value-add methods and techniques.
Even simple responsibility allocation can lead to different methods being used. An architect or technical lead may say particular developers work on the front-end, others on the back-end and so on. Such segregation requires that developers communicate, through agreed structures or interfaces, with one another.
This concept of architects selecting the methods and directing the overall methodology is not lost on methodologists or product vendors. The increased awareness around DSL's (Domain Specific Languages) is evidence of the understanding that architecture and engineering types are moving away from loads of documents with stick-man drawings to something that is more useful to them.
This is not lost on product vendors such as Microsoft who are trying to productize Software Factories in Visual Studio Team System. While Jack Greenfield from Microsoft is struggling to get buy-in to the Microsoft version of the Software Factory, the fact that the ideas are more aligned with software architects and engineers as implementers of the teams' methodology to me indicates a better chance of success than the MDA (Model Driven Architecture) nirvana - which is still too document centric and has a waterfall (evil) feeling.
The Future of Methodology
On a daily basis when working in corporate IT environments you get the feeling that there is no basis, no method behind the chaos in projects. However, if you are prepared to stand back and analyse the processes at work you will see that the disciplines are often there and development do try and do things properly, as an engineering team would. The methods that they use may not be bound up in a huge book or corporate 'Development Standards' file given to new recruits – but in many cases they do exist.
The challenge is to understand more clearly what we are doing and to embrace the changes and new methods that will no doubt be possible with new tools and influenced by inevitable technological advancement in our field. The biggest problem is getting the non technical influencers, such as business and project management to understand that we are neither selling them the latest fad nor behaving like cowboys.
Simon Munro