Once-upon-a-time Universal Credit (UC), the UK government’s plan for a comprehensive rebuild of the country’s social welfare payments system, was going to be an IT showcase.
It would use the increased power of networks to update people’s benefit payments in “real time” – aiding labour market flexibility as work patterns changed as well as being designed to take advantage of increased computing power and capacity to ensure that the benefit system was operated as an integrated whole and not a disjointed patchwork.
More than even all this, it would be built using “agile” methods to ensure that it was on time and to budget and that the client – namely the UK taxpayer – got the system wanted as opposed to the system that could be bolted together most simply by the software writers.
None of this is now going to happen.
And to the government’s shame they have even debased their own, quite successful, project management system by pretending that UC is something other than a catastrophic failure.
Today, under the cover of a media focused on reporting a major set of domestic election results, the government has slipped out the annual report of the “major projects authority” (MPA) – the body it established to take a grip on the biggest pieces of government procurement.
The MPA has undoubtedly been a big success – as its report shows it has made a positive intervention in a number of projects and forced ministers to take firm action on weaker projects – even to the extent of scrapping them. But on UC the clarity of that process has been sacrificed to political considerations.
For UC has plainly totally failed but rather than admit it, the MPA has created a whole new assessment category – “reset” – and decided that UC is now a new project to be assessed from scratch.
What we should be getting instead is an honest appraisal of why the project failed and what lessons should be learned – especially in development of major software projects. But that would involve admitting that the government has consistently misled people about the nature of the development process, its timetable and its effectiveness. And that is just not going to be allowed to happen.
Various leaks and briefings have, in the last few days, added to the sense that “Universal Credit” – the UK government’s plan to consolidate all welfare entitlements into a single payment that would then respond, in “real time” to changing claimant circumstances and so do more to “make work pay” – is coming closer than ever to being one of the biggest IT project failures of all time.
Successful delivery of the project appears to be unachievable. There are major issues on project definition, schedule, budget, quality and/or benefits delivery, which at this stage do not appear to be manageable or resolvable. The project may need re-scoping and/or its overall viability reassessed.
In other words, the whole sorry thing is on the verge of collapse. In parallel the Cabinet Office’s “government digital service” (GDS) are withdrawing support for the project – seemingly refusing to pour more taxpayers’ money down the Universal Credit drain.
But only a few days ago we got another angle on all this when Computer Weekly reported that the DWP were freezing out its existing contractors and seeking to bring all the work in-house (the Guardian reports that the GDS don’t think the DWP are up to this.)
What is more, it seems that the DWP are again proposing to build the UC system using “Agile Development” – at least the “second stage” (which is in fact what was meant to have been done by now first time around using agile).
Back in 2010/11 the DWP told us agile was going to solve all their problems and UC was proudly promoted as the world’s biggest agile project (including by many who probably should have known better). Agile flopped, and badly, and the DWP have been forced to return to traditional “waterfall” development methods – which are nominally slower – often much slower – but which guarantee no later stage of a project is begun until the earlier bits are working (hence the analogy of water cascading downhill in a series of waterfalls).
Now agile’s boosters are keen to tell us that “agile and world’s biggest don’t mix” but it seems the DWP haven’t heard and are again telling us (or rather, themselves, all this is through leaks and off-the-record briefings) that agile is the secret sauce that will drive them on to victory over the Cabinet Office and the Treasury.
That ought to be a giveaway that it doesn’t actually contain any code or maths or anything else that might require some dedicated concentration, but that does not mean it is not worth reading (or listening to) if you are a programmer, manage programmers or in some way are responsible for the development or purchase of software (it is plain that few or no people at the DWP have read this book – given their travails over the “Universal Credit” project – someone should stick a copy in each minister’s red box pronto).
I have never worked as a professional software developer – though I have written code for money – but still I found this book had a lot of insight, and even manages to explain things such as the halting problem and infinite recursion in a way that non computer scientists are likely to grasp without boring those of us who know what these are.
The book is incomplete, though in that it was written before what looks like the final collapse of the project it describes – the Chandler PIM – in 2008/9 when founder Mitch Kapor withdrew. Chandler sounded like a great idea (like Universal Credit?) but, as the project drags on and on, one begins to wonder what on earth the development team were up to for most of the time they worked on it.
Well worth reading.
One problem about the audio though – I know the American fashion is to mispronounce French words and I don’t want to sound like a European prig (after all this book is about a vital technology in which the US leads the world) – but it goes too far when Albert Camus‘s name is repeatedly made to rhyme with bus!
Every time they have been challenged the DWP say “things are fine now, so the critical report is out of date”. But, of course, each subsequent report reveals things were far from fine when the previous denial was issued.
I have never worked somewhere where decision making was so apparently poor at senior levels … and communications from that level was totally nonexistent. This programme should be a case study for how not to engage with your people to get the most out of them.
There is a divisive culture of secrecy around current programme developments and very little in the way of meaningful messages for staff or stakeholders explaining what will happen and when.
Reading these comments you can see why so many defenders of Agile – and there are plenty of them – get so annoyed when UC is or was described as an Agile project. Plainly – if these comments are any guide – it never was Agile at all.
But the fact is that the Agile boosters were not unhappy about it being described that way at the start.
I was present when the announcement that UC would be delivered by Agile methods was made at the Institute of Government. No one there thought fit to say – as so many Agile defenders do now – that a project this big, this mission critical to government, should not be attempted via Agile.
The government have failed to be honest about the melt down of the UC project – repeatedly claiming that any piece of bad news is merely an historical datum and that Things Are Better Now. That level of dishonesty is scandalous: we deserve to be told about the reality, not fed rubbish.
But the software industry needs to show it is capable of honesty too. Too many people chased after the money and were not willing to be honest about how crazed this whole project was from the very start.
Hopefully UC, in some form or other, will still be delivered. Many of the poorest and must vulnerable in our society depend on it working. Thankfully the government has now also given itself a fair amount of time to clean up its mess. But it is already clear that – like most software projects in the public and private sector – this one has failed to deliver all its specified features on time and to budget.
In a programme as complex as universal credit, which includes new IT developments and changes to existing IT assets, both agile and waterfall methods may be appropriate at different times. As examples, initial development used agile techniques while, in its final stages of testing for the pathfinder from April 2013, the programme is using the waterfall approach—a standard DWP testing methodology.
This is the answer of Mark Hoban, Minister of State at the DWP, in a written answer to a parliamentary question delivered on 12 March and seemingly unnoticed by most people except Computer Weekly – who have hit the nail very firmly on the head:
The DWP gave up using the “agile” method of software development for Universal Credit, the coalition government’s flaghsip reform programme, last month.
It had before now repeatedly claimed agile was the way it would keep Universal Credit on track. The coalition government had meanwhile singled agile out as a major part of its flagship strategy to stop IT projects going calamitously and expensively wrong.
The Major Projects Authority – part of the Cabinet Office – was going to press-gang government departments to use agile methods on big software builds. Universal Credit was the government’s first – and immensely ambitious – big stab at agile…
…The u-turn raises questions about whether the government was wrong to pin its hopes on agile as the way to crack the government IT problem: did DWP ditch agile because it didn’t work? Was agile not all it was cracked up to be?
Or did DWP make such a bodge of agile that Universal Credit is now likely to be a bodge as well. Did failure compel it to fall back quickly to the well-worn waterfall path in an attempt to get things back on track?
A DWP spokesman said neither case was true.
“Just because we are not using agile doesn’t mean agile is inherently flawed, or that Universal Credit is inherently flawed,” he said. “It probably means agile is at a point where agile is not appropriate.”…
…Then last year it started to look like DWP was going to manage only a token roll-out of Universal Credit in October 2013. That’s when Duncan Smith came out with the most complete description of to agile software development, and the most clear commitment to it that, really, anyone in the software community could ever hope to imagine. Universal Credit wasn’t just agile. It really was agile. And it would remain so to the bitter end of the roll-out in 2017.
“There is a lot of ignorance at the moment in the media… saying, ‘You are not going to be ready on time’. The truth is the time that we deliver this is 2017. So that is over four years. We start that process in October,” Duncan Smith told the Work and Pensions Committee on 17 September 2012.
“The whole point about the agile process – which I find frustrating at times, because we cannot quite get it across to people – is to understand that agile is about change.
“It is about allowing you to get to a certain point in the process: one leap – check it out, make sure it works.
“And as you go into the next leap, you come up with something that says, ‘We can rectify some of the issues in this, and make that even more efficient’.
“You are constantly rolling forward, improving and making more efficient things. There is a constant retrospective change that goes on to complete that system, and that is what will happen all through those four years,” he pledged.
I think I need to give a little bit more background on the politics of the decision by the DWP to trumpet its use of “agile” methods and how, bluntly, the department has misused to potential of agile to give it cover in its huge gamble with public money and the living standards of millions of the least well off.
Of course we have to start from the basic fact that most software projects – whether they are in the public or private sector – fail. The failure could be relatively small – a budget overshoot or a lack of sought for capability. Or the failure could be huge – your space craft blows up on launch, your ambulances are never dispatched and people die and so on (these last two are real and will be familiar to almost anyone who has done a development methodologies course).
The 1997 – 2010 Labour government had some successes in software driven projects – the UK Passport Service, for instance, is now much more efficient. But it also had a fair number of high profile failures – especially in its efforts to modernise computer use in the health service (though the major ambulance dispatch failure was not in this time, but a software update did fail) – especially attempts to create a single patient electronic record.
The then Conservative opposition used this often – David Cameron in particular repeatedly suggesting that it was because the Labour government had tried to buy a single supercomputer to run the NHS: something he must have known was simply untrue but presumably worked well in focus groups.
So software projects were and are a hot political topic.
The new government, coming to office in May 2010 did several things to get a grip on failing projects. Firstly, they went for a good old fashioned gouge of the contractors’ margins: essentially saying cut your prices on existing contracts if you even want to be considered for future work. Secondly, they said that new software projects had to seriously consider using free and open source software to avoid proprietary lock-in and thirdly, they said that a new centralised control mechanism had to be applied to ensure that No 10 and the Cabinet Office had a grip on costs and efficiency: it was out of this that the Major Projects Authority, that has now reported UC is close to failure, came.
The three elements have generally worked well. It pains me to give this government political credit, but essentially they deserve it.
Yet UC has been allowed to escape this framework, and “agile” was the excuse given for this.
In, I think, early 2011 the Institute for Government published a report on government software projects and recommended that “agile” methods be used. They cited an experimental, relatively small scale project by the Metropolitan Police Service as an example of how agile could work successfully in government.
I attended the launch seminar which was rather more like a religious mission meeting than a serious seminar on how to get the best value for public money. The room, in one of the government’s finest buildings in St. James’s, was packed to bursting with representatives from small software houses, who saw agile as their ticket to the big time and certainly none of them were going to suggest that “world’s biggest” and “Agile” was a risky mix.
At the meeting the DWP announced that they would be using “Agile” as the basis on which UC was developed. I was only an MSc student but even I thought this looked like exactly the sort of project that the textbooks said Agile was not designed for – but I wasn’t confident enough to say that then and nobody else in the room seemed remotely interested in hearing such a thing.
But it didn’t take long once the meeting was over for people to point out that this was a high risk proposition: but it also became crystal clear that the Secretary of State in the department had decided that Agile was the secret sauce for government IT and that it, and it alone, would lead him to the promised land.
For well over a year it has been an open secret that the Treasury want to pull the plug on UC because they do not believe it can be delivered in anything like a working form to budget. And the signs are all there – essentially the project has already failed as its scope has been cut repeatedly and its final implementation date put back and back.
But the politics of the Conservative Party do not allow anyone in government to say this openly and even when the government’s own project watchdog says the system is on the brink of collapse the department come out and rubbish the assessment – even at the price of contradicting themselves.
This would be funny were it not for the fact that in just a few months millions of the poorest people in Britain will depend on this system working if they are to eat, to heat their homes and clothe their children.