The report into Wormwood Scrubs prison by HM Inspector of Prisons highlights the challenges faced by the government in its stated intent to commit to a sustained prison reform agenda.

The report, following the prison’s inspection during November and December 2015, describes an institution in dire and desperate need. Hardly any progress had been made since the previous inspection 18 months earlier.

It found that safety had deteriorated, levels of violence and official use of force were twice as high as comparative prisons and there had been two suicides since the last inspection. The majority of the prisoners were locked in their cells for more than 22 hours a day and the prison had a “significant rat problem”.

Furthermore, the impact of the privatisation of large sections of the Probation Service is laid bare with the revelation that the proportion of prisoners being made homeless on release has risen from 5% to 40% since resettlement services were taken over by London Community Rehabilitation Company. In understated language, deputy chief inspector Martin Lomas, says the prison “continues to fall short of expected standards”.

Frances Crook, chief executive of the Howard League for Penal Reform, says the findings underline the flaw in prison policy to date. “The same tired excuses will be used – that the prison is too old – but if old buildings were the problem we would be tearing down Oxbridge,” she says. “Prisons with too many prisoners and too few staff will fail, no matter how old they are. We cannot go on cramming more and more people into jails without any thought for the consequences.”

The failings at Wormwood Scrubs will come as no surprise to David Cameron, who said in February that the state of our prisons system should “shame us all”.

It is also likely to reinforce the justice secretary Michael Gove’s plans for a number of “reform prisons”, which, like the academy schools he championed when education secretary, would see governors released from central control and given autonomy to run their prisons as they see fit. The chosen jails are due to be announced ahead of the prison reform bill due in next month’s Queen’s speech. The bill will set out the legal framework for the new prisons, which could take over nearby failing jails on the same model as the development of school academy chains.

Both Cameron and Gove believe prisoners should be viewed as “assets”, not liabilities, and that jails are a place where people are sent as punishment, not to be punished further.

The government’s new direction for prisons is remarkable not just because it is the first time in a long time that there has been any seriously stated political will for prison reform (even New Labour’s “tough on crime, tough on the causes of crime” policy did nothing to reduce prison numbers), but because it comes from a Conservative government which has traditionally argued for a hardline prison policy. When he was justice secretary, Ken Clarke fought for a “rehabilitation revolution” but was soon replaced by Chris Grayling whose policies harked back to Michael Howard’s.

Despite the efforts of numerous prison reform groups and prisoner rehabilitation charities over the past 20 years or so, prison numbers have continued to rise while prison conditions have deteriorated. Since 1993, when then Conservative home secretary Howard declared that “prison works”, the prisoner population in England and Wales has increased by more than 40,000. Today, it stands at just under 86,000. Last September, 70 of the 117 prisons in England and Wales were officially overcrowded.

Before stepping down last year as HM prisons inspector, Nick Hardwick described the prison system as being in its worst state for 10 years, with some places of “violence, squalor and idleness.”. Rubbishing the government’s declared policy of a “rehabilitation revolution”, he said: “It is hard to imagine anything less likely to rehabilitate prisoners than days spent mostly lying on their bunks in squalid cells watching daytime TV.”

Sally Coates’ review of prison education aims to change that by making it much easier for inmates to access courses and recommends the introduction of a Teach First-style scheme for graduates who will work as prison officers and function as teaching assistants on the wings for two years.

But James Timpson, the new chair of the Prison Reform Trust, says while he feels “genuinely optimistic” about the proposed reforms, his concern is how long it will take to implement them.

“A prisoner, sitting in his cell today is going to think ‘Oh great, everything is going to change for the better.’ Not straight away. It’s going to take time.”

In the year preceding Hardwick’s final report there were a record number of prisoner-on-prisoner killings, more than 20,000 incidents of self-harm and a self-inflicted death every few days. The negative impact of the prison experience is borne out by the staggering level of reoffending on release. According to Ministry of Justice figures, just under half of all adults released from prison reoffend within the first 12 months. For those serving sentences of a year or less, the figure jumps to 60% and for juveniles it is around two-thirds.

Michael Spurr, chief executive of the National Offender Management Service, is hopeful of change. Responding the inspector’s report he said, “Wormwood Scrubs is an old, crowded, Victorian prison. It is challenging to run and we have struggled to make the improvements necessary to meet inspectorate recommendations and to provide a purposeful regime for prisoners. Poor industrial relations over a protracted period haven’t helped – but the governor now has an agreed improvement plan in place and progress is being made.I’ve been writing about prison issues since 1994 when I was just 10 years into the 20 I would eventually serve. I experienced the effects of Howard’s policies – the overcrowding and the ever-diminishing quality of regimes up until my release in 2004. I witnessed the increased violence and self-harm, the escalation of drug and mental health problems and the inordinate levels of stress suffered by prison staff and I could never figure out why any government would tolerate such a socially damaging prison situation.

Too often disingenuous anti-prisoner rhetoric from various politicians made it almost impossible for the system to improve. Budget cuts of hundreds of millions of pounds over the past few years only made matters worse as prison officer numbers fell from 24,000 in 2010 to just over 14,000 by 2014.

The scale of the government’s ambition to turn around the failings of the system, like the size of the challenge it faces, is huge. If the prison reform bill is to succeed it will need more staff and funding at the very least. The problems at Wormwood Scrubs are just the tip of the iceberg of what Cameron and Gove are up against.