Why social housing hurts the badly off
Imagine we paid benefits entirely in food stamps. If we gave recipients cash—we worry—they'd spend it on stuff we don't like, such as booze and fags. To head this off, we give them £15,000 a year each in non-transferable food vouchers. Really, they only need about £5,000 in supermarket shops a year, but since they can't do anything else with them, they eat excellent food and shop only in Waitrose, while they scrimp and save when it comes to clothes, heating, transport, and housing. You'd much rather they gave you less, but they allowed you to spend it yourself.
This is crazy policy, right? But this is exactly what we do with social housing (and, indeed, housing benefit). When you secure a council flat, you are effectively being given the difference between its market rent and what you pay as an in-kind handout. Apart from the poorer council architecture, upkeep, and administration, it's almost identical to if they were paying your rent for you. The "economic cost" of the activity includes the difference between its most valuable use and its current use. If a tenant has an income of £10,000 but gets an additional £10,000 worth of housing each year from the council, it is a bit like forcing them to spend half of their total income on housing.*
Imagine council tenants were given the money in cash, instead of this implicit in-kind benefit, and councils sold their properties off to fund the payment, to be rented at market rates. One possible outcome is that they would re-rent the properties they are currently in, at the market rate. Indeed, I suspect that if this policy were really implemented, many would do just this to begin with. But over time, it's extremely unlikely that tenants would want to spend so much of their money on renting premium land in the centre of town—they would decide to move.
It might be countered: these people probably have jobs, families, friends or other connections in the area. This is true, and the appropriate way to weigh up these claims is letting them decide for themselves. 99.9% of Londoners make the same decision themselves, with cash instead of vouchers, if you include the benefit: we trade off cost of commute, range of amenities, and distance from friends. If we were forced to spend more, in order to live in a more desirable area than we'd pick on our own, we'd be worse off than if we could just choose how to spend that money ourselves. Council tenants are no different.
In fact, my initial analogy was far from the most general I could have made. The situation is much simpler: it's the basic parable of exchange under restrictions. The council tenant has something others want—a right to live somewhere—and that others would be willing to pay the tenant for. Indeed, most tenants would almost certainly accept this payment in exchange for moving out; they'd prefer the money. Both the prospective buyer and the prospective seller could be made better off if this capitalist act between consenting adults weren't banned. But it is.
*Actually, it may be interpreted a lower fraction: applying this method to housing implies we should apply it to healthcare, education, and pretty much everything else the state pays in kind. Housing is somewhat different because it's mostly a market, with few getting most of their housing from state bodies—replacing healthcare with a cash payment would require massive change in the market, replacing council housing would not.