The Conservatives don’t attract too few women. They attract too few of everyone.

David Cameron tells Angela Eagle to “calm down, dear”. A number of female Conservative MPs decide they will not stand at the next election. Another is deselected. Ed Miliband derides the dearth of women on the government front bench. Several women are replaced in their jobs heading public bodies, prompting Harriet Harman to observe that it is “raining men in the Tory Party”.

To some, these stories and others like them constitute evidence that the Conservatives have a problem with women. As evidence goes, this is pretty flimsy. The academics Peter Allen and Philip Cowley have pointed out that within three years the Cameron Cabinet included exactly the same number of women as Tony Blair’s did at the same point in his premiership (though Mr Blair had rather more to choose from). Proportionately more female Labour MPs are standing down next year than female Tories. If the defenestration of Anne Mackintosh tells us something about Tory activists’ attitude to women, what about the dumping of Tim Yeo? And does anyone other than Harriet Harman – or even she, really – believe that ministers decided not to re-appoint Suzi Leather or Sally Morgan to their respective quango jobs simply because they are female?

However tendentiously, these events are taken not only as evidence of one problem but an explanation for another: if the Tories don’t like women, it’s hardly surprising that women don’t like the Tories.

But is this true? Labour’s lead has for some time been higher among women than it is among men. But this is the wrong number to look at. Analysis by YouGov has found that, aggregating their daily polls month by month between last September and January, the Conservative vote share among men was 33%, while among women it was just … 33%. My own recent research says the same. In the 8,000-sample poll for Project Blueprint conducted last November, the Tories were on 30% among men, just one point above their share among women.

Non-Tory women, though, were more inclined to support Labour, while men were more likely to back UKIP – hence the “gender gap” in the Labour lead. This, too, was supported by the YouGov analysis.

For the Tories, of course, there is only so much comfort to be drawn from the fact that women are no less likely to support them than men – or, to put it another way, that they are equally unappealing to both sexes. But are they put off (or attracted) for different reasons?

In my polls I regularly ask what are the most important issues facing the country. In Project Blueprint, the three most popular answers were identical for men and women: “getting the economy growing and creating jobs”, “controlling immigration” and “managing the economy overall”. There was a difference in the number four slot, however: for men it was “cutting the deficit and the debt”, while for women it was “improving the NHS”. Women were also much less likely than men to choose “standing up for Britain’s interests in Europe”.

Men were more disposed than women to ascribe positive attributes to the Conservative Party, though they agreed on the order in which they applied – from being “willing to take tough decisions for the long term” at the top to “representing the whole country, not just some types of people” at the bottom. But the same applied for Labour and the Lib Dems. We find this in all polling: whatever the question, women are more likely to say (or admit) that they don’t have an opinion.

Which of these things matter most when it comes to deciding which party to vote for? The four most important drivers for Conservative voters were the same for both men and women: believing the party “shares my values”, that it is “on the side of people like me”, a positive view of David Cameron, and the belief that the country would be better off after five years of Conservative government than under Labour. The fifth most important driver for men is the belief that the Tories “want to help ordinary people get on in life”; for women it is that they would “tackle the cost of living and improve living standards for people like me”.

Believing that the party “shares my values” and is “on the side of people like me” also drives the decision to vote Labour, for both sexes. The Ed Miliband factor appears to work differently for each, however: for men, it is important to see him as the best PM; for women, it matters more whether they like him as a person.

When it comes to the Liberal Democrats the key factors are once again values, the feeling that they are “on my side”, plus being “competent and capable”. (With respect to Nick Clegg’s party, so few people think this that the opinion inevitably goes hand in hand with a Lib Dem vote).

Click here for poll data on the Conservatives and the “gender gap”

For proponents of the gender gap theory, David Cameron is one of the main problems: women are said to find the PMQ put-downs, the occasional “Flashman” demeanour and the Bullingdon factor particularly forbidding. Yet there is not a great deal of evidence for this.

I asked people for the first word or phrase that came to mind when they thought of David Cameron. Of the 20 most frequent answers from women, 13 were negative and seven positive or neutral – the same as for men. Sixteen words in the women’s top twenty list matched sixteen in the men’s (albeit in a different order: “competent”, the first positive word on both lists, was number three among men but number twelve among women).

The four words on the men’s list that did not appear on the women’s were “Eton”, “good”, “Tory” and “determined”. The four on the women’s list that did not appear on the men’s were “Conservative”, “pompous”, “incompetent” and, notably, “family” – something that applies objectively to all three leaders but only registered for Cameron and only among women.

Asking the same question for Miliband elicited a different flavour of abuse. Indeed the top three words given for each leader might sum up the election campaign in a nutshell: in the blue corner, “posh”, “out of touch” but “competent”; in the red corner, “weak”, “idiot” but “trying”.

The six words unique to the women’s top twenty for Miliband also had a different theme to those of their male counterparts. While women registered bemusement and irritation (“nothing”, “boring”, “who?”, “unsure”, “slimy”, “annoying”), men had concerns over ideology and aptitude (“unions”, “out of his depth”, “geek”, “red”, “inexperienced”, “socialist”).

Curiously, the same phenomenon applied to Nick Clegg. Is it better to be thought “spineless”, “wet” and “soft” (women), or a “two-faced” “lapdog” “opportunist” (men)? Another strategic dilemma for the Deputy PM to ponder.

Some nuances of opinion, then, can be inferred from this analysis: at a stretch, it might suggest that at the margins women place more emphasis on public services, the cost of living and the relatability of leaders, while arguments about aspiration, competence and the macro economy work better with men. But it would be a stretch. When it comes to the policy priorities of men and women, their views of the parties, reaction to leaders, likelihood to vote Conservative and their reasons for doing so, there is little to choose between them.

The Conservative Party’s problem, then, is not that it is not attracting enough women; it is that it is not attracting enough of anybody. The striking thing in the polls is not the gender gap but the overall numbers of either sex who are prepared to say the Tories stand for opportunity or are on their side. It is not women who have a problem with the Conservatives, it is voters.

The “inside baseball” controversies are a distraction from this fact. There are all sorts of reasons why people who could be Tory voters think the party is out of touch with their lives; the minutiae of MP selections, appointments to public bodies and the acceptability or otherwise of put-downs used in the heat of PMQs are among the less important. It is not that these things do not matter – they do. But for most of the people the Tories need to reach, a great many things matter a great deal more.

In Project Blueprint I asked people whether, after the next election, they would expect to be better off after five years of Conservative government or five years of Labour. Men and women were equally likely to say Labour, but fewer women than men named the Tories; instead they were more likely to say it would make no difference who was in charge. Research by IpsosMORI has found women more concerned than men about paying their bills, their children’s job prospects, and their ability to maintain their standard of living and retire as planned.

These things put beltway stories illustrating the supposed “women problem” into perspective. Over nearly a decade of political research I have heard any number of complaints about the Conservative Party, but I could count on the fingers of one hand the number of times misogyny or chauvinism has been raised. Part of the reason for this may be that I am usually speaking to voters who are “in play” – willing to consider switching between parties in one direction or another. I suspect that people who are particularly attuned to this kind of story, or who think of women as a special interest group with their own “issues”, are less likely to be open to a party that prefers to think of people as individuals rather than in categories. I further speculate that when Harriet Harman tried on The Andrew Marr Show to badger Michael Gove into guaranteeing that the next head of Ofsted would be a woman before the selection process had even begun, the few who agreed with her were all members of the Labour Party.

Does that mean Tories should ignore the question of how women are represented among them? Certainly not. The party should strive to attract more female candidates, along with more from ethnic minorities and from backgrounds that have not traditionally supported the Conservatives – not as an exercise in tokenism, but to help ensure the party’s pool of talent is as wide and deep as it can be.

David Cameron has made some effort – and with some success, despite risking the ire of established supporters and activists – to increase the number of female Conservative candidates and MPs. Forty nine were elected in 2010, up from seventeen at the previous election. This is right, though I am doubtful that their relative scarcity is a big factor in putting women off. Labour’s higher lead over the Tories among women than men is the direct opposite of the position forty years ago. It seems facetious to point out that this reversal has coincided with a quintupling of the number of women on the Tory benches, but it at least makes the point that there is no straightforward link.

Even so, work must continue to make the Conservative Party more like the country it seeks to represent. Meanwhile, the Tories should concentrate on showing they are a party for everyone. If there are such things as women’s issues, they include economic security, public services, character, motivation and leadership – not a bad platform for a campaign.

Among those the Tories win back by broadening their appeal will be many of those who have drifted to UKIP. Then, ironically enough, the Tories might really have a gender gap – but nobody would be talking about it because they would be winning. After all, who is agonising over Labour’s “men problem”?

Related Stories
Keep up to date with political & polling news
Sign up to our newsletter below