What don’t we know about Manitoba?

Hey there, time traveller!
This article was published 29/10/2015 (3344 days ago), so information in it may no longer be current.

Justin Trudeau has promised to bring back the long form census, and statisticians across the country are eagerly awaiting this campaign promise to be fulfilled. But beyond all the hand-wringing over survey designs and response rates, what does the lack of a compulsory census really mean for us?

The federal government collects basic information like age and sex for everyone in Canada every five years, and it is still compulsory to respond to this survey. In addition, the government used to send out a long form census – containing questions about more detailed attributes like income, ethnicity and language, to one in five Canadians to get more detailed demographic information on the country.

The compulsory long form census was replaced by a voluntary National Household Survey in 2011. Even though it was sent out to more people – one in three Canadians – it had a much lower response rate because it was voluntary.

The lower response rate has led to all sorts of problems in the research community and reduced usually mild mannered statisticians to fits of rage. As Manitoba’s chief statistician Wilf Falk puts it, "We do not know what information is good and what information is bad."

So what is it that we don’t know about Manitoba, that we would’ve known if we had the compulsory long form census? Here are five examples of unknowns in the voluntary NHS.

1. We don’t have any detailed demographic data on one-third of the rural municipalities in Manitoba, or all of these places:

The map above shows the communities for which Statistics Canada has suppressed NHS data because less than 50 per cent of residents who got the NHS survey completed it.

How many people who speak Spanish live in Roland? How about those earning below $25,000 a year in Minto? How about the proportion of people getting Employment Insurance in Birtle? Thanks to the voluntary NHS, we have no idea.

Because of the low response rates, Statistics Canada has attached a Global Non-Response rate to every area so people using the data know how many survey respondents didn’t respond to the NHS sent to them. If less than 50 per cent responded, the agency won’t release data on that area because they’ve deemed it too unreliable.

Under the previous compulsory census, the agency had a much higher standard for response rate – at least 75 per cent – but expectations have had to be lowered under the new NHS.

You can still ask for this suppressed data from Statistics Canada – but the agency makes no promises about its dependability. In short, the data is quite possibly "garbage," as Chief Statistician Falk puts it.

 

2. We don’t know how Manitoba has changed between 2006 and 2011

The voluntary NHS data cannot be compared to the 2006 long form census accurately, because of the different ways it was collected and compiled.

"People want to be able to do historical comparisons. So the 2011, how has it changed from 2006? Two different methodologies," Falk said.

"We can measure the change but… how much of that is because of the methodology change? We don’t know. In a sense, it could have not changed at all, but the change is all due to the change in methodology."

 

3. We don’t have key economic data on these downtown Winnipeg neighbourhoods

 

Because of the same problem with low response rates, Statistics Canada has been forced to suppress key demographic information for these downtown Winnipeg neighbourhoods. We don’t know what the median income is in these areas, for example, or how many low income people live there.

Problematically for social organizations and governments that work in these areas, we don’t know for sure if they have got richer or poorer.

While these neighbourhoods were completely suppressed, there are other neighbourhoods with low response rates where researchers are hesitant to use NHS data. The data is further muddled when trying to look at specific groups in the city.

“If you’re using data for, say, the city of Winnipeg where you had a higher response rate… that’s not bad if you’re looking at all of Winnipeg," explained Falk.

"But if you want to say, ‘Okay, what’s happening with people who live in Winnipeg but are recent immigrants?’ Is that data representative of those individuals? We don’t know."

 

4. Some groups are over or underestimated in the NHS

Statistics Canada admits that certain groups are less or more likely to respond to the voluntary NHS, which means numbers on them may not be fully reliable – especially when drilled down to the neighbourhood level. Here’s a table from the agency showing some groups that are more or less likely to complete the volutary NHS survey.

(A positive ‘bias indicator’ shows overestimation of that group. So for example, the number of Filipino people in Canada recorded by the NHS was probably an overestimate)

5. Even if we get back the long form census in 2016, we’ll be stuck with 2011’s gaps forever

A lot of researchers may decide to completely ignore the 2011 survey when the 2016 census comes out, choosing instead to compare with the 2006 census and forget 2011 ever happened. But Falk said that still leaves a gap in historical information.

"Even if it did come back, we’ve got this fuzzy NHS from 2011. So we have good data for 2006, wobbly data for 2011, and then good data for 2016 when it comes out," Falk said.

"The NHS is going to live with us for a long period of time."