Last time I mentioned that I wanted to do something a little simpler than logistic regressions. So I’ve moved on to doing some very simple hypothesis tests. I originally intended to move away from politics as a topic, but a series of reports by the Hansard Society caught my eye. The Audit of Political Engagement is a survey and analysis of British citizens’ engagement in democratic politics that has been running for 15 years. I couldn’t resist! I thought it would be fun to perform hypothesis tests on some of the headline figures in the 2018 report that refer to changes since 2017.
2017 and 2018 Reports
I don’t want to get into the methodological issues here with the two reports except to say that my analysis assumes that the survey is a perfectly random sample of the population. It isn’t.
Anyway, the executive summary of the 2018 report notes some interesting differences between the 2017 and 2018 survey. Certainty to vote, interest in politics, and self-assessed knowledge of politics have all increased. More people have undertaken a political activity in the last year than the previous whilst people feel that they have more influence over local politics.
Before I discuss my methodology and results, I just want to go over the variables I’m measuring and what they mean. I have 7 variables in total and each is expressed as a percentage. The variables are derived from identical questions in the 2017 and 2018 surveys so I’m on solid ground there.
Certainty to vote measures the percentage of people who responded that they would be absolutely certain to vote in an immediate general election.
Interest in politics measures the percentage of people who said they were either very or fairly interested in politics, as opposed to not very or not at all.
Knowledge of politics and knowledge of parliament measure the percentage of people who felt they knew a great deal or a fair amount about politics and parliament respectively, as opposed to not very much or nothing at all.
Local influence measures the percentage of people who felt they had a great deal or some influence over decision making locally, as opposed to those who felt they had not very much or none at all.
Finally, political activity measured the percentage of people who said they had conducted some form of activity to influence decisions, laws, or policies within the last 12 months. These activities were very broad and included (but wasn’t limited to) voting in an election, signing a petition, donating money, contacting politicians, and active campaigning.
This measure of political activity is highly sensitive to election years since the most common form of political activity is voting. Since the UK held a general election in 2017, I was concerned that any increase in political voting in 2018 would simply reflect this fact rather than any underlying increase in political activity. The 2018 report doesn’t mention this, but I thought it would be fun to test political activity, excluding voting in an election, as its own variable.
Using two-tailed t-tests on the difference in means between the two samples, I could test the significance of the results. My null hypothesis in each case will be that there is no difference in the mean between the two samples. The alternative hypothesis is simply that the means differ (in either direction).
Performing these tests provided me with a p-value for every hypothesis I was testing. The p-value is simply the probability of drawing samples at least as adverse to the null hypothesis as the samples actually drawn under the conditions of the null hypothesis. If the p-value is very small, then it is unlikely that the samples would be drawn under the null hypothesis. That is, it’s likely that the alternative hypothesis is true. I rejected the null hypothesis in every case where the p-value was lower than 0.05 (at 95% confidence).
This chart shows the 7 variables as measured in the two surveys. Generally we can see that these have increased from 2017 to 2018, as the 2018 report notes. I know it’s difficult to see much here. Scroll down for a better chart.
Below is a table with each variable I tested which shows the 2017 and 2018 means, the difference in percentage points between 2018 and 2017, and the associated p-value. Results with * are significant at the 0.05 level.
|2017 (%)||2018 (%)||Difference (pp)||p-value|
|Certainty to Vote||58.4||64.5||6*||0.0008131|
|Political Activity (Excluding Voting)||42.9||48.5||5.6*||0.0024778|
|Knowledge of Parliament||48.7||53.7||5*||0.0071890|
|Interest in Politics||55.3||60||4.7*||0.0099537|
|Knowledge of Politics||52.2||55.9||3.8*||0.0419352|
This chart shows the differences much more clearly. I’ve included the 95% confidence intervals to show the range of uncertainty. It also shows quite clearly why an increase in local influence was not a significant result.
In general, the differences that the 2018 report highlighted between 2017 and 2018 were significant at the 0.05 level. The only variable that I conducted a hypothesis test on that wasn’t significant was local influence. There was only a very small increase of 0.7 percentage points between 2017 and 2018. Such a result would be quite likely under the null hypothesis, so it could not be rejected in this case.
Alright, so most of the variables I tested did change significantly between 2017 and 2018. I mentioned earlier that I didn’t want to get into any methodological issues. I assumed that the samples were random, and didn’t use weights for my analysis. I’m aware that the surveys were conducted using slightly different methods than pure random sampling. I haven’t tried to correct for that here.
I’m going to continue with some hypothesis testing in my next blog, looking again at the 2018 report of the Audit of Political Engagement. Instead of looking at changes between 2017 and 2018, I’ll be looking exclusively at differences between groups in the 2018 survey. I will try to discuss some of the methodological issues I’ve mentioned here, but I make no promises!