Why is Math harder than English?

Why do most people who are not math-heads find it so hard compared to, say, English? I think that one reason is that when students are learning English or History they are, for the most part, being asked to employ skills that they have already mastered (language) and use in everyday life. When it comes to Mathematics, however, students are being asked to employ an essentially artificial skill set that they don't make regular use of in everyday life.

There's indirect evidence that this is indeed true in that one of the ways to lead students towards mastery of math skills is to encourage them to use math skills in everyday life and, for that matter, to model that use yourself.