@ J_for_Jones - “But in theory it’s perfectly possible that in each case you just happened to learn this word early on - maybe even in your very first lesson”
The question is one of probability, and can be described using Bayes’ theorem. Imagine that we have a person, we want to know if they are at a C1 level in language X, and we give them the shoelace test and they pass it. Prior to giving them the shoelace test, based on what we previously knew about their ability in the language, we estimated that they had a certain probability of being at a C1 level, which we call P(C1). What we want to know is the probability of them being at a C1 level given the fact that they passed the shoelace test - let us call this P(C1 | pass). Bayes’ theorem states that this is
P(C1 | pass) = P(C1) * P(pass | C1) / P(pass)
where P(pass | C1) is the probability of them passing the shoelace test if they were at the C1 level, and P(pass) is the probability of them passing the shoelace test regardless of their level. The important part of this equation is the term P(pass | C1) / P(pass). Since P(pass | C1) is greater than P(pass), i.e. somebody at the C1 level is more likely to know the word for shoelace than most learners, the term P(pass | C1) / P(pass) is greater than 1, and therefore P(C1 | pass) is greater than P(C1), i.e. the probability of them being at a C1 level is greater after giving them the shoelace test than it was prior to giving them the shoelace test.
Assume now that they failed the test, we now have the equation
P(C1 | fail) = P(C1) * P(fail | C1) / P(fail)
where the quantities mean the same as before, but with reference to them failing instead of passing. The probability of somebody failing the shoelace test at a C1 level is lower than the probability of a random learner of language X failing it, therefore the term P(fail | C1) / P(fail) is less than 1, and therefore P(C1 | fail) is lower than P(C1), i.e. the probability of the person being at a C1 level is lower after failing the test than it was prior to the test being given.
Let us instead ask whether or not they are at the A1 level. We have the same equation as before, but now it looks like
P(A1 | pass) = P(A1) * P(pass | A1) / P(pass)
where P(A1 | pass) is the probability of the person being at an A1 level given that they passed the shoelace test, P(A1) is the probability of them being at the A1 level before the test, P(pass | A1) is the probability of them passing the test given that they are at the A1 level, and P(pass) is the probability of them passing regardless of their level. In this case, P(pass) is greater than P(pass | A1), i.e. the A1 learner is less likely than the average learner of the language to know the word for shoelace, and therefore P(pass | A1) / P(pass) is less than 1 and P(A1 | pass) is less than P(A1), i.e. the probability of them being at an A1 level is reduced by the fact that they know the word for shoelace.
In the above reasoning, I have assumed that we are only considering groups of learners of language X and not the general world population as a whole. If instead we considered the world population as a whole, things are slightly different, but the general reasoning holds.
Sorry for bringing mathematics into this, but I find this stuff interesting, and I know other people do. People who don’t find it interesting should just ignore this post. Anything who sees anything wrong with my reasoning should of course tell me.