I completely forgot to respond.
I see where you are coming from and it raises one question: If you can apprehend the sound from context or whatever, why shouldn’t you be able to apprehend the semantic, or why should it be harder?
You can’t read the sound, so to speak. You just see the symbols and may have some additional information (the aforementioned context or the grammatical function, for example) and have to link this to the other information, may it be phonetics or meaning. I would argue that which one is more accessible to you depends on what you focus more on.
If a person starts to learn Japanese because he is a big anime fan, for example - and I am aware that this is a cliché - but such a person would probably have already spend some time watching those series in Japanese before learning the language and will do so once started. So it is pretty natural that he might find it easier to associate the symbols with the phonetics.
I personally started to spend some time studying Japanese because Japanese texts contain chinese characters, so I can study them in context, as I want to get to know them better in order to boost my vocabulary aqcuisition in Korean. So for me, the Japanese pronounciation of those words is less relevant. It’s the meaning I am interested in. So I find it easier to remember that aspect.
So it depends on ones reason to learn the language and the focus imho.