To speak with a native speaker you have to know actively at least 1500 words. To try speaking with a native speaker without knowing words - it’s ridicoulous, you won’t be able to understand anything.
Listening is very important, of course. But maybe you have very good ‘ears’ to understand by listening words which you have never seen - for me it’s impossible! I have first to see new words several times, it means - to read them, and only after that I can understand them by listtening and also not always because the pronunciation of different native speakers is quite different.