Importance of grammar in language learning

Do pay a lot of attention to grammar when you learn the language or does it became natural part of your languages ability when you are learning ?

I have started learning German and at first I learned a lot of grammar. After a while I felt so tired by it that I quit learning grammar at all and now I only listen and read. I am not sure if it is good strategy to avoid grammar at all. For me English grammar is not so complicated like German grammar.
What is more unfortunatelly no one in my neighborhood speak German so I can only count on myself.
How do you think am I doing right ?

I occasionally refer to grammar rules and tables but am unable to remember them. My German improved when I stopped trying to learn the endings and focused on listening and reading.