I am an American, but my wife is Japanese. We’ve been married seven years and speak mostly Japanese at home. Many of our friends are Japanese, and I would estimate that I spend at least half of my day listening and/or speaking Japanese.
For the past 7 years we’ve lived in Portuguese speaking countries (Portugal, Brazil and Mozambique), but never in Japan. I studied Portuguese intensively in a language school for the first year, but after that I haven’t really studied the language. On the other hand, I spend a couple hours a week studying Japanese especially trying to improve my reading comprehension.
Despite more exposure to Japanese and much more effort spent studying it, my Portuguese is still stronger. I can read newspapers and novels in Portuguese and understand 80-90% of most conversations. In Japanese, I still cannot read a newspaper or understand a the TV news except at a very basic level.
So, FOR ME, Japanese is a much harder language than Portuguese. 
Cheers,
Ben