To be fair, all things considered, if you have to learn either language from scratch from a native language that has no link[1] whatsoever to it, I think English is harder than Japanese.
1. by which I mean, in the case of English, for example, if your native language is not french, german, etc.
In fact, I think English is harder to learn for Japanese native speakers than Japanese for English native speakers.
Yeah, I can't really speculate. It seems to me that the #1 biggest obstacle for Japanese speakers of English is getting past the Japanese education system, and actually speaking. Beyond that, I don't know.
But as a general principle, I believe that learning any new language is really, really hard. The people who claim to master languages in a few weeks are usually full of it (and by "usually", I mean "essentially always".)
1. by which I mean, in the case of English, for example, if your native language is not french, german, etc.
In fact, I think English is harder to learn for Japanese native speakers than Japanese for English native speakers.