That is mostly correct; immigrants account for about 3% of a total 123mil population. Your point does stand, but we are quite visible in Japan -- especially in Tokyo.
You are correct that we do not threaten jobs either. A large majority of the foreign population is working low/unskilled jobs. Generally, the native population is not wanting those jobs.
Those undesirable jobs can also be highly visible. Maybe the most frequent place Japanese notice visible minority immigrants is at convenience stores. So maybe it makes the population feel overrepresented. I see RWers post frequently about convenience store workers at least.
You are correct that we do not threaten jobs either. A large majority of the foreign population is working low/unskilled jobs. Generally, the native population is not wanting those jobs.