edit: I have installed nmap package on the linux relay by hand (apt-get install nmap) and it is working.
"nmap -sS [IP]" shows me the right name and MAC of the target device and a list of open ports etc.
Therefore it must be a BCM problem, if it is working by hand?
edit 2: never mind. Now it is working after I installed and tried it by hand. I changed nothing else. Don't ask me ...
Thanks for the feedback