If you want people to be able to run it on Windows and Mac you'll need to recompile, yeah. Or provide the source for them to recompile themselves.
Mac expects Mach-O binaries rather than the ELF binaries linux uses so it needs recompiling. Windows obviously have their own format, it's called PE.
If ...