USB N64 Controller and Retroarch is practically incompatible to configure correctly. The only solution I found was to map the n64 controller incorrectly to trick it. Breaks it on all other consoles.
How to bind controller buttons in Retroarch on Linux
retroarch controllers input number file configuration mapping - RetroPie Forum
I can't understand how to set up my controller in Retroarch -- the only available device in 'device index' is an unrelated gamepad, all my custom bindings are shown as "N/A" in
Update for the RetroPie Setup Script: Support for RetroArch's Autoconfig Functionality and (wired) Xbox 360 controller support - petrockblock
How to Setup a Controller on Retroarch on PC - YouTube