The N64 has 60 interrupts per second internally for the video interrupt, so when the game is not able to render the scene within 2 "half-frames" (1 frame), it extends into a 3rd half-frame (1.5 frames) or more.
Also the dust that appears from sliding on the ground can really cause a lot of lag in certain areas.
Just wanted to add that Caivs tested this on console and it didn't work for him. We triple checked for typos and made sure everthing was correct but the game just wouldn't start. Anyway I made a video on emulator, sorry in advance for bad gameplay (I don't run this game):
We managed to get this working on console, see here for the fixed codes.
The JP code ended up working with a direct port. The US code would crash on boot if the interrupt handler was hooked, and has been adjusted so that it's only hooked after the first input frame.