Left and Right Click Stop Working in Hyper-V

I've had this odd problem for what feels like months now.

When I'm remoted into a Hyper-V host I will randomly lose the ability to left and right click in a VM.  My only fix is to reboot the VM but this leads to problems if I'm working on something critical which I can't reboot during the middle of the day.

What's weird is I don't lose the ability to click inside the rest of the host and only in that VM.  It's like this for different VMs over different hosts so its not just one particular server.

Any help or tips would be great.