My next step is to take it back to loading of the animation and see if I can see any problems with the matrices as they are being loaded. Luckily the model I'm using is stored in a human readable format so I can compare the data for the key frames easily.
The only other problem that I could see it being is an issue with the quaternions for the keyframes not being correct. This again can be tested by doing some debugging at the animation load time.
Hopefully I will get there soon.
Keep Coding :)