pics - hoorah!
its kinda hard to tell cos most of the photos from the rear aren't quite in line with the cage
good you got something sorted on the hanger measuring side - would concur that you should try and be a little more patient and sort that out more accurately - front to back as well as bottom to top. Taking the wheel out and reversing it so the cassette isn't in the way would be a good idea too.
The derailleur cage does look a little bent out of alignment on some of these images - which would obviously cause issues - but I'm not 100% sure of this
The other thing I noticed is that this appears to be the GS or so called 'medium cage' version of this RD which has a rated capacity of 33T - NOT 45T - the SGS or "super long cage" version that DOES have a 45T capacity looks like this:
With you're setup your significantly exceeding the rated capacity (11-40T at the back is 29T and then I'm presuming at least a 10T difference at the front between the 2 chainrings? so over 39T for sure. Even if its a 36T out back, you're still over, but admittedly not by much and it would probably be ok)
You could probably buy a longer cage to swap out (if its indeed bent that might not be a bad idea anyway)
This is the parts diagram for this derailleur - the only difference between GS and SGS is the inner and outer cage parts which can be purchased separately
SJScycles would the place I would go to see - they stock lots of older shimano spares - looks like the M770 parts are the same as M771:
inner plate:
https://www.sjscycles.co.uk/gear-spares/shimano-deore-xt-rdm770-inner-plate-sgs-type-y5w510000/
I couldn't find the corresponding outer plate assembly on SJS's site
it might be cheaper to buy another derailleur TBH - but make sure you get an SGS one - you can find loads of suitable models for about 20 quid
While it looks like your chain is sized correctly for big/big (ie the angle isn't too bad in the photos and I'd agree that its' not stretching the RD too much) if I'm correct the issues will be more the higher gears in general. Especially with that tiny granny ring, the cage probably isn't long enough to provide adequate chain tension on the smaller sprockets which would cause a lot of chain slap and possibly the jockey wheels essentially clattering against one another. Would be good to get an image of small/small combination as well
The derailleurs guide pulley also looks uncomfortably close to that 40T sprocket at the back (again if it is a 40T, its too close if its a 36T as well!) - I think you're trying to push too big a sprocket for this derailleur without an extension or at least more B screw adjustment