What is the maximum tooth sprocket you could fit on the back using a Deore DX rear derailleur?

[bold stelvis]

setting H and L screws should have no effect on the derailleur beyond what they are designed to do - ie limit how far inboard and outboard it can move. If something has caused damage its far more likely its because one of those limits is set incorrectly and the derailleur has crashed into either the frame or (far more likely) the wheel spokes - think you would notice that however!!

Another thing that could break it is overly tight chain tension (ie incorrect chain length or obviously over capacity setup where it can't take up the slack properly or gets stretched out too much)

I think this is another point where some photos of your setup would MASSIVELY help anyone see what might be up - I think if you put your mind to it you could sort some out

 

[bold stelvis]

sounds like the chain wasn't routed properly between the jockey wheels and the cage maybe? (or buggered links that stick out can cause catching issues also)

 

[bold stelvis]

also - and I get you want to do this stuff yourself, but...a visit to the local bike shop might be an idea
just have someone suitably qualified take a look