I've had the same issue with mine ever since I got it. I put it down to the fact that the previous owner managed to clip both rear corners, as both outer lights were cracked, and the left side has a small dent just below the light, as well as the paint being cracked on the bumper on both sides.
When I fitted the towbar, I discovered that the bumper was a replacement one, and I also noticed what appeared to me to be rear end damage. There is a cross beam along the back and it appears to have been pushed in, although this may be the design of it, I'm not sure. As like you, I was hoping that the side brackets could be adjusted, but didn't see anything.
This is what the rear end of mine look like:
__________________
2001 A8 D2 FL 3.7 Q - LPG Conversion, Engine: AKC, Gearbox:FBB, Colour:LY5X Aqua Blue pearl effect, Interior:WJ light beige/dark beige valcona leather with burr walnut inserts
My DIY jobs -
Replaced alarm horn,
Replaced Coolant Temp sensor,
Updated RNS-D firmware,
Installed Reversing Camera and Digital TV,
Dashcam Installation,
Retrofitted Rear Blinds,
Auto-Dimming Rear View Mirror,
Chrome Boot Struts
Sold! 
2008 S8 D3 5.2 V10 - Engine: BSM, Gearbox:KLW, Colour:LY7F Suzuka Grey Pearl Effect, Interior:Black/Black with Silver/Black valcona leather with carbon atlas inserts, Bang & Olufsen, Heated windscreen, Soft-close doors, TV, DAB, AMI, Quad Zone, Auto Boot, Auto dim Mirrors, Keyless entry, PDC with rearview camera, AFS II Headlights
My DIY jobs -
Fan Jet Windscreen Washers,
Retrofitted Electric Side Blinds