Thanks everyone, it sounds like this might be less crazy than I thought. It's unfortunately also a very short driveway, maybe 16 ft, then sidewalk, then narrow street (depending on how many cars are parked), so not so much room to get things lined up, which is also why I thought of the trailer dolly, so I can swing the front out into the street in a way that I may or may not have room to do while hooked to the SUV depending on how cars are parked on the street.
There's a car repair place next to us and some major construction going on across the street, so I've watched some really big trucks come through and maneuver their way into driveways, as well as many tows get backed into the shop next door. It's doable by the pros at least. I've just never done anything other than watch.