Now would the weight of my passengers come off of the tow rate or off of the vehicle GVW.
Actually it comes off ALL of those. Any weight carried by the truck is included in the truck's GVW (Gross Vehicle Weight, so must be within the GVWR. Since the truck GVW is part of the Gross Combined Weight, it must be contained by the GCWR as well. And the "maximum tow rating" quoted in the brochures is essentially the GCWR minus the EMPTY truck weight, any weight added to the truck reduces the capacity left for a trailer.
They usually quote the truck's empty weight with a full fuel tank and a 150 lb driver. If you weigh more than that, you need to subtract that as well.
Note that none of these numbers imply anything about performance. There is no statement expressed or implied that you will enjoy "good" performance by staying within that envelope. But obviously, the further below it you are, the better off you will be. On hills your problem is going to be transmission gearing - you don't have enough gears to apply the available power efficiently. You will keep up with car traffic only if you are willing to rev the engine fairly high, something most people are reluctant to do. It's noisy, scary sounding and probably makes the temperature gauge rise as well. You will want to ease up on the accelerator and that is when you will wish you had a bigger engine, higher ratio rear axle, 5 or 6 speed transmission and the other features of a "big truck".