A transmission brake such as the Allison Retarder uses the transmission hydraulics to slow the vehicle. Basically it uses hydraulic pressure against a rotor on the transmission output shaft and thus slows the rotation of the drive shaft. Yes, pumping hydraulic fluid against the spinning of the shaft makes heat - that's the shaft energy being dissipated into the fluid, which them must be cooled by the engine/transmission cooling system. All brakes work by converting mechanical energy to heat, so its just a question of where and how that is done. With an engine or exhaust brake, that heat is shed into the engine and from there to the cooling system. The service brakes (disc or drum) shed it into the wheels and brake housing, which is typically finned to help send the heat into the air.
This video explains the Allison operation, both under power and retarding.: https://www.youtube.com/watch?v=v4AJ2BvUMG4
