The alarm is likely due to the low battery. If the alarm comes on with the water pump, that makes sense because if the battery is already low the water pump will drag it down real low.
As to the water pump running all the time...it is likely a bypass pump like a Surflo 4048 pump. A bypass pump is designed to run water down to very low flow without the pump cycling on and off. It does this by bypassing water internally rather than the pressure switch shutting the pump off at low flow. However, if the bypass pressure gets too close or overlaps the shutoff pressure the pump will not stop running.
For some reason, unknown to me, a low battery affects the pressure setting. The pump may run fine when it has good electric power but with a low battery it just continues bypassing internally and does not stop running. You can cure this by adjusting the pressure screw. This is done by turning a very small 5/64 Allen screw on the end of the pump. On the 4048 pump, and most bypass pumps, there are two adjustment screws. The pressure screw is in the center at the end of the pump head. The bypass screw is offset from the center. You want to adjust the pressure screw. To stop the continuous running, on the 4048 pump, you insert a 5/64 Allen wrench in the center screw. Turn the pump on (faucets off...pump running) and turn the screw about a half turn. If the pump does not stop go back to the original position and turn a half turn in the other direction. When the pump stops turn the screw just a "little" further. Try turning the faucet on and off several times to insure that the pump now shuts off when the faucet is closed.
My pump manual did not call out the pressure adjustment. I found this out by emailing the pump manufacture.