You have to turn the shower OFF using the valve on the wall.  If you don’t turn it off there then when you use the toilet or otherwise make the water pump run it will put too much pressure in that line and the water will come out of that connection.