While contact with a sponge is not life threatening, some species of sponges will give off toxins if touched. If a person were to touch a sponge deliberately or even accidentally brush up against one, the toxins that are in many sponges may cause skin irritation.
