Can't say this for certain since I've never managed to get a group and a mission where I can die, at the same time, but my guess is once your lives run out, the mission fails. What this basically does is it means you don't have to make sure nobody dies in order for the mission to succeed, yet at the same time, you can't just keep coming back from the nearest dock until you win.

If your Group lives run out, you'll get a message saying that your ship cost got too high and you fail the mission.