Yeah I think it really depends on common sense and circumstances - for example, we have a small house, so I hear the moment the splashing stops, and I pop my head in. I wouldn't go outside while the kids were in the bath, but I might send an email, grab a cuppa, do a quick tidy - anything where I can hear them and pop my head in every few minutes.
Meg, our kids can be in the bath for awhile too, the bathroom is small and has a ducted heating vent, plus gets the west sun, so our bathroom is the warmest room in the house!
Everyone's circumstances are different - if my kids put their heads underwater, or could turn on taps, or were quiet in the bath, I might do things differently. I don't think it's wrong for you to be super cautious though, it sounds smart to me.