I went through your same predicament 2 years ago (wow time flies). I carry a G43 in an alien gear holster around my little one for around 12 hours a week. From what I've found is if you have a good modernist holster and practice proper safety techniques you will be fine. I know it doesn't seem right that it's that easy but it really is. My biggest problem is when holding/picking up the kid my shirt seems to ride up more than I like. It's become a habit now even when not carrying to pull my shirt down after picking him up.
My XDS is secured tightly in the holsters I have, I am not so much concerned with the child accessing it on my person because my trigger is properly covered and secured and my holsters have decent retention for being a hybrid with no active system. I am really more concerned with the size and carry location. Even know when carrying at 3-5 O'clock I am forever having my shirt ride up. Once I throw on a book bag, diaper bag, baby/toddler moving all around dropping things (me having to bend over constantly.... etc). I can only imagine what will happen lol.