I would think it highly unlikely that his wife could swim, women are not expected to be seen on the beaches, nor his children.
Unlike the Calais Jungle (predominantly young, male economic migrants) where wives and children have been effectively abandoned in their home countries, this family clearly were hoping for a future, any future other than staying in Syria with IS rule looking likely.
Blame the people traffickers, they're the ones raising (false?) hopes, taking the money and condemning so many to a horrible death by putting them in unseaworthy boats, abandoning others and so on.
In reply to OP? No, he shouldn't be accused, absolutely no way poor man. Show some compassion for goodness sake.