One of the best (i.e., only) secular arguments I've ever heard against moral acceptance of polyamory/polygamy came from a book I once read on evolutionary psychology.

Imagine a society of 200 people, 100 men and 100 women, all ranked numerically in order of their desirability as mates. In a society that enforces monogamy, man #1 marries woman #1, man #2 marries woman #2, and so on down the line. In a society that permits polygamy (which is, the book said, most commonly manifested as polygyny), man #1 marries women #1-4, man #2 marries women #5-7, and so on, and you wind up with a subclass of undesirable men who cannot find counterparts. Historically, the book argued, societies that allowed for patriarchs and kings with harems of women necessarily found themselves dealing with hordes of low-class unfuckables, males whose only avenues to pass on their genetic lineage were... shall we say... unsavoury.

Of course, this presumes a very simplistic version of humanity, one in which romance is fundamentally about sex and sex is fundamentally about reproduction. As our civilization has progressed, arguably, we have become less and less utilitarian, less and less bestial. It would not surprise me if, in the coming years, polyamorous relationships move away from the polygynous skew and become accepted by popular culture in much the same way that homosexuality has.

As a side note, I hope someone draws that Post-It note thing.