Petroleum grease will do even faster damage to Buna-N O-rings, but given enough time silicone grease may indeed swell a Buna-N O-ring enough to make it noticeably tight. Silicone lube can also dissolve the surface of silicone O-rings (typically, red, blue or green post O-rings) leaving deposits stuck to the inside of a disconnect...
Cheers!