I don't know about "off" flavors, but you might get overpowering flavors. You should consider putting the nibs in a sanitized bag so you can pull them once you hit your desired flavor profile. (of just rack to another fermenter if you don't like bags)