What Mac model do you have? I have a late 2012 MacMini, with an Apple Keyboard attached, and a mid 2013 Mac Book Air.
One other thin: are you using OS 10.11, or OS 10.11.1? I am using the latter. Also, I did not do an upgrade from 10.10.5 to 10.11.1. I did a clean installation of 10.11.1 on both of my machines. Not sure if that makes a difference, but thought I'd mention it.