I assume I can do this in Layout, by learning the buttons and then changing something about the way they work.
Yep... you could create 'dummy' screen controls assigned to the offending MIDI messages, and then set them to "Do Not Pass Thru" in the Hardware Inspector (Layout Mode).
If you are running MS2.2, you could also do this (without the Screen Controls) using the Assignments and Mappings pane in Edit Mode.
A glance through the A37 manual (section 5.3) suggests that some of the controller buttons can be assigned to transmit a whole bunch of MIDI messages when you change a Patch on the keyboard:
I think you will need to create Screen Controls manually for each of these CC messages in order to block them. MS seems to be "assigning" to the last message of a bunch received (CC65)
In addition, the A37 would seem to transmit Bank Select (CC0 and CC32) which MS ignores.
It could be worth using MIDI Monitor to check exactly what is being transmitted from the A37: