It seems your bracket holes and the case holes are fighting each other, one set of holes is forcing the buttons one way while the other is forcing them to go another way.
You may be able to expand the holes in your blue bracket to give your buttons more play.
Also, sometimes attacking/screwing your board in place helps push the buttons down a little more. Not sure in this case though, the little lips on the buttons should be flush onto the inside of the case all around. So my suggestion is carving your blue holes a little wider so that you can see the inside of the case from all directions when viewing it through the blue hole.