Hi, sorry to hear you are having problems with the model.
Yes, the original DMG buttons are suposed to fit. I just checked the CAD model, the holes are 10.75mm in diameter. The is exactly the same diameter as the holes in my two original Game Boy DMG cases. Is it possible that the plastic in the print slightly deformed when cooling? That would be one explanation. Another might be that not all original DMG cases (and hence probably not all buttons) are exactly the same size. Some are off by half a millimeter or so in some parts. And so are repro cases.
Either way, if you want to edit the model, the .F3D format comes with the whole edit history and should allow you to adjust the model to your needs without too much trouble.