Our FBM offer gets Buy Box over our FBA offer

Wondering if anyone has experienced this and would love to get some advice.

We have our own brand and sell our items directly. We have no competitors for our own items.

For items that can be fulfilled by both FBA and FBM, our FBM is winning the buy box. We actually prefer the FBA offer. First, it would use FBA fulfillment which is cheaper. Second, our Sponsored Ads and Listing would have the Prime badge (high CTR and possibly conversion).

The only solution I see is to change our FBM shipping template so that it would be slower than FBA. Just seem inefficient that Amazon would do that. I thought FBA would beat FBM?!