Easy ship fake pickup attempts

I’m a new Amazon seller doing around 1 order per day. Sometimes the pickup delivery partner doesn’t come for pickup, but when I call him he says “I came but you didn’t answer.”

The problem is I’m pretty sure nobody actually came. This has happened more than once and I’m worried about late shipment metrics affecting my account.

Anyone else facing this? How do you deal with it?