Buy box wins hugely effected

I had a couple orders in one day and after dropping them off late at night I came back home tired and forgot to confirm dispatch, 10 mins after I saw I forgot to confirm it I immediately updated it but now my late shipment rate is really high. And I’ve been told this effects my buy box, what should I do