fix(metrics): vmbackup lost a whole cycle to a startup race #26
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/vmbackup-startup-race"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
victoriametrics-0reportedbackup FAILED. Not S3, not credentials:Both containers start at the same time, so vmbackup fired before vmsingle was listening - and the loop then slept 6 hours before retrying. One startup race cost a whole backup cycle. Replica 1 only succeeded by timing luck, which is why this looked intermittent.
/healthbefore the first backup (busyboxwgetis already in the image).victoriametrics-0 reported 'backup FAILED'. It was not S3 and not credentials: Creating snapshot fatal cannot create snapshot: Post .../snapshot/create: dial tcp4 127.0.0.1:8428: connect: connection refused Both containers start at the same time, so vmbackup fired before vmsingle was listening. The loop then slept 6 hours before trying again, so ONE startup race cost a whole backup cycle. Replica 1 only succeeded by timing luck, which is why this looked intermittent. Two changes: Wait for vmsingle to answer /health before the first backup. busybox wget is already in the image. Retry a failed backup in 5 minutes instead of 6 hours. A transient failure must not cost a full cycle.