Skip to content

Add postpone-cut-over-flag-file interactive command #1561

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

yktakaha4
Copy link

A Pull Request should be associated with an Issue.

Related issue: #1560

Description

This PR enable specifying postpone-cut-over-flag-file via the interactive command.
This allows postponing the cut-over even after execution has started, eliminating the need to restart the process.

  • contributed code is using same conventions as original code
  • script/cibuild returns with no formatting errors, build errors or unit test errors.

Testing

After starting the gh-ost execution without postpone-cut-over-flag-file option, a second session is launched to send the postpone-cut-over-flag-file command.

First session:

$ ./bin/gh-ost --host=127.0.0.1 --user=root --password=my-secret-pw --database=dummy --table=employee --alter="MODIFY COLUMN id BIGINT" --allow-on-master --initially-drop-ghost-table --initially-drop-old-table --serve-socket-file=/tmp/ghost.sock --initially-drop-socket-file --execute
[2025/05/25 16:46:37] [info] binlogsyncer.go:191 create BinlogSyncer with config {ServerID:99999 Flavor:mysql Host:127.0.0.1 Port:3306 User:root Password: Localhost: Charset: SemiSyncEnabled:false RawModeEnabled:false TLSConfig:<nil> ParseTime:false TimestampStringLocation:UTC UseDecimal:true RecvBufferSize:0 HeartbeatPeriod:0s ReadTimeout:0s MaxReconnectAttempts:0 DisableRetrySync:false VerifyChecksum:false DumpCommandFlag:0 Option:<nil> Logger:0xc00038a1e0 Dialer:0x76bc00 RowsEventDecodeFunc:<nil> TableMapOptionalMetaDecodeFunc:<nil> DiscardGTIDSet:false EventCacheCount:10240 SynchronousEventHandler:<nil>}
[2025/05/25 16:46:37] [info] binlogsyncer.go:443 begin to sync binlog from position (binlog.000002, 583132108)
[2025/05/25 16:46:37] [info] binlogsyncer.go:409 Connected to mysql 8.0.40 server
[2025/05/25 16:46:37] [info] binlogsyncer.go:868 rotate to (binlog.000002, 583132108)
# Migrating `dummy`.`employee`; Ghost table is `dummy`.`_employee_gho`
# Migrating a83c44d9d62a:3306; inspecting a83c44d9d62a:3306; executing on thinkahashi
# Migration started at Sun May 25 16:46:37 +0900 2025
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/gh-ost.throttle
# Serving on unix socket: /tmp/ghost.sock
Copy: 0/1108847 0.0%; Applied: 0; Backlog: 0/1000; Time: 0s(total), 0s(copy); streamer: binlog.000002:583135811; Lag: 0.04s, HeartbeatLag: 0.04s, State: migrating; ETA: N/A
Copy: 0/1108847 0.0%; Applied: 0; Backlog: 0/1000; Time: 1s(total), 1s(copy); streamer: binlog.000002:583140368; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: N/A
Copy: 19000/1108847 1.7%; Applied: 0; Backlog: 0/1000; Time: 2s(total), 2s(copy); streamer: binlog.000002:585014381; Lag: 0.13s, HeartbeatLag: 0.04s, State: migrating; ETA: 1m55s
Copy: 47000/1108847 4.2%; Applied: 0; Backlog: 0/1000; Time: 3s(total), 3s(copy); streamer: binlog.000002:587775115; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 37s
Copy: 77000/1108847 6.9%; Applied: 0; Backlog: 0/1000; Time: 4s(total), 4s(copy); streamer: binlog.000002:590728197; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 34s
Copy: 108000/1108847 9.7%; Applied: 0; Backlog: 0/1000; Time: 5s(total), 5s(copy); streamer: binlog.000002:593772998; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 32s
Copy: 137000/1108847 12.4%; Applied: 0; Backlog: 0/1000; Time: 6s(total), 6s(copy); streamer: binlog.000002:596628412; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 33s
Copy: 165000/1108847 14.9%; Applied: 0; Backlog: 0/1000; Time: 7s(total), 7s(copy); streamer: binlog.000002:599388356; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 33s
2025-05-25 16:46:44 ERROR User commanded 'postpone-cut-over-flag-file' without specifying file path
Copy: 194000/1108847 17.5%; Applied: 0; Backlog: 0/1000; Time: 8s(total), 8s(copy); streamer: binlog.000002:602251532; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 31s
Copy: 221000/1108847 19.9%; Applied: 0; Backlog: 0/1000; Time: 9s(total), 9s(copy); streamer: binlog.000002:604909622; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 32s
Copy: 248000/1108847 22.4%; Applied: 0; Backlog: 0/1000; Time: 10s(total), 10s(copy); streamer: binlog.000002:607570475; Lag: -6.26s, HeartbeatLag: 0.04s, State: migrating; ETA: 31s
Copy: 279000/1108847 25.2%; Applied: 0; Backlog: 0/1000; Time: 11s(total), 11s(copy); streamer: binlog.000002:610622068; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 26s
Copy: 309000/1108847 27.9%; Applied: 0; Backlog: 0/1000; Time: 12s(total), 12s(copy); streamer: binlog.000002:613578989; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 26s
Copy: 338000/1108847 30.5%; Applied: 0; Backlog: 0/1000; Time: 13s(total), 13s(copy); streamer: binlog.000002:616438109; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 26s
Copy: 366000/1108847 33.0%; Applied: 0; Backlog: 0/1000; Time: 14s(total), 14s(copy); streamer: binlog.000002:619198318; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 26s
Copy: 395000/1108847 35.6%; Applied: 0; Backlog: 0/1000; Time: 15s(total), 15s(copy); streamer: binlog.000002:622054019; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 24s
Copy: 417000/1108847 37.6%; Applied: 0; Backlog: 0/1000; Time: 16s(total), 16s(copy); streamer: binlog.000002:624220796; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 31s
Copy: 448000/1108847 40.4%; Applied: 0; Backlog: 0/1000; Time: 17s(total), 17s(copy); streamer: binlog.000002:627281184; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 21s
# Migrating `dummy`.`employee`; Ghost table is `dummy`.`_employee_gho`
# Migrating a83c44d9d62a:3306; inspecting a83c44d9d62a:3306; executing on thinkahashi
# Migration started at Sun May 25 16:46:37 +0900 2025
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/gh-ost.throttle
# postpone-cut-over-flag-file: /tmp/postpone.flag [set]
# Serving on unix socket: /tmp/ghost.sock
Copy: 449000/1108847 40.5%; Applied: 0; Backlog: 0/1000; Time: 17s(total), 17s(copy); streamer: binlog.000002:627281533; Lag: 0.03s, HeartbeatLag: 0.06s, State: migrating; ETA: 21s
Copy: 477000/1108847 43.0%; Applied: 0; Backlog: 0/1000; Time: 18s(total), 18s(copy); streamer: binlog.000002:630140181; Lag: 0.13s, HeartbeatLag: 0.14s, State: migrating; ETA: 21s
Copy: 507000/1108847 45.7%; Applied: 0; Backlog: 0/1000; Time: 19s(total), 19s(copy); streamer: binlog.000002:633098646; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 20s
Copy: 538000/1108847 48.5%; Applied: 0; Backlog: 0/1000; Time: 20s(total), 20s(copy); streamer: binlog.000002:636062230; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 18s
Copy: 560000/1108847 50.5%; Applied: 0; Backlog: 0/1000; Time: 21s(total), 21s(copy); streamer: binlog.000002:638326931; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 24s
Copy: 590000/1108847 53.2%; Applied: 0; Backlog: 0/1000; Time: 22s(total), 22s(copy); streamer: binlog.000002:641280010; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 17s
Copy: 614000/1108847 55.4%; Applied: 0; Backlog: 0/1000; Time: 23s(total), 23s(copy); streamer: binlog.000002:643640898; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 20s
Copy: 644000/1108847 58.1%; Applied: 0; Backlog: 0/1000; Time: 24s(total), 24s(copy); streamer: binlog.000002:646598050; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 15s
Copy: 675000/1108847 60.9%; Applied: 0; Backlog: 0/1000; Time: 25s(total), 25s(copy); streamer: binlog.000002:649655167; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 13s
Copy: 704000/1108847 63.5%; Applied: 0; Backlog: 0/1000; Time: 26s(total), 26s(copy); streamer: binlog.000002:652511593; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 13s
Copy: 726000/1108847 65.5%; Applied: 0; Backlog: 0/1000; Time: 27s(total), 27s(copy); streamer: binlog.000002:654682359; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 17s
Copy: 755000/1108847 68.1%; Applied: 0; Backlog: 0/1000; Time: 28s(total), 28s(copy); streamer: binlog.000002:657545144; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 12s
# Migrating `dummy`.`employee`; Ghost table is `dummy`.`_employee_gho`
# Migrating a83c44d9d62a:3306; inspecting a83c44d9d62a:3306; executing on thinkahashi
# Migration started at Sun May 25 16:46:37 +0900 2025
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/gh-ost.throttle
# postpone-cut-over-flag-file: /tmp/postpone.flag [set]
# Serving on unix socket: /tmp/ghost.sock
Copy: 764000/1108847 68.9%; Applied: 0; Backlog: 0/1000; Time: 28s(total), 28s(copy); streamer: binlog.000002:658430876; Lag: 0.03s, HeartbeatLag: 0.05s, State: migrating; ETA: 11s
Copy: 784000/1108847 70.7%; Applied: 0; Backlog: 0/1000; Time: 29s(total), 29s(copy); streamer: binlog.000002:660406061; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 11s
Copy: 812000/1108847 73.2%; Applied: 0; Backlog: 0/1000; Time: 30s(total), 30s(copy); streamer: binlog.000002:663166650; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 10s
Copy: 842000/1108847 75.9%; Applied: 0; Backlog: 0/1000; Time: 31s(total), 31s(copy); streamer: binlog.000002:666122993; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 8s
Copy: 865000/1108847 78.0%; Applied: 0; Backlog: 0/1000; Time: 32s(total), 32s(copy); streamer: binlog.000002:668388501; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 10s
Copy: 893000/1108847 80.5%; Applied: 0; Backlog: 0/1000; Time: 33s(total), 33s(copy); streamer: binlog.000002:671149707; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 7s
Copy: 924000/1108847 83.3%; Applied: 0; Backlog: 0/1000; Time: 34s(total), 34s(copy); streamer: binlog.000002:674200324; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 5s
Copy: 951000/1108847 85.8%; Applied: 0; Backlog: 0/1000; Time: 35s(total), 35s(copy); streamer: binlog.000002:676860051; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 5s
Copy: 980000/1108847 88.4%; Applied: 0; Backlog: 0/1000; Time: 36s(total), 36s(copy); streamer: binlog.000002:679715515; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 4s
Copy: 1010000/1108847 91.1%; Applied: 0; Backlog: 0/1000; Time: 37s(total), 37s(copy); streamer: binlog.000002:682572602; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 3s
Copy: 1028000/1108847 92.7%; Applied: 0; Backlog: 0/1000; Time: 38s(total), 38s(copy); streamer: binlog.000002:684376396; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 4s
Copy: 1058000/1108847 95.4%; Applied: 0; Backlog: 0/1000; Time: 39s(total), 39s(copy); streamer: binlog.000002:687403913; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: 1s
Copy: 1090000/1108847 98.3%; Applied: 0; Backlog: 0/1000; Time: 40s(total), 40s(copy); streamer: binlog.000002:690563339; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: due
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 0/1000; Time: 41s(total), 40s(copy); streamer: binlog.000002:693616499; Lag: 0.03s, HeartbeatLag: 0.10s, State: migrating; ETA: due
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 0/1000; Time: 41s(total), 40s(copy); streamer: binlog.000002:693617691; Lag: 0.03s, HeartbeatLag: 0.04s, State: postponing cut-over; ETA: due
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 0/1000; Time: 42s(total), 40s(copy); streamer: binlog.000002:693622326; Lag: 0.03s, HeartbeatLag: 0.04s, State: postponing cut-over; ETA: due
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 0/1000; Time: 43s(total), 40s(copy); streamer: binlog.000002:693626896; Lag: 0.03s, HeartbeatLag: 0.04s, State: postponing cut-over; ETA: due
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 0/1000; Time: 44s(total), 40s(copy); streamer: binlog.000002:693631463; Lag: 0.03s, HeartbeatLag: 0.04s, State: postponing cut-over; ETA: due
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 0/1000; Time: 45s(total), 40s(copy); streamer: binlog.000002:693635611; Lag: 0.13s, HeartbeatLag: 0.14s, State: postponing cut-over; ETA: due
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 0/1000; Time: 46s(total), 40s(copy); streamer: binlog.000002:693640179; Lag: 0.13s, HeartbeatLag: 0.14s, State: postponing cut-over; ETA: due
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 0/1000; Time: 47s(total), 40s(copy); streamer: binlog.000002:693645168; Lag: 0.03s, HeartbeatLag: 0.04s, State: postponing cut-over; ETA: due
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 0/1000; Time: 48s(total), 40s(copy); streamer: binlog.000002:693649317; Lag: 0.13s, HeartbeatLag: 0.14s, State: postponing cut-over; ETA: due
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 0/1000; Time: 49s(total), 40s(copy); streamer: binlog.000002:693653886; Lag: 0.13s, HeartbeatLag: 0.14s, State: postponing cut-over; ETA: due
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 0/1000; Time: 50s(total), 40s(copy); streamer: binlog.000002:693658095; Lag: 0.13s, HeartbeatLag: 0.14s, State: postponing cut-over; ETA: due
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 0/1000; Time: 51s(total), 40s(copy); streamer: binlog.000002:693662665; Lag: 0.13s, HeartbeatLag: 0.14s, State: postponing cut-over; ETA: due
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 0/1000; Time: 52s(total), 40s(copy); streamer: binlog.000002:693667652; Lag: 0.03s, HeartbeatLag: 0.04s, State: postponing cut-over; ETA: due
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 0/1000; Time: 53s(total), 40s(copy); streamer: binlog.000002:693671797; Lag: 0.13s, HeartbeatLag: 0.14s, State: postponing cut-over; ETA: due
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 1/1000; Time: 54s(total), 40s(copy); streamer: binlog.000002:693677890; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: due
# Migrating `dummy`.`employee`; Ghost table is `dummy`.`_employee_gho`
# Migrating a83c44d9d62a:3306; inspecting a83c44d9d62a:3306; executing on thinkahashi
# Migration started at Sun May 25 16:46:37 +0900 2025
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/gh-ost.throttle
# postpone-cut-over-flag-file: /tmp/postpone.flag
# Serving on unix socket: /tmp/ghost.sock
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 0/1000; Time: 55s(total), 40s(copy); streamer: binlog.000002:693681608; Lag: 0.03s, HeartbeatLag: 0.11s, State: migrating; ETA: due
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 0/1000; Time: 55s(total), 40s(copy); streamer: binlog.000002:693682800; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: due
Copy: 1121010/1121010 100.0%; Applied: 0; Backlog: 0/1000; Time: 56s(total), 40s(copy); streamer: binlog.000002:693687520; Lag: 0.03s, HeartbeatLag: 0.04s, State: migrating; ETA: due
[2025/05/25 16:47:29] [info] binlogsyncer.go:225 syncer is closing...
[2025/05/25 16:47:29] [info] binlogsyncer.go:988 kill last connection id 133
[2025/05/25 16:47:29] [info] binlogsyncer.go:255 syncer is closed
# Done

Second session:

$ echo postpone-cut-over-flag-file | nc -U /tmp/ghost.sock
User commanded 'postpone-cut-over-flag-file' without specifying file path

$ echo postpone-cut-over-flag-file=/tmp/postpone.flag | nc -U /tmp/ghost.sock
Postponed
# Migrating `dummy`.`employee`; Ghost table is `dummy`.`_employee_gho`
# Migrating a83c44d9d62a:3306; inspecting a83c44d9d62a:3306; executing on thinkahashi
# Migration started at Sun May 25 16:46:37 +0900 2025
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/gh-ost.throttle
# postpone-cut-over-flag-file: /tmp/postpone.flag [set]
# Serving on unix socket: /tmp/ghost.sock
Copy: 449000/1108847 40.5%; Applied: 0; Backlog: 0/1000; Time: 17s(total), 17s(copy); streamer: binlog.000002:627281533; Lag: 0.03s, HeartbeatLag: 0.06s, State: migrating; ETA: 21s

$ echo postpone-cut-over-flag-file=/tmp/postpone.flag | nc -U /tmp/ghost.sock
Postponed
# Migrating `dummy`.`employee`; Ghost table is `dummy`.`_employee_gho`
# Migrating a83c44d9d62a:3306; inspecting a83c44d9d62a:3306; executing on thinkahashi
# Migration started at Sun May 25 16:46:37 +0900 2025
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/gh-ost.throttle
# postpone-cut-over-flag-file: /tmp/postpone.flag [set]
# Serving on unix socket: /tmp/ghost.sock
Copy: 764000/1108847 68.9%; Applied: 0; Backlog: 0/1000; Time: 28s(total), 28s(copy); streamer: binlog.000002:658430876; Lag: 0.03s, HeartbeatLag: 0.05s, State: migrating; ETA: 11s

$ ls -l /tmp/postpone.flag
-rwxr-xr-x 1 yktakaha4 yktakaha4 0 May 25 16:46 /tmp/postpone.flag

$ rm /tmp/postpone.flag
rm: remove regular empty file '/tmp/postpone.flag'? y

@Copilot Copilot AI review requested due to automatic review settings May 25, 2025 07:58
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for the postpone-cut-over-flag-file interactive command, allowing users to postpone the cut-over phase by creating a flag file at runtime rather than on startup.

  • Introduces Server.createPostponeCutOverFlagFile and hooks it into the interactive command handler
  • Adds documentation for the new command in interactive-commands.md
  • Provides unit tests for the flag-file creation logic in server_test.go

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
go/logic/server.go Added createPostponeCutOverFlagFile and interactive command handling for postponing cut-over
go/logic/server_test.go Added tests covering successful creation and idempotence of the postpone-flag file
doc/interactive-commands.md Documented the new postpone-cut-over-flag-file=<path> command
Comments suppressed due to low confidence (1)

go/logic/server_test.go:72

  • Consider adding a test case where createPostponeCutOverFlagFile is called with a path in a non-existent directory to assert that an appropriate error is returned.
func TestServerCreatePostponeCutOverFlagFile(t *testing.T) {

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant