[Autotvm] stuck during tuning

Troubleshooting
#1

I am tuning a resnet-18 network and below is the tuning parameters. The tuning is stuck at 60/1000 , irrespective of the value of n_trails and early_stopping(as long as n_trails is < 60, tuning works).

tuning_option = { # ‘log_filename’: log_file,

'tuner': 'xgb',
'n_trial': 1000,
'early_stopping': 600,

'measure_option': autotvm.measure_option(
    builder=autotvm.LocalBuilder(timeout=10),
    runner=autotvm.LocalRunner(number=10, repeat=3, timeout=4, min_repeat_ms=150),
    # runner=autotvm.RPCRunner(
    #     '1080ti',  # change the device key to your key
    #     '0.0.0.0', 9190,
    #     number=20, repeat=3, timeout=4, min_repeat_ms=150)
),

}

tuning output:(first line prints - len(tsk.config_space),n_trial,tsk_trial)

462000 1000 1000 [Task 1/16] Current/Best: 55.86/ 327.70 GFLOPS | Progress: (60/1000) | 136.54 s

#2

You can try random tuner first to see if it’s an issue from XGBoost.

#3

Random tuner works. The issue is with XGBoost. How can i fix this?

#4

XGBoost stops only temporarily at the 60th config and proceeds after waiting for a long time. Tuning one task to completion(n_trails=500) seems to take around one hour. Is there any way to speed it up?

#5

The time was used to train a search model but it’s not usual to take that long. A recent issue suggests to use XGBoost 0.9.0 to avoid some unexpected errors. You could take a try, but I’m not sure if it works for your case.