XGBTuner crashes on Windows auto-tuning


#1

I don’t know python very well, but it seems this could be related to Windows, Python, “multiprocessing”…and not having the if name == ‘main’: guard?

Sorry if this is documented somewhere, but I couldn’t find anyone else having the issue.

Traceback (most recent call last):

File “”, line 1, in

File “C:\Users\jeremiah.morrill\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py”, line 105, in spawn_main
exitcode = _main(fd)

File “C:\Users\jeremiah.morrill\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py”, line 114, in _main
prepare(preparation_data)

File “C:\Users\jeremiah.morrill\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py”, line 225, in prepare
_fixup_main_from_path(data[‘init_main_from_path’])

File “C:\Users\jeremiah.morrill\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py”, line 277, in _fixup_main_from_path
run_name=“mp_main”)

File “C:\Users\jeremiah.morrill\AppData\Local\Programs\Python\Python37\lib\runpy.py”, line 263, in run_path
pkg_name=pkg_name, script_name=fname)

File “C:\Users\jeremiah.morrill\AppData\Local\Programs\Python\Python37\lib\runpy.py”, line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)

File “C:\Users\jeremiah.morrill\AppData\Local\Programs\Python\Python37\lib\runpy.py”, line 85, in _run_code
exec(code, run_globals)

File “C:\src\mediatubes\ml\tensortubes\models\mxnet\retinaface\r50_v1\test.py”, line 216, in
tune_and_evaluate(tuning_option)

File “C:\src\mediatubes\ml\tensortubes\models\mxnet\retinaface\r50_v1\test.py”, line 176, in tune_and_evaluate
tune_tasks(tasks, **tuning_opt)

File “C:\src\mediatubes\ml\tensortubes\models\mxnet\retinaface\r50_v1\test.py”, line 134, in tune_tasks
tuner_obj = XGBTuner(tsk, loss_type=‘rank’, num_threads=1)

File “C:\Users\jeremiah.morrill\AppData\Roaming\Python\Python37\site-packages\tvm-0.6.dev0-py3.7-win-amd64.egg\tvm\autotvm\tuner\xgboost_tuner.py”, line 74, in init
log_interval=log_interval // 2)

File “C:\Users\jeremiah.morrill\AppData\Roaming\Python\Python37\site-packages\tvm-0.6.dev0-py3.7-win-amd64.egg\tvm\autotvm\tuner\xgboost_cost_model.py”, line 145, in init
self._reset_pool(self.space, self.target, self.task)

File “C:\Users\jeremiah.morrill\AppData\Roaming\Python\Python37\site-packages\tvm-0.6.dev0-py3.7-win-amd64.egg\tvm\autotvm\tuner\xgboost_cost_model.py”, line 164, in _reset_pool
self.pool = multiprocessing.Pool(self.num_threads)

File “C:\Users\jeremiah.morrill\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py”, line 119, in Pool
context=self.get_context())

File “C:\Users\jeremiah.morrill\AppData\Local\Programs\Python\Python37\lib\multiprocessing\pool.py”, line 176, in init
self._repopulate_pool()

File “C:\Users\jeremiah.morrill\AppData\Local\Programs\Python\Python37\lib\multiprocessing\pool.py”, line 241, in _repopulate_pool
w.start()

File “C:\Users\jeremiah.morrill\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py”, line 112, in start
self._popen = self._Popen(self)

File “C:\Users\jeremiah.morrill\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py”, line 322, in _Popen
return Popen(process_obj)

File “C:\Users\jeremiah.morrill\AppData\Local\Programs\Python\Python37\lib\multiprocessing\popen_spawn_win32.py”, line 46, in init
prep_data = spawn.get_preparation_data(process_obj._name)

File “C:\Users\jeremiah.morrill\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py”, line 143, in get_preparation_data
_check_not_importing_main()

File “C:\Users\jeremiah.morrill\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py”, line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.’’’)

RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

#2

I added the __main__
check on the entrypoint of my script:

> if __name__ == '__main__':
>     tune_and_evaluate(tuning_option)

This allows it to get further, but eventually crashes another way:

> [Task  1/39]  Current/Best:    7.31/ 828.67 GFLOPS | Progress: (36/540) | 222.44 sTraceback (most recent call last):
> 
>   File "test.py", line 217, in <module>
>     tune_and_evaluate(tuning_option)
> 
>   File "test.py", line 176, in tune_and_evaluate
>     tune_tasks(tasks, **tuning_opt)
> 
>   File "test.py", line 154, in tune_tasks
>     autotvm.callback.log_to_file(tmp_log_file)])
> 
>   File "C:\Users\jeremiah.morrill\AppData\Roaming\Python\Python37\site-packages\tvm-0.6.dev0-py3.7-win-amd64.egg\tvm\autotvm\tuner\xgboost_tuner.py", line 86, in tune
>     super(XGBTuner, self).tune(*args, **kwargs)
> 
>   File "C:\Users\jeremiah.morrill\AppData\Roaming\Python\Python37\site-packages\tvm-0.6.dev0-py3.7-win-amd64.egg\tvm\autotvm\tuner\tuner.py", line 150, in tune
>     self.update(inputs, results)
> 
>   File "C:\Users\jeremiah.morrill\AppData\Roaming\Python\Python37\site-packages\tvm-0.6.dev0-py3.7-win-amd64.egg\tvm\autotvm\tuner\model_based_tuner.py", line 266, in update
>     self.cost_model.fit(self.xs, self.ys, self.plan_size)
> 
>   File "C:\Users\jeremiah.morrill\AppData\Roaming\Python\Python37\site-packages\tvm-0.6.dev0-py3.7-win-amd64.egg\tvm\autotvm\tuner\xgboost_cost_model.py", line 181, in fit
>     x_train = self._get_feature(xs)
> 
>   File "C:\Users\jeremiah.morrill\AppData\Roaming\Python\Python37\site-packages\tvm-0.6.dev0-py3.7-win-amd64.egg\tvm\autotvm\tuner\xgboost_cost_model.py", line 321, in _get_feature
>     ret = np.empty((len(indexes), feature_len), dtype=np.float32)
> 
> TypeError: 'NoneType' object cannot be interpreted as an integer

#3

As far as I know, AutoTVM doesn’t work on Windows unfortunately (related discussion).


#4

I successfully autotuned a cuda mobilenet based model using gridsearch on windows. just xgb seems to have some issues.