OpenCL Runtime error


#1

I am trying to run this code with opencl target, but I get the following error (it runs with llvm target without a problem). Does anyone know how to fix it? Thank you.
import numpy as np
import sys
import tvm
import nnvm.compiler
from tvm.contrib import graph_runtime as runtime
import logging
import nnvm.symbol as sym
from nnvm.testing import utils

def main():


    out_channels = 16
    data = sym.Variable(name="data")
    simple_net = sym.conv2d(data=data, kernel_size=(3,3), channels=out_channels, padding = (1, 1), use_bias=True)
    simple_net = sym.batch_norm(data=simple_net)
    simple_net = sym.relu(data=simple_net)

    batch_size = 1
    data_shape = (batch_size, 3, 224, 224)
    net, params = utils.create_workload(simple_net, batch_size, data_shape[1:])
    logging.basicConfig(filename='simple_net.log',level=logging.DEBUG)

    target = "opencl"

    with nnvm.compiler.build_config(opt_level=0):
        graph, lib, params = nnvm.compiler.build(
        net,target=target, shape={"data": data_shape}, params=params)


    try:
        f = open('simple_net_graph_ir.txt', 'w')
    except IOError:
        cprint("Could not create file to dump Graph IR: " + f, 'red')
    f.write(graph.ir())
    f.close()

    try:
        f = open('simple_net_llvm_code.txt', 'w')
    except IOError:
        cprint("Could not create file to dump code: " + f, 'red')
    if target == "llvm":
        f.write(lib.get_source())
    else:
        #this will export the host side code
        f.write(lib.get_source())
        #this will export the device side code
        f.write(lib.imported_modules[0].get_source())
    f.close()

    ctx = tvm.context(target, 0)
    data = np.random.uniform(-1, 1, size=data_shape).astype("float32")
    module = runtime.create(graph, lib, ctx)
    module.set_input(**params)
    module.set_input("data", data)
    module.run()
    out_shape = (batch_size, out_channels, 224, 224)
    out = module.get_output(0, tvm.nd.empty(out_shape))
    out = out.asnumpy()


    return

if __name__ == '__main__':
    main()

[14:13:35] /homes/tvm/src/runtime/opencl/opencl_device_api.cc:242: Initialize OpenCL platform ‘Intel® OpenCL HD Graphics’
[14:13:35] /homes/tvm/src/runtime/opencl/opencl_device_api.cc:251: No OpenCL device any device matched given the options: accelerator mode
[14:13:35] /homes/tvm/src/runtime/opencl/opencl_device_api.cc:269: opencl(0)=‘Intel® Gen9 HD Graphics NEO’ cl_device_id=0x3142120
: CommandLine Error: Option ‘disable-inlined-alloca-merging’ registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options
terminate called after throwing an instance of ‘dmlc::Error’
what(): [14:13:35] /homes//tvm/src/runtime/workspace_pool.cc:96: Check failed: allocated_.size() == 1 (2 vs. 1)

Stack trace returned 10 entries:
[bt] (0) /homes/.local/lib/python3.5/site-packages/tvm-0.4.0-py3.5-linux-x86_64.egg/tvm/libtvm.so(dmlc::StackTraceabi:cxx11+0x5b) [0x7f7540e261bb]
[bt] (1) /homes/.local/lib/python3.5/site-packages/tvm-0.4.0-py3.5-linux-x86_64.egg/tvm/libtvm.so(tvm::runtime::WorkspacePool::Pool::Release(DLContext, tvm::runtime::DeviceAPI*)+0x634) [0x7f75411ef614]
[bt] (2) /homes/.local/lib/python3.5/site-packages/tvm-0.4.0-py3.5-linux-x86_64.egg/tvm/libtvm.so(tvm::runtime::WorkspacePool::~WorkspacePool()+0x3f) [0x7f75411edcef]
[bt] (3) /homes/.local/lib/python3.5/site-packages/tvm-0.4.0-py3.5-linux-x86_64.egg/tvm/libtvm.so(tvm::runtime::cl::OpenCLThreadEntry::~OpenCLThreadEntry()+0xd) [0x7f754124fa9d]
[bt] (4) /lib/x86_64-linux-gnu/libc.so.6(__call_tls_dtors+0x3f) [0x7f754f1345ff]
[bt] (5) /lib/x86_64-linux-gnu/libc.so.6(+0x39f27) [0x7f754f133f27]
[bt] (6) /lib/x86_64-linux-gnu/libc.so.6(+0x3a045) [0x7f754f134045]
[bt] (7) /usr/lib/llvm-6.0/lib/libLLVM-6.0.so.1(llvm::report_fatal_error(llvm::Twine const&, bool)+0xa3) [0x7f753d936113]
[bt] (8) /usr/lib/llvm-6.0/lib/libLLVM-6.0.so.1(+0x831248) [0x7f753d936248]
[bt] (9) /usr/lib/llvm-6.0/lib/libLLVM-6.0.so.1(+0x8255ed) [0x7f753d92a5ed]

Aborted (core dumped)


#2

Just want to follow up on the error. I also encounter this problem when running on Intel HD Graphics. In my case, No OpenCL device error can be solved by using sudo. But even in root privilege, I get the LLVM error as well: : CommandLine Error: Option ‘disable-inlined-alloca-merging’ registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options
terminate called after throwing an instance of ‘dmlc::Error’. I followed the instructions of this post, but it doesn’t work for me. Also in my case, it happened during compiling after import tvm.


#3

Seems like, this is known issue https://bugs.llvm.org/show_bug.cgi?id=30587


#4

@tqchen For this error: terminate called after throwing an instance of ‘dmlc::Error’
what(): [14:13:35] /homes//tvm/src/runtime/workspace_pool.cc:96: Check failed: allocated_.size() == 1 (3 vs. 1), I find that “alloca” generated from src/pass/lower_tvm_builtin.cc is executed twice but “free_stmt” is never executed. Any idea why?


#5

[14:13:35] /homes/tvm/src/runtime/opencl/opencl_device_api.cc:269: opencl(0)=‘Intel® Gen9 HD Graphics NEO’ cl_device_id=0x3142120
: CommandLine Error: Option ‘disable-inlined-alloca-merging’ registered more than once!

fix for GEN is merged into master. Currently you have to build NEO from sources to run TVM on Intel GPU


#6

@cuda-geek So I built from source https://github.com/intel/intel-graphics-compiler, and then generate .deb packages for igc-opencl and igc-core. Then I installed the other deb packages from https://github.com/intel/compute-runtime/releases. But I still get the above LLVM error. Is there anything I did wrong?


#7

@Laurawly, honestly, I have not tried this mixed setup… I build everything from sources. It might be the case that you have both versions of compiler installed and NEO from runtime uses the broken one. I’d have tried clean build-from-source setup first if I were you


#8

@cuda-geek So I built everything from source (and before that I uninstalled all intel related packages). But I’m still getting the same error.


#9

@cuda-geek, I just checked and found that my error message was actually was “: CommandLine Error: Option ‘disable-symbolication’ registered more than once!” A little different from yours.


#10

It doesn’t really matter the exact command line option name. It depends on the static variable linkage order. In this case please also make sure that you build against the same version of llvm/clang in TVM and in IGC. I use 7.0 from

LLVM_DIR:PATH=/usr/lib/llvm-7/cmake

#11

@cuda-geek Could you let me know what are the commit numbers you used for building IGC form source including the sub-directories’ commit numbers. Also did you follow the installation instructions from here: https://github.com/intel/intel-graphics-compiler/blob/master/README.md


#12

I have exactly the same issue even I built everything from source…

I built everything from source following the instructions from:




Also I saw the patch https://github.com/intel/intel-graphics-compiler/pull/49/commits is included.

As the instruction in igc specified, llvm is built from source and most of the libs are static.
To make sure I’m using the same LLVM for both tvm and Neo, I uninstalled the preinstalled LLVM in system and export the env PATH to include the path for which the bin folder is built from llvm source.

Then I tried the tutorial get_started.py and changed the target to ‘opencl’:

: CommandLine Error: Option ‘disable-symbolication’ registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options

I wonder if I should NOT use the LLVM built from source but the one installed by apt-get? In that case, llvm is loaded as shared libs and won’t have such symbol duplication?


#13

@ Laurawly, did you solve this issue?


#14

@dliang0406 No, the method above didn’t work for me. But I was able to make it work by separating the compilation and run or by using rpc: One machine has only TVM runtime, another is build with OpenCL. Then we use RPC to run.


#15

I have found a solution, which works for me and should be helpful also for others. It’s been tested on Ubuntu 18.04.

  1. Follow building and installation guide available here

  2. Download correct LLVM version from GitHub (probably official release should also work) - at the time of writing it is version 7.0.0. You can see version number that has to be downloaded in IGC installation part of #1

  3. Build and install llvm (only llvm is required from package downloaded in #2) according to instruction

  4. Set USE_LLVM variable in config.cmake to <your_llvm_install_dir>/bin/llvm-config

  5. After building TVM everything should be working