Hive driver is not allowed to run

21310122

I wanted to connect the impala db and the driver is forbidden.

Why?

 

This is my code, trying to get test a connection, but it got the error message:"no suitable driver for ***"。

I checked this code is other project before and it went well. 

 

 

So I debuged it and found the error line, it's in the RapidMiner code that a class named "DriverAdapter" cannot be found:

 

Does anybody know how to fix it?

 

mborbely

Hi,

 

Sorry, I was a bit too hasty about my suggestion. Since this is a project dependency, you can only use it if you have it as a submodule in your project. But for that you'll need the source code, which is not possible, since this module is not open-source. You could provide this dep in a way which would only require this module as a Maven artifact, but unfortunately not even the artifact is public.

You can however try some alternative solutions. One of things this extensionconfig does is that it creates a Plugin-Dependencies entry in your META-INF/MANIFEST.MF file, which tells Studio that this extension will require some other extension for functioning. Therefore you could try adding an entry like this to your manifest file: "Plugin-Dependencies: rmx_jdbc_connectors[8.1.0];" Note that this can only work because in this particular case you don't really need the source code of this module at compile-time, only at runtime. And I'm not really sure that this is even going to work, but it's worth a try.

 

The other thing is that Studio only tries to load this driver adapter class for this particular jdbc prefix ("jdbc:hive2://") because you registered an entry for it in Manage Database Drivers. You could also try removing that entry, in which case org.apache.hive.jdbc.HiveDriver will be used directly instead of going to the DriverAdapter first. Problem is that your extension cannot provide the necessary driver classes (as a fat jar for example), because they are only loaded dynamically by the extension class loader, and are not on the classpath of the application, and DriverManager cannot handle this. https://stackoverflow.com/questions/14478870/dynamically-load-the-jdbc-driver This is basically the reason this DriverAdapter magic is in place. So you'll have to put the jdbc jar into the lib folder of your Studio home. Of course if you want to end up with something that's reusable by other people, this is not the best solution.

 

Cheers,

Máté

 

mborbely

Hi,

 

First of all please note that this question does not belong to the Radoop Forum. In the future please use the Developer Forum for such questions.

 

Second, it seems like your extension cannot load the DriverAdapter class, which is located in the rapidminer-extension-jdbc-connectors artifact. If the classloader of your plugin cannot access it, you're probably missing the following entry from your extension config.

 

extensionConfig {
  dependencies {
    extension project(':rapidminer-extension-jdbc-connectors')
  }
}

 

Cheers,

Máté

sgenzer

hi @mborbely - placement was done by me. I saw "Hive" in the message and immediately moved it here to Radoop. It's a valid point whether or not it belongs here or Developer. I can move it.

Scott

 

mborbely

Thanks for your reply. I can understand why one would think this belongs here, but you don't even need to have Radoop installed to bump into this problem. So even though we do similar things in Radoop, this doesn't really belong here. Also many people might be able to answer this question who would never read the Radoop thread. Therefore I think it's better placed there, but it's not a huge problem if it stays.

 

Cheers,

Máté

21310122

First, thank you for your answer, but I can't get the dependency.

Do you know the reason?

 

@mborbely@sgenzer

21310122

Thank you a lot, the website (https://stackoverflow.com/questions/14478870/dynamically-load-the-jdbc-driver ) that you provided to me is very useful!

:smileyhappy:

 

But I got another problem with the lib folder.

According to the website you mentioned(https://stackoverflow.com/questions/14478870/dynamically-load-the-jdbc-driver) ,I need to point the jar file using code like this:

URL u = new URL("jar:file:/path/to/pgjdbc2.jar!/");
String classname = "org.postgresql.Driver";
URLClassLoader ucl = new URLClassLoader(new URL[] { u });
Class.forName(classname, true, ucl);
DriverManager.getConnection("jdbc:postgresql://host/db", "user", "pw");

 

I tried two ways to use this code.

（1）as you suggested, I added needed jar files to the lib folder of RM Studio.

But I realize that I can't use the absolute path in the code to generate URL parameter. Otherwise once the installation address is modified , the required jar file cannot be found.

So how can I get the relative path of RM Studio in my extension project?

 

（2）I chosen to add lib folder in my extension project rather than RM Studio installation folder.

 

This is my code to get the relative path and the jar file.

 

Then I tested the getConnection() method and the result was ok, I got the data from hive database.

 

But when I use it as operator in RM Studio and put this code in doWork() method, it went wrong.

First, the lib folder in my extension project is empty after generates the jar file and the location is not in the resources folder.

 

second, this code is no longer right anymore, it returns null.

private static URL libUrl = ClassLoader.getSystemClassLoader().getResource("lib");

 

And what make me more confused is this result:

 

So what's the result of classpath when I use RM studio with my extension installed？

 

（3）Finally, I used the absolute path and it went ok in RM Studio.

 

 

 

Here is my situation:

    If I use absulte path, it will be less flexible.

    But if I use relative path, it works fine in my local extension project but get error when running as a doWork() method in RM Studio.

 

So If I want to use relative path, what value of URL should I set?

Thank you a lot.

 

@mborbely

 

mborbely

Hi,

 

You slightly misunderstood what I was trying to say about classloaders. You don't want to create a new URLClassloader yourself. My point is that loading jdbc drivers dynamically instead of leveraging Java System-Classpath classloader makes your job harder, since DriverManager cannot handle these classes properly. However RapidMiner Studio creates a separate classloader for your extension and uses it to load the classes which are bundled into your extension fat jar. Therefore if you package your Driver classes into your extension, it will effectively be loaded like in the example of the Stackoverflow link, and you'll end up with the problems mentioned there.

However if you put it into Studio lib folder, it will be loaded by the System-Classpath classloader, and you can bypass all these problems. But whoever wants to use your solution will also have to perform this step, which is acceptable if you only intend it to be used by a handful of people, but a bit clumsy if you want to share it with a larger audience.