The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here

Why is my fetch taking so long?

DU_caDU_ca Member Posts: 3 Learner I
in Help
I am able to connect to my Oracle database system, but the "Fetching Database Tables" process takes 45 minutes to complete.  Why is this happening?

Answers

  • rfuentealbarfuentealba RapidMiner Certified Analyst, Member, University Professor Posts: 568 Unicorn
    Hello @DU_ca:

    It appears to be something not fully compliant in the JDBC driver for Oracle that requires an extra step. Let's call @mschmitz and ask him if he knows someone who can help us, because I tried to clone the RapidMiner Studio source code and found out that the open source version of it doesn't include the JDBC connectors, so I just used Navicat and copy/pasted :-(

    All the best,

    Rodrigo.
  • DU_caDU_ca Member Posts: 3 Learner I
    Thanks for trying, Rodrigo.  Unfortunately, I'm still having this problem and so I guess RapidMiner is not a good option for me.
  • jczogallajczogalla Employee-RapidMiner, Member Posts: 144 RM Engineering
    Hi @DU_ca!
    Can you maybe share your process and/or explain what exactly you want to do?
    If you are trying to load all your data from your database, this might be quite some effort to do if you have a huge database. RapidMiner has to convert each table into its own memory format, which might take a lot of RAM and time.
    If you want to manipulate a lot of your database data, especially preprocessing, it maybe would help you to use the in-DB extension from the marketplace, as discussesd here: https://community.rapidminer.com/discussion/55642/challenge-with-rm-server-running-out-of-memory#latest
    I hope this helps!
    Jan

  • sgenzersgenzer Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager
    using the caching tools in the Jackhammer extension can also be helpful here...

    https://oldworldcomputing.com/en/tutorial-introduction-to-caching-functions-of-the-jackhammer-extension-by-old-world-computing/

    Scott
  • rfuentealbarfuentealba RapidMiner Certified Analyst, Member, University Professor Posts: 568 Unicorn
    Hey @jczogalla and @sgenzer,

    Since I've found this problem too, I can provide a few details:

    This is not about the amount of data. In fact, the error comes from trying to retrieve the list of tables from a database in the Parameters list. However, this was solved in RapidMiner 9.3. It's still a little slow (40 seconds on my MacBook i9-9900) but manageable.



    @DU_ca, you may want to upgrade to 9.3 to see if this works.

    All the best,

    Rodrigo.

  • DU_caDU_ca Member Posts: 3 Learner I
    Upgrading to 9.3 didn't help.  It's still taking a long time.  I also use SQL Developer for some of my work and it loads everything within seconds.  Why the difference?
  • sgenzersgenzer Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager
    hi @DU_ca so as was explained above, RapidMiner is a completely different architecture to SQL. If your db has a mountain of tables, RM has to go through the schema, grab them all, and then display them in its own architecture. 

    I guess if it were me I would take a small step back and ask what are you trying to do? Most of the time if you have a large db it is far more efficient to simply push pre-written SQL queries (e.g. write them in SQL Developer) using the Read Database operator and then grab the results for future work in RapidMiner. I am certain that SQL Developer is going to be a lot easier & faster for writing SQL than the very basic editor in RapidMiner. It's not what RM is built for. Most data scientists have a variety of tools that they use in tandem...no one tool is going to be the most efficient in all circumstances.

    Does that help?

    Scott

  • sgenzersgenzer Administrator, Moderator, Employee-RapidMiner, RapidMiner Certified Analyst, Community Manager, Member, University Professor, PM Moderator Posts: 2,959 Community Manager
    hi @Blodgett20 what kind of accurate solution are you looking for that has not been addressed here? We're happy to help but I'd like to know more about what exactly your struggles are.

    Thanks.

    Scott

Sign In or Register to comment.