The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here

Dimensionality Reduction with SVD

Muhammed_Fatih_Muhammed_Fatih_ Member Posts: 93 Maven
edited October 2019 in Help
Dear RapidMiner Community, 

I am currently conducting a simple Singular Value Decomposition (SVD) process based on a TDM (Term Document Matrix) I've generated consdering Communication data. The matrix has 9662 attribute columns (terms) and in total 72826 rows. My SVD process is running for four days and has not finished yet. Especially the SVD operator itself is still loading (see attached). 

Could you tell me what a "normal" computational time for such a matrix is with regard to dimenisonality reduction with SVD? Do I really need several days to compute SVD?

Thank you in advance for your help & support ! 

Best regards

Muhammed 


Tagged:

Answers

  • rfuentealbarfuentealba RapidMiner Certified Analyst, Member, University Professor Posts: 568 Unicorn

    Given the 9662x72826 matrix and the nature of the calculations, I would say it's very likely that your process is still working. I would take a look at how the computer memory, processor and swap are behaving, because if the data doesn't fit into RAM, it will begin swapping on disk, making disk access (and the entire process) effectively slower.

    Hope this can help,

    Rod.
  • Muhammed_Fatih_Muhammed_Fatih_ Member Posts: 93 Maven
    Hello @rfuentealba, 

    thank you for your answer. The memory-, processor behaviour looks like the following: 



    I have a 32 gb memory on my computer. What would you say? Do you think, that additonal memory is needed to compute the process? Or are there alternatives with regard to SVD computation in order to speed up the process? 

    Thank you in advance for your answer. 

    Muhammed
  • rfuentealbarfuentealba RapidMiner Certified Analyst, Member, University Professor Posts: 568 Unicorn

    I am not a Microsoft user (haven't used it since 1995) and I don't know the internals of that OS, hence I had a lot of trouble reading what does memory paging means. I don't see that the machine is using swap memory, which is a clear indicative that your computer needs more memory and it's trying to use the hard disk for it. However, I found somewhere else that Windows only uses swap in case of a crash.

    I think @Marco_Boeck or @pschlunder can have a better technical understanding on what's happening here. Quoting @mschmitz too.

    Now, NLP processes always take a lot of time, this is why I can't say "it's normal" or "it's not". If you could provide a little sample, I could test a few things for you on my servers.

    All the best,

    Rodrigo.
Sign In or Register to comment.