The Altair Community is migrating to a new platform to provide a better experience for you. In preparation for the migration, the Altair Community is on read-only mode from October 28 - November 6, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here
"Association Rules with Thousands of Products"
a_isabel_marque
Member Posts: 3 
Contributor I
Hi,
I am trying to create association rules.
My dataset have 30496 transactions and 18270 products and it is stored in a MySQL database like this:
transaction | product
1 | 0987
1 | 6543
2 | 1234
... | ...
Before use FP-Growth operator I convert data with Nominal to Numerical operator and aggregate the rows of the same transaction. After that I convert data again with Numerical to Binomial operator.
I run the process with RapidAnalytics but it returns the error "java.lang.OutOfMemoryError: GC overhead limit exceeded ". And I used Free Memory operator.
How can I solve this problem? It is possible to create association rules with this number of products?
Other question, it is possible to convert data to a sparse matrix?
I am studing other approach in other database. I installed in a PostgreSQL database the MADlib library. And with the same dataset, MADlib function create the association rules in 30 seconds without any transformation of the data!!!
Best regards.
I am trying to create association rules.
My dataset have 30496 transactions and 18270 products and it is stored in a MySQL database like this:
transaction | product
1 | 0987
1 | 6543
2 | 1234
... | ...
Before use FP-Growth operator I convert data with Nominal to Numerical operator and aggregate the rows of the same transaction. After that I convert data again with Numerical to Binomial operator.
I run the process with RapidAnalytics but it returns the error "java.lang.OutOfMemoryError: GC overhead limit exceeded ". And I used Free Memory operator.
How can I solve this problem? It is possible to create association rules with this number of products?
Other question, it is possible to convert data to a sparse matrix?
I am studing other approach in other database. I installed in a PostgreSQL database the MADlib library. And with the same dataset, MADlib function create the association rules in 30 seconds without any transformation of the data!!!
Best regards.
Tagged:
0
Answers