Real-time consumer monitoring

Hemsi

Hi, 
I am currently researching various ways of setting up a real-time consumer monitoring tool which should entail the following: 
- using publicly available (daily or weekly) data from social media, forums, blogs, search engines, etc. 
- analyzing current and upcoming trends (eg. are there any topics which are growing in relevance)
- integrating macroeconomic data
- integrating demographic data 
- visualizing the data through dashboards and ideally integrate the analysis / reports into Salesforce 

Any thoughts on how to best approach that? Social media data of Facebook, Instagram has been restricted more and more after the API change, any workarounds? 
Would be great if you've have some ideas on how to approach that, what tools to use, etc. 

BalazsBarany

Hi,

seems like you're setting up a new department in your company ;-)

Yes, Facebook stopped giving out too much information after many data scandals. 

But there are some basic questions you need to answer. Are you prepared to properly process personally identifying information according to the different laws? GDPR for EU residents, the Californian law etc.?
Are you able to run the necessary infrastructure (scheduled process execution, in parallel if possible, a powerful database, ...)?
Do you have a good way of matching your customers to external identities (Facebook etc.)? A method for breaking down statistical/macroeconomic/demographic data to the customers? Which problem are you trying to solve? Can you quantify it or find a success criterion?

With external data sources you can forget real-time anyway. Most API interfaces are poll based, so you need to periodically check them. There are some services that have event based interfaces, though, mostly implemented through webhooks. This might be a use case for AI Hub, or you implement it with some other solution. 

There are companies that do this and offer the customer's properties (if they know them) in an API, identified by the email address or other criteria. I've seen projects trying these but stopping because the data weren't useful.

Feel free to ask when you have more focused questions I can help with.

Regards,
Balázs

p3t3r

Hi Balázs, 

I am a newbie so forgive me if this is an irrelevant question. I'm trying to figure out if RapidMiner can help me collect data from a variety of internet sources, particularly social media platforms (i.e. twitter, tik tok, etc) and search engines (i.e. google) to determine what celebrities are most popular at any given moment. Is this something that RapidMiner is capable of? 

BalazsBarany

Hi @p3t3r,

RapidMiner is a good tool for aggregating information from different data sources and then analyzing the data.
 
It doesn't have special interfaces for a lot of social media platforms (it has one for Twitter but not for Tiktok for example). You might want to check out n8n or Zapier for those. 

"What celebrities are most popular" is a question you need to operationalize. Are you interested in mentions? Only positive mentions? Positive minus negative mentions? Only in one country or language or multiple? Etc. 

Mentions can be easily filtered with a predefined list, but you need a process for identifying and adding "new" celebrities to the list. Positive and negative mentions are "sentiment analysis", you can do this in RapidMiner or use readily available web services with operators in RapidMiner for it. 

So RapidMiner would be the central tool for the entire process, but you would probably look into using other tools in addition.

Regards,
Balázs