Why your Models need Maintenance

MartinLiebig

People often think a given model can just be put into deployment forever. In fact, the opposite is true. You need to maintain your models like you maintain a machine. Machine Learning models can get off or broken overtime. This sounds odd to you because they have no moving pieces? Well, you might want to have a close look on change and drifts of concept.

Change of Concept

Let’s start off with an example. If you try to build a predictive maintenance model for an air plane, you often create columns like

Error5_occured_last_5_mins

as an input for your model. But what happens if error number 5 is not error number 5 anymore? Software updates can drastically change the data you have. They fix known issues but also encode your data in a different way. If you take the post-update data as an input for your pre-update model — it will do something, but not what you expected. This phenomenon is called change of concept.

Drift of Concept

A very similar phenomenon is drift of concept. This happens if change is not drastic but emerging slowly. An industrial example is encrustment of a sensor. This happens over time and a measured 100 degrees are not 100 degrees anymore. An example in customer analytics are adoption processes of new technology. People did not use iPhones at once, but slowly adopted to it. A column like “HasAnIphone” would mean a very tech-savvy person — in 2007. Today this indicates an average person.

What Can I Do?

 An example for window based relearning. The pattern to detect circles moves over time. Only recent data points are included to built a model.

 

A common approach to overcome concept drifting is window based relearning. ....... Read more on my medium.com page