IoT data consolidation

As innovation happens, the need of consolidating legacy systems into a new system many times creates opportunities of improving the overall solution, delivering far better performant system.

The client

A leading German manufacturer of heating systems.

Hundreds of thousands of sensors in the field

A number of systems manage hundreds of thousands of sensors in the field. These multiple systems need to be consolidated into a new system, all historical data needs to be migrated and consolidated. Processing the historical data from legacy systems take five entire days. Processing a current, daily dataset takes about 10 minutes.

Consolidation is needed for innovation

Mathminds engaged in a mission of implementing new pipelines, integrating additional datasets containing data coming from sensors connected to a legacy system. Regardless of the system involved, the data should be consolidated onto a new data schema, in order to enable innovation as the business progresses into a new age of services available to customers.

Enabling a solution

Mathminds rewrote the entire pipeline and implemented additional features as per user requirements. We employed Apache Spark, being able to reduce the processing time of the entire historical data from five days to nearly 10 minutes. Processing a dataset containing a single day can be done in less than a minute.

Conclusion

• All sources of information are consolidated as per client’s requirements.
• The new implementation scales horizontally and performs nearly 700 times faster than the existing implementation.