Abstract

Clustering is a central problem in data mining and statistical pattern recognition with a long and rich history. The advent of Big Data has introduced important challenges to existing clustering methods in the form of high-dimensional, high-frequency, time-varying streams of data. Up-to-date research on Big Data clustering has been almost exclusively focused on addressing individual aspects of the problem in isolation, largely ignoring whether and how the proposed methods can be extended to address the overall problem. We will discuss an incremental divisive clustering approach for high-dimensional data that has storage requirements that are low and more importantly independent of the stream size, and can identify changes in the population distribution that require a revision of the clustering result.