Abstract

Knowledge Discovery in Databases, or KDD for short, is often described as to identify valid, useful, meaningful, unknown, and unexpected relationships in large databases. As the terms “unknown” and “unexpected” make obvious KDD is an exploratory process. Data mining – the main technique employed in KDD – concentrates typically on building statistical models or classifiers and predictors based on machine learning and thus covers only the aspects of the KDD definition that deal with “useful” and “meaningful”. Techniques for truly discovering the unknown and the unexpected are much less prevalent and certainly have not yet made their way from the research labs into commercial software platforms in contrast to the widely available data mining tools. Large organisations like BT that collect huge amounts of data about its internal processes and its interactions with customers are facing a discovery challenge. How do we detect relevant patterns across our distributed data silos that point to important future events or developments that are yet still unknown and unexpected and how can we do this pro-actively?

About the author: Dr. Detlef Nauck is a Chief Research Scientist at BT’s Intelligent Systems Research Centre. He is also a Visiting Professor and Bournemouth University and a Private Docent at the Otto-von-Guericke University of Magdeburg. He has won two BCS medals and has served as an Associate Editor of IEEE Transactions on Systems, Men and Cybernetics – Part B. Dr. Nauck’s current research interests include the application of Computational Intelligence, Intelligent Data Analysis and Machine Learning in different areas of Business Intelligence.