Discovering Reliable Approximate Functional Dependencies

Abstract. Given a database and a target attribute of interest, how can we tell whether there exists a functional, or approximately functional dependency of the target on any set of other attributes in the data? How can we reliably, without bias to sample size or dimensionality, measure the strength of such a dependency? And, how can we efficiently discover the optimal or \(\alpha\)-approximate top-\(k\) dependencies? These are exactly the questions we answer in this paper.

As we want to be agnostic on the form of the dependency, we adopt an information-theoretic approach, and construct a reliable, bias correcting score that can be efficiently computed. Moreover, we give an effective optimistic estimator of this score, by which for the first time we can mine the approximate functional dependencies from data with guarantees of optimality. Empirical evaluation shows that the derived score achieves a good bias for variance trade-off, can be used within an efficient discovery algorithm, and indeed discovers meaningful dependencies. Most important, it remains reliable in the face of data sparsity.

Important Dora has been superceded by Fedora

Implementation

the Java source code (June 2017) by Panagiotis Mandros.

Related Publications

Mandros, P, Boley, M & Vreeken, J Discovering Reliable Dependencies from Data: Hardness and Improved Algorithms. In: Proceedings of the IEEE International Conference on Data Mining (ICDM'18), IEEE, 2018. (full paper, 8.9% acceptance rate; overall 19.9%)
Mandros, P, Boley, M & Vreeken, J Discovering Reliable Dependencies from Data: Hardness and Improved Algorithms. Technical Report 1809.05467, arXiv, 2018.
Mandros, P, Boley, M & Vreeken, J Discovering Reliable Approximate Functional Dependencies. In: Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pp 355-363, ACM, 2017. (oral presentation, 8.6% acceptance rate; overall 17.5%)
Mandros, P, Boley, M & Vreeken, J Discovering Reliable Approximate Functional Dependencies. Technical Report 1705.09391, arXiv, 2017.