Technical Report MSC-2021-31

Title: ML Based Lineage in Databases
Authors: Michael Leybovich
Supervisors: Oded Shmueli
PDFCurrently accessibly only within the Technion network
Abstract: There has been extensive research on data provenance, mostly concerned with annotating the results of database (DB) queries with provenance information which is useful in explaining query results at various resolution levels. In this work, we track the lineage of tuples throughout their database lifetime. That is, we consider a scenario in which tuples (records) that are produced by a query may affect other tuple insertions into the DB, as part of a normal workflow. As time goes on, provenance explanations for such tuples become deeply nested, increasingly consuming space, and resulting in decreased clarity and readability. We present a novel approach for approximating lineage tracking. We use Machine Learning (ML) and Natural Language Processing (NLP) techniques; mainly, word embedding. The basic idea is summarizing (and approximating) the lineage of each tuple via a small set of constant-size vectors (the number of vectors per-tuple is a hyperparameter). For explicitly (and independently of DB contents) inserted tuples - the vectors are obtained via a pre-trained word vectors model over their underlying database domain ``text". During the execution of a query, we construct the lineage vectors of the final (and intermediate) result tuples in a similar fashion to that of semiring-based exact provenance calculations. We extend the + and * operations to generate sets of lineage vectors, while emphasizing the ability to propagate information and preserve the compact representation. Therefore, our solution does not suffer from space complexity blow-up over time, and it ``naturally ranks" explanations to the existence of a tuple in the DB. We devise a genetics-inspired improvement to our basic method. The data columns of an entity (and potentially other columns) are a tuple’s basic properties, i.e., the ``genes" that combine to form its genetic code. Thus, finding the lineage of a tuple in the DB is analogous to finding its predecessors via DNA examination. We design an alternative lineage tracking mechanism, that of keeping track of and querying lineage (via embeddings) at the column (``gene") level; thereby, we manage to better distinguish between the provenance features and the textual characteristics of a tuple. We further introduce several improvements and extensions to the basic method: column emphasis, Bloom Filters of queries, tuple creation timestamp, query dependency DAG and where provenance. We integrate our lineage computations into the PostgreSQL system via an extension (ProvSQL) and experimentally exhibit useful results in terms of accuracy against exact, semiring-based, justifications. In the experiments, we focus on tuples with multiple generations of tuples in their lifelong lineage and analyze them in terms of direct and distant lineage. The experiments suggest a high usefulness potential for the proposed approximate lineage methods and the further suggested enhancements. This especially holds for the column-based vectors method which exhibits high precision and per-level recall. Finally, we outline a search algorithm for vector sets, which is based on encoding a set of vectors via a ``long" single vector.
CopyrightThe above paper is copyright by the Technion, Author(s), or others. Please contact the author(s) for more information

Remark: Any link to this technical report should be to this page (, rather than to the URL of the PDF files directly. The latter URLs may change without notice.

To the list of the MSC technical reports of 2021
To the main CS technical reports page

Computer science department, Technion