Detecting inconsistencies in Big Data extraction : a heuristic approach
Miettinen, Johannes (2019-06-26)
Detecting inconsistencies in Big Data extraction : a heuristic approach
(26.06.2019)
Julkaisu on tekijänoikeussäännösten alainen. Teosta voi lukea ja tulostaa henkilökohtaista käyttöä varten. Käyttö kaupallisiin tarkoituksiin on kielletty.
suljettu
Julkaisun pysyvä osoite on:
https://urn.fi/URN:NBN:fi-fe2019062722222
https://urn.fi/URN:NBN:fi-fe2019062722222
Tiivistelmä
In the era of Big Data, cloud-based analysis services have become a popular support function for enterprises to invest in. By exploiting the data that business functions generate, one might be able to fine-tune their operation in a way that results in an edge in the marketplace. Such ventures inherently involve a number of data streams being directed towards a cloud platform, where they can be efficiently analyzed by a set of tools designed for Big Data analysis. This means that to garner value from data, the data pipelines need to be working properly.
Extracting Big Data onto a cloud analytics platform is an activity where monitoring is often lacking or completely neglected. The sheer volume of data makes it unappealing and quite unfeasible for a human observer to keep track of the extraction activities so that they could detect inconsistencies in either data intake or content. Incidentally, the ones that do eventually detect inconsistencies in the extracted data are human actors trying to run analyses on the data. This means the detection of inconsistencies has been delayed until the very last moment - when data is being leveraged.
This thesis explores mechanisms for early detection of inconsistencies in Big Data extraction. The mechanisms are expected to work in a heuristic manner, meaning requiring only the data itself as input, due to a pervasive absence of control records or metadata in the domain of Big Data. This means the mechanisms must be able to infer normal activity in terms of volume and content to be able to detect inconsistencies. To produce such mechanisms, the thesis reviews the relevant literature for data quality measurements that allow data completeness and consistency to be measured in-data, without external points of reference.
Ultimately, this thesis uncovers seven different measurement hypotheses that could be cultivated to assist in detecting volumetric and content-wise inconsistencies in Big Data extraction. The most promising hypothesis for detecting volumetric inconsistencies is implemented and improved upon by introducing two different time series prediction models to it: the statistics-based ARIMA and the LSTM recurrent neural network. All three implementations of the hypothesis are evaluated against generated extraction signals containing pre-labeled anomalies, where each is shown to exhibit pros and cons in relation to the others.
Extracting Big Data onto a cloud analytics platform is an activity where monitoring is often lacking or completely neglected. The sheer volume of data makes it unappealing and quite unfeasible for a human observer to keep track of the extraction activities so that they could detect inconsistencies in either data intake or content. Incidentally, the ones that do eventually detect inconsistencies in the extracted data are human actors trying to run analyses on the data. This means the detection of inconsistencies has been delayed until the very last moment - when data is being leveraged.
This thesis explores mechanisms for early detection of inconsistencies in Big Data extraction. The mechanisms are expected to work in a heuristic manner, meaning requiring only the data itself as input, due to a pervasive absence of control records or metadata in the domain of Big Data. This means the mechanisms must be able to infer normal activity in terms of volume and content to be able to detect inconsistencies. To produce such mechanisms, the thesis reviews the relevant literature for data quality measurements that allow data completeness and consistency to be measured in-data, without external points of reference.
Ultimately, this thesis uncovers seven different measurement hypotheses that could be cultivated to assist in detecting volumetric and content-wise inconsistencies in Big Data extraction. The most promising hypothesis for detecting volumetric inconsistencies is implemented and improved upon by introducing two different time series prediction models to it: the statistics-based ARIMA and the LSTM recurrent neural network. All three implementations of the hypothesis are evaluated against generated extraction signals containing pre-labeled anomalies, where each is shown to exhibit pros and cons in relation to the others.