Implementation and Optimization for Deep Compressed Learning
Tian, Yufei (2019-10-22)
Implementation and Optimization for Deep Compressed Learning
(22.10.2019)
Julkaisu on tekijänoikeussäännösten alainen. Teosta voi lukea ja tulostaa henkilökohtaista käyttöä varten. Käyttö kaupallisiin tarkoituksiin on kielletty.
suljettu
Julkaisun pysyvä osoite on:
https://urn.fi/URN:NBN:fi-fe2019121648316
https://urn.fi/URN:NBN:fi-fe2019121648316
Tiivistelmä
In recent years, Machine Learning (ML), especially deep learning, has developed rapidly and been widely applied. This trend has brought deep learning to face the problems of large data explosion and model complexity more and more. At the same time, Compressed Sensing (CS, or Compressed Sampling) technology, which can compress data with certain sparsity condition and then reconstruct it by non-linear method, significantly reduces the amount of data processing. However, CS is also limited to the complexity of reconstruction method and strict requirements on data sparsity.
This thesis investigates the Compressed Learning (CL) algorithm, which combines the Compressed Sensing and Machine Learning together. CL directly learns on the compressed measurement domain without reconstructing the raw signal. By compressing the data, CL reduces input size of learning network with similar learning results. Based on the traditional CL method, our work reduces the network parameters, extends the application scenarios and maintains the accuracy of CL further. Our main work and contributions are as follows:
1. In order to further reduce the network parameters and ensure the recognition accuracy, we propose three optimization methods for the traditional CL algorithm: In-dividual Network Parameters method under different measurement rate, CL with Long Short-Term Memory (LSTM) method and CL with Binarized Neural Network (BNN) method.
2. To simulate our optimization method, we have modeled normal Convolutional Neural Network (CNN), Network in Network (NIN), LSTM Network and BNN. With these network models, we design experiments on Modified National Institute of Standards and Technology (MNIST) database, In-house Ship-taken Picture dataset for recognition on driverless ships and Human Activity Recognition (HAR) dataset.
3. Through the experiment, our Individual Network Parameters optimization method has been proved to make the network parameters decrease further: on MNIST and In-house Ship-taken Picture dataset, the network parameters reduce to 19.11% and 52.65% respectively when measurement rate is 0.1. By optimizing the measurement matrix, we improve the accuracy on LSTM network under all tested measurement rate and prove LSTM network method is possible for CL. As for BNN method, when the measurement rate is above or equal to 0.05, our experiment proves that on MNIST, BNN method can achieve lower than 3.32% accuracy reducing with about 5 times faster inference time which is an acceptable result; while on Ship-taken Picture dataset, the accuracy is 10% lower than float point network method. This result illustrates CL with BNN method is not suitable for Ship-taken Picture dataset.
This thesis investigates the Compressed Learning (CL) algorithm, which combines the Compressed Sensing and Machine Learning together. CL directly learns on the compressed measurement domain without reconstructing the raw signal. By compressing the data, CL reduces input size of learning network with similar learning results. Based on the traditional CL method, our work reduces the network parameters, extends the application scenarios and maintains the accuracy of CL further. Our main work and contributions are as follows:
1. In order to further reduce the network parameters and ensure the recognition accuracy, we propose three optimization methods for the traditional CL algorithm: In-dividual Network Parameters method under different measurement rate, CL with Long Short-Term Memory (LSTM) method and CL with Binarized Neural Network (BNN) method.
2. To simulate our optimization method, we have modeled normal Convolutional Neural Network (CNN), Network in Network (NIN), LSTM Network and BNN. With these network models, we design experiments on Modified National Institute of Standards and Technology (MNIST) database, In-house Ship-taken Picture dataset for recognition on driverless ships and Human Activity Recognition (HAR) dataset.
3. Through the experiment, our Individual Network Parameters optimization method has been proved to make the network parameters decrease further: on MNIST and In-house Ship-taken Picture dataset, the network parameters reduce to 19.11% and 52.65% respectively when measurement rate is 0.1. By optimizing the measurement matrix, we improve the accuracy on LSTM network under all tested measurement rate and prove LSTM network method is possible for CL. As for BNN method, when the measurement rate is above or equal to 0.05, our experiment proves that on MNIST, BNN method can achieve lower than 3.32% accuracy reducing with about 5 times faster inference time which is an acceptable result; while on Ship-taken Picture dataset, the accuracy is 10% lower than float point network method. This result illustrates CL with BNN method is not suitable for Ship-taken Picture dataset.