Design and Implementation of High Concurrency Weibo Data : System Based on Node.js
Chen, Guo (2018-02-12)
Design and Implementation of High Concurrency Weibo Data : System Based on Node.js
Chen, Guo
(12.02.2018)
Tätä artikkelia/julkaisua ei ole tallennettu UTUPubiin. Julkaisun tiedoissa voi kuitenkin olla linkki toisaalle tallennettuun artikkeliin / julkaisuun.
Turun yliopisto
Tiivistelmä
With the progress of science and technology, People are more inclined to reveal their own voice, show their life status on a variety of social networking sites. Foreign websites Twitter, Facebook, and domestic Sina, Tencent are constantly playing the power of their platforms to change the traditional way of information dissemination. Under those circumstances, improving delivery efficiency and saving time to obtain value information brings great commercial value. Since society today is that of an information explosion, more and more websites are facing high concurrent access pressure.
The main task of this paper is to design and implement a high concurrency Weibo data system based on Node.js. First I analyzed the challenges of the high concurrency Web system, and summarized the existing high concurrency solutions. Then designed and analyzed the commonly used data collection schemes. According to the actual situation, the most effective technique was applied to this system. And the next chapter demonstrated system design and implementation, the overall system architecture , multicore HTTP server design, and functional module was illustrated first, while the basic functions are including but not limited to, user registration and login, crawling and updating microblog content, user information and microblog information (attitudes, comments, reposts, whether it is original etc.). Due to the whole system uses MVC design pattern, therefore the software was designed and implemented layer by layer. In order to ensure the normal operation of the whole system avoid the deficiencies of manual operations, timer module was implemented to regularly acquire data from the Sina server and stored in database. Furthermore, the front-end pages use AdminLTE framework to provide good visual representations.
Owing that Node.js is single-thread mode, so as to make full use of the multi-core servers, I designed and implemented an efficient HTTP server, which greatly improves the performance of the HTTP server. Then, the multiple tables was designed based on MySQL to persist the data. Besides, NoSQL database Redis was implemented as a data buffer layer to store data at memory level. It reduces the frequency of accessing to MySQL database directly, furthermore boosts response speed.
Finally, the system was tested in a real environment, and compared with the traditional HTTP server. It proved that Node.js is more suitable for high concurrency Web system.
The main task of this paper is to design and implement a high concurrency Weibo data system based on Node.js. First I analyzed the challenges of the high concurrency Web system, and summarized the existing high concurrency solutions. Then designed and analyzed the commonly used data collection schemes. According to the actual situation, the most effective technique was applied to this system. And the next chapter demonstrated system design and implementation, the overall system architecture , multicore HTTP server design, and functional module was illustrated first, while the basic functions are including but not limited to, user registration and login, crawling and updating microblog content, user information and microblog information (attitudes, comments, reposts, whether it is original etc.). Due to the whole system uses MVC design pattern, therefore the software was designed and implemented layer by layer. In order to ensure the normal operation of the whole system avoid the deficiencies of manual operations, timer module was implemented to regularly acquire data from the Sina server and stored in database. Furthermore, the front-end pages use AdminLTE framework to provide good visual representations.
Owing that Node.js is single-thread mode, so as to make full use of the multi-core servers, I designed and implemented an efficient HTTP server, which greatly improves the performance of the HTTP server. Then, the multiple tables was designed based on MySQL to persist the data. Besides, NoSQL database Redis was implemented as a data buffer layer to store data at memory level. It reduces the frequency of accessing to MySQL database directly, furthermore boosts response speed.
Finally, the system was tested in a real environment, and compared with the traditional HTTP server. It proved that Node.js is more suitable for high concurrency Web system.