To rapidly analyze and dig deep into vast amounts of online financial information to monitor public opinion, receive early warnings, and perform positive and negative business analysis.

China’s securities industry has entered the Big Data era. With vast amounts of financial information posted online, there are ample opportunities to monitor public opinion, receive early warnings and perform positive and negative business analysis. Financial service institutions have to be able to rapidly analyze and dig deeper into this data.

Faced with the massive amount of news material and documents submitted by listed companies, improving the efficiency of information processing was proving a headache. Furthermore, this information does not just include structured data from the website news pages and various data sources within the company. It also covers a wide range of unstructured information, including text, audio and video from various social media forums, blogs, message boards and messaging applications. The daily volume of queries can reach one million. Without a unified data processing platform, information retrieval was difficult.

Details

解決方法

Xi’an Panorama Data previously used open source technology to build a data processing platform, but information retrieval and stability were both poor. Based on these challenges, the company had a number of detailed requirements for the construction of its new data processing platform:

Smart. The range of data information is extremely varied and a number of smart methods are required to complete the automatic classification and smart retrieval of information, as well as tasks such as smart crawling of text, video, and audio sources. The efficiency of queries and publishing needs to be improved greatly without increasing staff.
Efficiency, stability, reliability and ease of use. Meet the requirement of long-term secure operation, while being capable of processing large amounts of data and improving the ease to operate.
High quality. Guarantee data quality by selecting appropriate video and audio file formats, reducing transcoding links, and reducing the quality loss caused by transcoding.
Capacity. Be able to process large amounts of unstructured data from different sources.
Scalability. Manage massive amounts of media and other data and be able to expand quickly with data growth.

A data processing system that would meet those requirements needed to be able to automatically crawl and process the data from Xi’an Panorama Data’s various internal data sources, as well as various structured and unstructured information on Panorama Network It needed to be able to understand the information using conceptual and contextual semantic association. OpenText Knowledge Discovery allows the user to find pattern and concept matches, and is able to automatically link these to the relevant accurate information across text, audio, and video from various media.

OpenText Knowledge Discovery is able to automatically analyze and sort any amount or type of data with great accuracy and speed. It is able to classify data into logically similar concept clusters on the basis of associated or similar themes, automate the originally daunting task of searching through various data source sites, and increase productivity.

It uses multiple retrieval methods, including arbitrary keyword searches and criteria searches. It also has a ‘fuzzy’ search feature that enables users who do not know the specific query content to check words that are similar to the input string and find relevant results. By indexing text label fields, a field label search can select field label combinations in a targeted manner and return the corresponding limited results.

Traditional data memory usually only allows one process to run, to ensure data updates are effective even in the event of software system failure. When update processes wait for each other because of a particular piece of data, resulting in a delay, this reduces the operating speed of the system. OpenText Knowledge Discovery is able to implement the distributed processing of large amounts of data and retrieve content distributed across multiple machines. Its original site management technology eliminates the need to replicate all data indexed in the current location, reducing storage costs and the risk of duplication. After indexing, data is parallel processed on multiple machines. Different query commands can be invoked at any time during retrieval. This greatly improves search and operation speeds and reduces processing times.

Also, the system supports an automatic clustering feature that can automatically analyze all the information content collected. This clusters similar files together based on the concepts in their content, while automatically generating category titles and analyzing for hotspots and trends. In every search, OpenText Knowledge Discovery can retrieve all relevant information based on the search result; and automatically provide those relevant information to users together with the search result. This allows users to access all relevant information based on time and relevancy, which also enhances work efficiency.

OpenText Knowledge Discovery [IDOL] helped to automatically search for and extract key concepts from a massive amount of text, video and audio data on a daily basis. This has significantly enhanced user experience and productivity, quality of information, and reduced operating costs.
Zhou Qing
Research and Development Engineer, Xi’an Panorama Data Co., Ltd.

結果

By using OpenText Knowledge Discovery to build a data processing platform, Xi’an Panorama Data significantly improved its employees’ query retrieval efficiency. OpenText Knowledge Discovery has enabled Xi’an Panorama Data to quickly access and understand all information assets from within the company and its Panorama Network website including text, images, audio, social media and video and to find content quickly and accurately. This enables it to provide tendency analysis reports that help listed companies get a handle on positive and negative public opinion quickly and make better strategic decisions. Accurate analysis results can also help enable ordinary investors to pinpoint their investment objectives and seize the best investment opportunities. In addition, regulatory authorities are able to oversee market trends in real-time and prevent financial risks, on the basis of public opinion towards listed companies.

According to Zhou Qing, research and development engineer at Xi’an Panorama Data, “Our data information is stored in different libraries according to the day, week and month. OpenText Knowledge Discovery can switch smoothly between libraries, ensuring the integrity of the retrieved information.”

OpenText Knowledge Discovery provides a full range of highly detailed time-coded data results. It can perform 2,000 queries per second across all indexed data, while its response time is less than a second. It has helped Xi’an Panorama Data to use different commands to automatically search for and extract key concepts from a massive amount of daily query information. This has significantly enhanced user experience and productivity, in addition to reducing operating costs.

Our information is stored in different libraries according to the day, week and month. OpenText Knowledge Discovery [IDOL] switches smoothly between libraries, ensuring the integrity of the retrieved information.
Zhou Qing
Research and Development Engineer, Xi’an Panorama Data Co., Ltd.

Xi’an Panorama Data’s affiliated media brands, including the Panorama Network website, Trading Day and World of Wealth, and its interaction platforms for listed company investors and public opinion monitoring services have been highly influential within the industry. OpenText Knowledge Discovery has helped the company to build a processing platform that provides a comprehensive view of all its business-critical data. This has enabled it to keep abreast of the latest information, create reports and manage important information, from social media posts to files created by productivity tools.

In the future, Xi’an Panorama Data wants to use more OpenText Big Data technology to mine even more valuable information from huge volumes of industry data in order to draw workable insights and increase its competitive advantage.

About Xi'an Panorama Data

Xi’an Panorama Data specializes in Chinese capital market information and building platforms for interaction between capital market participants. It wanted to improve the accuracy and speed of its data mining. This would enable faster, more detailed reporting, improving clients’ decision-making. Accurate analysis also helps strengthen regulatory oversight and reduces risk.