Because Big Data is a relative term and the use cases and infrastructure in every organization are different, QlikView offers two approaches to handling Big Data that put the power in the hands of our customer to best manage the inherent tradeoffs between user performance and data volume, variety, and velocity.
100% in-memory for maximum user performance
Despite tremendous advances in hard disk technology and the advent of solid state drives (SSDs), the throughput and latency of RAM vs. disk is still orders of magnitude apart. Therefore, the ideal configuration for split-second responses from QlikView is still to bring all the relevant data needed for analysis in memory.
QlikView apps can address the amounts of data that are needed to ensure the relevancy of the app for business users. Here’s how:
- Hardware advances. Recent trends in large memory available on standard Intel hardware enable QlikView to handle ever-larger volumes of data in memory (which provides users with a super-fast, interactive experience). Also, QlikView distributes the number-crunching calculations across all available processor cores to maximize the performance experienced by the user. Unlike technologies that simply “support” multiprocessor hardware, QlikView is optimized to take full advantage of all the power of multi-processor hardware, thereby maximizing performance and the hardware investment.
- QlikView compression. Data brought into QlikView’s in-memory engine can typically be compressed to a tenth of its original size, meaning a single 256GB server can handle uncompressed data sets near 2TB in size. QlikView’s compression scheme means the more redundancy in the data values, the greater the compression.
- Distributed servers. With distributed servers in a clustered environment, apps can be hosted on different servers. For example, an app containing a smaller amount of aggregated data could be run on a server with less memory while an app with large amounts of detailed data could be configured to run on a larger server, all without the user having to know where the apps are physically hosted.
- Multi-tiered architecture. QlikView can be deployed such that one server runs in the background extracting and transforming large amounts of data while another server runs the user-facing app and does not have the added burden of handling back-end tasks. An additional benefit to IT from QlikView’s multi-tiered architecture is that the transactional data source only has to be accessed once. That data can then be reused in multiple QlikView apps without a fresh extract.
- Incremental load. Administrators can configure QlikView to load only data that is new or has changed since the last load, thus greatly reducing the bandwidth required from any data source.
- Document chaining. Separate QlikView apps, potentially running on different servers, can share selection states, thus a user may be running a small dashboard or summary app and seamlessly switch over to a large app containing detailed data.
Many QlikView customers follow this approach as it satisfies their requirements for access to Big Data while preserving high performance.
Implementing QlikView has cost less than 20% of the alternative solutions. The payback period was just a few months.
Mats-Olov Eriksson, The main architect of the analytics system