2017-05-29

How to Build HyperCube-Based Autonomous BI Dashboards

The field of data analytics and data science has supplied no shortage of data visualization and reporting technologies. There are many available commercial platforms as well as an endless number of free open technologies such as pre-built packages provided in R or Python.

The available technologies range from specialized reporting tools specifically structured for enterprise BI reporting to open-source free-form libraries that plug-in to standard programming languages and require the user to script the solution themselves.

Many of these technologies are often taught and heralded as the standard go-to solution for the design and delivery of data analytics and business intelligence reports. These technologies, however, are mostly insufficient and fall short of being a complete solution to the end-to-end business analytics challenges that many businesses face.

The insufficiencies arise from the fact that even though these technologies have a strong focus on creating charts and graphs - they exhibit significant limitations and shortcomings in the pre-steps required for the creation of actionable reports.

The pre-steps to create actionable reports are often the most difficult and time-consuming components of the BI process and include tasks such as:

  1. Data acquisition, curation, and transformation
  2. Preparation and cleansing of data
  3. Extraction of new derived/inferred data points for reporting
  4. Conversion of data into multi-dimensional analysis structures
  5. Aggregation and summarization of data in a multi-dimensional context

The current best-practice established reporting technologies demonstrate significant limitations such as:

  1. A lack of autonomy
  2. Lots of development overhead
  3. High complexity in the tasks required for acquisition and preparation of data
  4. Lack of reusability of developed solutions
  5. Reliance on code or script based languages for data preparation and transformation
  6. High overhead adapting existing solutions to meet new objectives as business processes and requirements change

An optimal next-generation data analysis and BI framework must exhibit the following characteristics to be satisfactory as a candidate for a complete solution to the current and emerging BI requirements that most businesses face:

  1. A high focus on autonomy
  2. Minimal development overhead while maximizing the flexibility, reusability, and impact of analysis output
  3. Ability to acquire data from any source, structure, or API without reliance on code
  4. Ability to handle all data preparation tasks and the computation/derivation of new reporting data points without relying on script
  5. Easy to understand, audit, and describe the analysis steps used to produce the results and reports
  6. Easy to distribute reports and views across an organization
  7. Facilitate the continuous automation of all report generation and distribution tasks
  8. Strong focus on sharing and collaboration
  9. Must be as powerful and significantly faster than code
  10. Provides the ability to drill-down and pivot across all critical business data n-dimensions deep
  11. Ability to scale to report on large volumes of data
  12. Minimal overhead adapting and modifying existing solutions as business requirements and processes change
  13. Provides AI as an intermediate mechanism for facilitating the generation of reports and views
  14. High focus on reproducibility and reusability of developed solutions

In this blog post, I present a more powerful approach to BI Dashboard development and reporting than the current best-practice established technologies.

The example we will focus on in this blog post will demonstrate how to use a hypercube to create an N-dimensional interactive drill-through BI dashboard from raw sales data.

In the video below I provide a worked example of how to develop an autonomous data analytics workflow which does the following:

  1. Loads raw data into our working data container
  2. Computes and derives new data points for reporting using generic expressions
  3. Builds a multi-dimensional HyperCube from the data
  4. Computes and summarizes various statistics across the HyperCube
  5. Consumes the HyperCube to create different n-dimensional drill down visualizations
  6. Consumes the HyperCube to construct different pivot tables and left-to-right hierarchical reports with multiple filterable axes
  7. Delivers the analysis results to the cloud

I then demonstrate how to combine the various multi-dimensional analysis result items into a dashboard to create drill-through n-dimensional views.

I show how to subscribe users to the dashboard to efficiently distribute and deliver our finished reports.

Finally, I show how to deploy the workflow to the Flow Autonomous Agent Framework to continuously execute our reporting tasks on a schedule. The agent framework allows us to automate the acquisition, preparation, and continuous feed of refreshed data to our dashboard.

The approach explored in this blog post is easier, faster, more scalable, and more powerful than the methods supplied by all current best-practice technologies for BI reporting.

All techniques demonstrated in this blog post can be done with the basic version of Flow. Flow Basic is free for personal use. If you do not have a Flow account you can register here.

The video below shows the worked example of the HyperCube BI dashboard solution outlined above. Check out the video here: