More DV Readings from 2013

Data Visualization readings - last 4 months of 2013.

(time to read is shrinking...)

0. The Once and Future Prototyping Tool of Choice
http://tableaufriction.blogspot.com/2013/07/the-once-and-future-prototyping-tool-of.html

1. Block by Block, Brooklyn’s Past and Present
http://bklynr.com/block-by-block-brooklyns-past-and-present/

2. Data Visualization and the Blind
http://www.perceptualedge.com/blog/?p=1756

3. WHY ABRAHAM LINCOLN LOVED INFOGRAPHICS
http://www.newyorker.com/online/blogs/elements/2013/10/why-abraham-lincoln-loved-infographics.html#

4. Old Charts
[youtube=https://www.youtube.com/watch?v=r2q8kzdxbac]

5. Back To Basics
http://www.quickintelligence.co.uk/back-to-basics/

6. In-Memory Data Grid Key to TIBCO's Strategy
http://www.datanami.com/datanami/2013-10-21/in-memory_data_grid_key_to_tibco_s_strategy.html

7. Submarine Cable Map
http://visual.ly/submarine-cable-map?view=true

8. Interview with Nate Silver:
http://blogs.hbr.org/2013/09/nate-silver-on-finding-a-mentor-teaching-yourself-statistics-and-not-settling-in-your-career/

9. Qlikview.Next will be available in 2014
http://apandre.wordpress.com/2013/09/25/qlikview-next/

10. Importance of color?
http://www.qualia.hr/why-is-color-so-important-in-data-visualization/#

11. Qlikview.Next has a gift for Tableau and Datawatch
http://apandre.wordpress.com/2013/10/24/qlik-next-has-gift/

12. (October 2013) Tableau posts 90% revenue gain and tops 1,000 staffers, files for $450 million secondary offering
http://www.geekwire.com/2013/tableau-software/#

13. The Science Of A Great Subway Map
http://www.fastcodesign.com/3020708/evidence/the-science-of-a-great-subway-map

14. SEO Data Visualization with Tableau
http://www.blastam.com/blog/index.php/2013/10/how-to-create-awesome-seo-data-visualization-with-tableau/

15. John Tukey “Badmandments”
http://www.kdnuggets.com/2013/11/john-tukey-badmandments-lessons-from-great-statistician.html#


Supplementary BADMANDMENTS:

• 91. NEVER plan any analysis before seeing the DATA.


• 92. DON'T consult with a statistician until after collecting your data.


• 94. LARGE enough samples always tell the truth

16. Thinking about proper uses of data visualization.
http://data-visualization-software.com/finally-some-clear-thinking-about-proper-uses-of-data-visualization/

17. Big BI is Stuck: Illustrated by SAP BusinessObjects Explorer
http://www.perceptualedge.com/blog/?p=727

18. IBM (trying to catch up?) bets on big data visualization
http://www.zdnet.com/ibm-bets-on-big-data-visualization-7000022741/

19. Site features draft designs and full views of the Treemap Art project (By Ben Shneiderman)
http://treemapart.wordpress.com/
http://www.cs.umd.edu/hcil/treemap-history/
http://www.cs.umd.edu/hcil/treemap/
http://treemapart.wordpress.com/full-views/
http://treemapart.wordpress.com/category/draft-designs/


20. A Guide to the Quality of Different Visualization Venues
http://eagereyes.org/blog/2013/a-guide-to-the-quality-of-different-visualization-venues

21. Short History of (Nothing) Data Science
http://www.forbes.com/sites/gilpress/2013/05/28/a-very-short-history-of-data-science/

22. Storytelling: Hans Rosling at Global Health - beyond 2015


23. DataWatch Quarterly Review: Rapid Growth Finally Materializing
http://seekingalpha.com/article/1872591-datawatch-quarterly-review-rapid-growth-finally-materializing

24. QlikView Extension – D3 Animated Scatter Chart
http://www.qlikblog.at/2574/qlikview-extension-animated-scatter-chart/



25. SlopeGraph for QlikView (D3SlopeGraph QlikView Extension)
http://www.qlikblog.at/3093/slopegraph-for-qlikview-d3slopegraph-qlikview-extension/

26. Recipe for a Pareto Analysis
http://community.qlikview.com/blogs/qlikviewdesignblog/2013/12/09/pareto-analysis

27. Color has meaning
http://www.juiceanalytics.com/design-principles/color-has-meaning/#


28. TIBCO's Return To License Growth Frustratingly Inconsistent
http://seekingalpha.com/article/1909571-tibcos-return-to-license-growth-frustratingly-inconsistent

29. Automated Semantics and BI
http://www.forbes.com/sites/danwoods/2013/12/30/why-automated-semantics-will-solve-the-bi-dashboard-crisis/

30. What is wrong with definition of Data Science?
http://www.kdnuggets.com/2013/12/what-is-wrong-with-definition-data-science.html


31. Scientific data became so complex, we have to Invent new Math to deal with it
http://www.wired.com/wiredscience/2013/10/topology-data-sets/

32. Samples

Happy New 2014!

My Best Wishes for 2014 to all visitors of this Blog!

2013 was very successful year for Data Visualization (DV) community, Data Visualization vendors and for this Data Visualization Blog (number of visitors per grew from average 16000 to 25000+ per month).

From certain point of view 2013 was the year of Tableau - it went public, Tableau has now the largest Market Capitalization among DV Vendors (more than $4B as of Today) and its strategy (Data to the People!) became the most popular among DV users and it had (again) largest YoY revenue growth (almost 75% !) among DV Vendors. Tableau already employed more than 1100 people and still has 169+ job openings as of today. I wish Tableau to stay the Leader of our community and to keep their YoY above 50% - this will not be easy.

Qliktech is the largest DV Vendor and it will exceed in 2014 the half-billion dollars benchmark in revenue (probably closer to $600M by end of 2014) and will employ almost 2000 employees. Qlikview is one of the best DV product on market. I wish in 2014 Qlikview will create Cloud Services, similar to Tableau Online and Tableau Public and I wish Qlikview.Next will keep Qlikview Desktop Professional (in addition to HTML5 client).

I wish TIBCO will stop trying to improve BI or make it better - you cannot reanimate a dead horse; instead I wish Spotfire will embrace the approach "Data to the People" and act accordingly. For Spotfire my biggest wish is that TIBCO will spin it off the same way EMC did with VMWare. And yes, I wish Spofire Cloud Personal will be free and enabled to read at least local flat files and local DBs like Access.

2014 (or may be 2015?) can witness new, 4th DV player coming to competition: Datawatch bought recently Panopticon and if it will complete integration of all products correctly and add features which other DV vendors above already have (like Cloud Services), it can be very competitive player. I wish them luck!

Microsoft released in 2013 a lot of advanced and useful DV-related functionality and I wish (I recycling this wish for many years now) that Microsoft finally will package the most its Data Visualization Functionality in one DV product and add it to Office 20XX (like they did with Visio) and Office 365 instead of bunch of plug-ins to Excel and SharePoint.

It is a mystery for me why Panorama, Visokio and Advizor Solutions still relatively small players, despite all 3 of them having an excellent DV features and products. Based on 2013 IPO experience with Tableau may be the best way for them to go public and get new blood? I wish to them to learn from Tableau and Qlikview success and try this path in 2014-15...

For Microstrategy my wish is very simple - they are only traditional BI player who realised that BI is dead and they started in 2013 (actually before then 2013) a transition into DV market and I wish them all success they can handle!

I also think that a few thousands of Tableau, Qlikview and Spotfire customers (say 5% of customer base) will need (in 2014 and beyond) more deep Analytics and they will try to complement their Data Visualizations with Advanced Visualization technologies they can get from vendors like http://www.avs.com/

My best wishes to everyone! Happy New Year!

Data Visualization as a Service

With releases of Spotfire Silver, Tableau Online and attempts of a few Qlikview Partners (but not Qliktech itself yet) to the Cloud and providing their Data Visualization Platforms and Software as a Service, the Attributes, Parameters and Concerns of such VaaS or DVaaS ( Visualization as a Service) are important to understand. Below is attempt to review those "Cloud" details at least on a high level (with natural limitation of space and time applied to review).

But before that let's underscore that Clouds are not in the skies but rather in huge weird buildings with special Physical and Infrastructure security likes this Data Center in Georgia:



You can see some real old fashion clouds above the building but they are not what we are talking about. Inside Data Center you can see a lot of Racks, each with 20+ servers which are, together with all secure network and application infrastructure contain these modern "Clouds":



Attributes and Parameters of mature SaaS (and VaaS as well) include:

• Multitenant and Scalable Architecture (this topic is too big and needs own blogpost or article). You can review Tableau's whitepaper about Tableau Server scalability here: http://www.tableausoftware.com/learn/whitepapers/tableau-server-scalability-explained


• SLA - service level agreement with up-time, performance, security-related and disaster recovery metrics and certifications like SSAE16.


• UI and Management tools for User Privileges, Credentials and Policies.


• System-wide Security: SLA-enforced and monitored Physical, Network, Application, OS and Data Security.


• Protection or/and Encryption of all or at least sensitive (like SSN) fields/columns.


• Application Performance: Transaction processing speed, Network Latency, Transaction Volume, Webpage delivery times, Query response times


• 24/7 high availability: Facilities with reliable and backup power and cooling, Certified Network Infrastructure, N+1 Redundancy, 99.9% (or 99.99% or whatever your SLA with clients promised) up-time


• Detailed historical availability, performance and planned maintenance data with Monitoring and Operational Dashboards, Alerts and Root Cause Analysis


• 
  

  Disaster recovery plan with multiple backup copies of customers’ data in near real time at the disk level, a 

  
  

  multilevel backup strategy that includes disk-to-disk-to-tape data backup where tape backups serve as a secondary level of backup, not as their primary disaster recovery data source.

  
  


• Fail-over that cascades from server to server and from data center to data center in the event of a regional disaster, such as a hurricane or flood.

While Security, Privacy, Latency and Hidden Cost usually are biggest concerns when considering SaaS/VaaS, other Cloud Concerns surveyed and visualized below. Recent survey and diagram are published by Charlie Burns this month:



Other survey and diagram are published by Shane Schick in October 2011 and in February of 2013 by KPMG. Here are concerns, captured by KPMG survey:



As you see above, Rack in Data Center can contain multiple Servers and other devices (like Routers and Switches, often redundant (at least 2 or sometimes N+1). Recently I designed the Hosting Data VaaS Center for Data Visualization and Business Intelligence Cloud Services and here are simplified version of it just for one Rack as a Sample.

You can see redundant network, redundant Firewalls, redundant Switches for DMZ (so called "Demilitarized Zone" where users from outside of firewall can access servers like WEB or FTP), redundant main Switches and Redundant Load Balancers, Redundant Tableau Servers, Redundant Teradata Servers, Redundant Hadoop Servers, Redundant NAS servers etc. (not all devices shown on Diagram of this Rack):

5 Elements of Visualization: Story is only 4th

2400 years ago the concept of Data Visualization was less known, but even than Plato said "Those who tell stories rule society".

I witnessed multiple times how storytelling triggered the Venture Capitalists (VCs) to invest. Usually my CEO (biggest BS master on our team) will start with a "60-seconds-long" short Story (VCs called them "Elevator Pitch") and then (if interested) VCs will do a long Due Diligence Research of Data (and Specs, Docs and Code) presented by our team and after that they will spend comparable time analyzing Data Visualizations (Charts, Diagrams, Slides etc.) of our Data, trying to prove or disprove the original Story.

Some of conclusions from all these startup storytelling activity were:

• 
  

  Data: without Data nothing can be proved or disproved (Action needs Data!)

  
  


• 
  

  View: best way to analyze Data and trust it is to Visualize it (Seeing is Believing!)

  
  


• 
  

  Discovery of Patterns: visually discoverable trends, outliers, clusters etc. which form the basis of the Story and follow-up actions

  
  


• 
  

  Story: the Story (based on that Data) is the Trigger for the Actions (Story shows the Value!),

  
  


• 
  

  Action(s): start with drilldown to a needle in haystack, embed Data Visualization into business, it is not an Eye Candy but a practical way to improve the business

  
  


• 
  

  Data Visualization has 5 parts: Data (main), View (enabler), Discovery (visually discoverable Patterns), Story (trigger for Actions) and finally the 5^th Element – Action!

  
  


• 
  

  Life is not fair: Storytellers were there people who benefited the most in the end... (no Story no Glory!).

  
  

And yes, Plato was correct - at least partially and for his time. Diagram above uses analogy with 5 Classical Greek Elements. Plato wrote about four classical elements (earth, air, water, and fire) almost 2400 years ago (citing even more ancient philosopher) and his student Aristotle added a fifth element, aithêr (aether in Latin, "ether" in English) - both men are in the center of 1st picture above.

Back to our time: the Storytelling is a hot topic; enthusiasts saying that "Data is easy, good storytelling is the challenge" http://www.resource-media.org/data-is-easy/#.URVT-aVi4aE or even that "Data Science is a Storytelling": http://blogs.hbr.org/cs/2013/03/a_data_scientists_real_job_sto.html . Nothing can be further from the truth: my observation is that most Storytellers (with a few known exceptions like Hans Rosling or Tableau founder Pat Hanrahan) ARE NOT GOOD at visualizing but they still wish to participate in our hot Data Visualization party. All I can say is “Welcome to the party!”

It may be a challenge for me and you but not for people who had a conference about storytelling: this winter, 2/27/13 in Nashville, KY: http://www.tapestryconference.com/ :

Some more reasonable  people referring to storytelling as a data journalism and narrative visualization: http://www.icharts.net/blogs/2013/pioneering-data-journalism-simon-rogers-storytelling-numbers

Tableau founder Pat Hanrahan recently talked about "Showing is Not Explaining". In parallel, Tableau is planning (after version 8.0) to add features that support storytelling by constructing visual narratives and effective communication of ideas, see it here:

Collection of resources on storytelling topic can be found here: http://www.juiceanalytics.com/writing/the-ultimate-collection-of-data-storytelling-resources/

You may also to check what Stephen Few thinks about it here: http://www.perceptualedge.com/blog/?p=1632

Storytelling as an important part (using Greek Analogy - 4^th Classical Element (Air) after Data (Earth), View (Water) and Discovery (Fire) and before Action (Aether) ) of Data Visualization has a practical effect on Visualization itself, for example:

• 
  

  if Data View is not needed for Story or for further Actions, then it can be hidden or removed,

  
  


• 
  

  if number of Data Views in Dashboard is affecting impact of (preferably short Data Story), then number of Views should be reduced (usually to 2 or 3 per dashboard),

  
  


• 
  

  If number of DataPoints is too large per View and affecting the triggering power of the story, then it can be reduced too (in conversations with Tableau they even recommending 5000 Datapoints per View as a threshold between Local and Server-based rendering).

  
  

 

Good Practices for Data Visualization

Below you can find samples of Guidelines and Good Practices for Data Visualization (mostly with Tableau), which I used recently.

Some of this samples are Tableau-specific, but others (may be with modifications) can be reused for other Data Visualization Platform and tools. I will appreciate feedback, comments and suggestions.

Naming Convention for Tableau Objects

• 
  

  Use CamelCase Identifiers: Capitalize the 1st letter of each concatenated word

  
  


• 
  

  Use Suffix for Identifiers with preceding underscore to indicate the type (example: _tw for workbooks).

  
  

Workbook Sizing Guidelines

• 
  

  Use Less than 5 Charts per Dashboard, Minimize the number of Visible TABs/Worksheets

  
  


• 
  

  Move Calculations and Functions from Workbook to the Data.

  
  


• 
  

  Use less than 5000 Data-points per Chart/Dashboard to enable Client-side rendering.

  
  


• 
  

  To enable Shared Sessions, don't use filters and interactivity if it is not needed.

  
  

Guidelines for Colors, Fonts, Sizes

• 
  

  To express desirable/undesirable points, use green for good, red for bad, yellow for warning.

  
  


• 
  

  When you are not describing "Good-Bad situation" (thanks to feedback of visitor under alias "SF") , try to use pastel, neutral and blind colors, e.g. similar to "Color Blind 10" Palette from Tableau.

  
  


• 
  

  Use "web-safe" fonts, to approximate what users can see from Tableau Server.

  
  


• 
  

  Use either auto-resize or standard (target smaller screen) sizes for Dashboards

  
  

Data and Data Connections used with Tableau

• 
  

  Try to avoid pulling more than 15000 rows for Live Data Connections.

  
  


• 
  

  For Data Extract-based connections 10M rows is the recommended maximum.

  
  


• 
  

  For widely distributed Workbooks use of Application IDs instead of Personal Credentials.

  
  


• 
  

  Job failure due expired credentials leads to suspension from Schedule, so try to keep embedded credentials up to date

  
  

Tableau Data Extracts (TDE)

• 
  

  If Refresh of TDE takes more than 2 hours, consider to redesign it.

  
  


• 
  

  Reuse and share TDEs and Data Sources as much as possible.

  
  


• 
  

  Use of Incremental Data Refresh instead of Full Refresh when possible.

  
  


• 
  

  Designate Unique ID for each row when Incremental Data Refresh is used.

  
  


• 
  

  Try to use free Tableau Data Exract API instead of licensed Tableau Server to create Data Extracts

  
  

Scheduling of Background Tasks with Tableau

• 
  

  Serial Schedules is recommended; avoid the usage of hourly Schedules.

  
  


• 
  

  Avoid scheduling during peak hours (8am-6pm), consider weekly instead of daily schedules.

  
  


• 
  

  Optimize Schedule Size, group tasks related to the same project into one Schedule, if total tasks execution exceeds 8 hours, split Schedule on a few with similar Name but preferably with different starting time.

  
  


• 
  

  Maximize the usage of Monthly and Weekly Schedules (as oppose to Daily Schedules) and usage of weekends and nights.

  
  

Guidelines for using Charts

• 
  

  Use Bars to compare across categories, use Colors with Stacked or Side-by-Side Bars for deeper Analysis

  
  


• 
  

  Use Line for Viewing Trends over time, consider Area Charts for Multi-lines

  
  


• 
  

  Minimize the usage of Pie Charts; when appropriate - use it for showing proportions. It is recommended to limit pie wedges to six.

  
  


• 
  

  Use Map to show geocoded data, consider use maps as interactive filters

  
  


• 
  

  Use Scatter to analyze outliers, clusters and construct regressions

  
  

You can find more about Guidelines and Good Practices for Data Visualization here: http://www.tableausoftware.com/public/community/best-practices

to DV or to D3 - that is the question

The most popular (among business users) approach to visualization is to use a Data Visualization (DV) tool like Tableau (or Qlikview or Spotfire), where a lot of features already implemented for you. Recent prove of this amazing popularity is that at least 100 million people (as of February 2013),  used Tableau Public as their Data Visualization tool of choice, see

http://www.tableausoftware.com/about/blog/2013/2/crossing-100-million-milestone-21304

However, to make your documents and stories (and not just your data visualization applications) driven by your data, you may need the other approach - to code visualization of your data into your story and visualization libraries like  popular D3 toolkit can help you. D3 stands for "Data-Driven Documents". The Author of D3 Mr. Mike Bostock designs interactive graphics for New York Times - one of latest samples is here:

http://www.nytimes.com/interactive/2013/02/20/movies/among-the-oscar-contenders-a-host-of-connections.html

and NYT allows him to do a lot of Open Source work which he demonstartes at his website here:

https://github.com/mbostock/d3/wiki/Gallery .

Mike was a "visualization scientist" and a computer science PhD student at #Stanford University and member of famous group of people, now called "Stanford Visualization Group":

http://vis.stanford.edu/people/

This Visualization Group was a birthplace of Tableau's prototype - sometimes they called it  "a Visual Interface" for exploring data and other name for it is Polaris:

http://www.graphics.stanford.edu/projects/polaris/

and we know that creators of Polaris started Tableau Software. One of other Group's popular "products" was a graphical toolkit (mostly in JavaScript, as oppose to Polaris, written in C++) for Visualization, called ProtoVis:

http://mbostock.github.com/protovis/

- and Mike Bostock was one of ProtoViz's main co-authors. Less then 2 years ago Visualization Group suddenly stopped developing ProtoViz and recommended to everybody to switch to D3 library

https://github.com/mbostock,

authored by Mike. This library is Open Source (only 100KB in ZIP format) and can be downloaded from here:

http://d3js.org/d3.v3.zip

In order to use D3, you need to be comfortable with HTML, CSS, SVG, Javascript programming, DOM (and other Web Standards); understanding of jQuery paradigm will be useful too. Basically if you want to be at least partially as good as Mike Bostock, you need to have a mindset of a programmer (I guess in addition to business user mindset), like this D3 expert:

http://www.jasondavies.com/

Most of successful early D3 adopters combining even 3+ mindsets: programmer, business analyst, data artist and even sometimes data storyteller. For your programmer's mindset you may be interested to know that D3 has a large set of Plugins, see:

https://github.com/d3/d3-plugins

and rich #API, see https://github.com/mbostock/d3/wiki/API-Reference

You can find hundreds of D3 demos, samples, examples, tools, products and even a few companies using D3 here: https://github.com/mbostock/d3/wiki/Gallery

Advizor Visual Discovery, Part 2

This is the Part 2 of the guest blog post: the Review of Visual Discovery products from Advizor Solutions, Inc., written by my guest blogger Mr. Srini Bezwada (his profile is here: http://www.linkedin.com/profile/view?id=15840828 ), who is the Director of Smart Analytics, a Sydney based professional BI consulting firm that specializes in Data Visualization solutions. Opinions below belong to Mr. Srini Bezwada.

ADVIZOR Technology

ADVIZOR's Visual Discovery™ software is built upon strong data visualization tools technology spun out of a distinguished research heritage at Bell Labs that spans nearly two decades and produced over 20 patents. Formed in 2003, ADVIZOR has succeeded in combining its world-leading data visualization and in-memory-data-management expertise with extensive usability knowledge and cutting-edge predictive analytics to produce an easy to use, point and click product suite for business analysis.

ADVIZOR readily adapts to business needs without programming and without implementing a new BI platform, leverages existing databases and warehouses, and does not force customers to build a difficult, time consuming, and resource intensive custom application. Time to deployment is fast, and value is high.

With ADVIZOR data is loaded into a "Data Pool" in main memory on a desktop or laptop computer, or server. This enables sub-second response time on any query against any attribute in any table, and instantaneously update all visualizations. Multiple tables of data are easily imported from a variety of sources.

With ADVIZOR, there is no need to pre-configure data. ADVIZOR accesses data "as is" from various data sources, and links and joins the necessary tables within the software application itself. In addition, ADVIZOR includes an Expression Builder that can perform a variety of numeric, string, and logical calculations as well as parse dates and roll-up tables - all in-memory. In essence, ADVIZOR acts like a data warehouse, without the complexity, time, or expense required to implement a data warehouse! If a data warehouse already exists, ADVIZOR will provide the front-end interface to leverage the investment and turn data into insight.
Data in the memory pool can be refreshed from the core databases / data sources "on demand", or at specific time intervals, or by an event trigger. In most production deployments data is refreshed daily from the source systems.

Data Visualization

ADVIZOR's Visual Discovery™ is a full visual query and analysis system that combines the excitement of presentation graphics - used to see patterns and trends and identify anomalies in order to understand "what" is happening - with the ability to probe, drill-down, filter, and manipulate the displayed data in order to answer the "why" questions. Conventional BI approaches (pre-dating the era of interactive Data Visualization) to making sense of data have involved manipulating text displays such as cross tabs, running complex statistical packages, and assembling the results into reports.

ADVIZOR's Visual Discovery™ making the text and graphics interactive. Not only can the user gain insight from the visual representation of the data, but now additional insight can be obtained by interacting with the data in any of ADVIZOR's fifteen (15) interactive charts, using color, selection, filtering, focus, viewpoint (panning, zooming), labeling, highlighting, drill-down, re-ordering, and aggregation.


Visual Discovery empowers the user to leverage his or her own knowledge and intuition to search for patterns, identify outliers, pose questions and find answers, all at the click of a mouse.

Flight Recorder - Track, Save, Replay your Analysis Steps

The Flight Recorder tracks each step in a selection and analysis process. It provides a record of those steps, and be used to repeat previous actions. This is critical for providing context to what and end-user has done and where they are in their data. Flight records also allow setting bookmarks, and can be saved and shared with other ADVIZOR users.
The Flight Recorder is unique to ADVIZOR. It provides:
• A record of what a user has done. Actions taken and selections from charts are listed. Small images of charts that have been used for selection show the selections that were made.
• A place to collect observations by adding notes and capturing images of other charts that illustrate observations.
• A tool that can repeat previous actions, in the same session on the same data or in a later session with updated data.
• The ability to save and name bookmarks, and share them with other users.

Predictive Analytics Capability

The ADVIZOR Analyst/X is a predictive analytic solution based on a robust multivariate regression algorithm developed by KXEN - a leading-edge advanced data mining tool that models data easily and rapidly while maintaining relevant and readily interpretable results.
Visualization empowers the analyst to discover patterns and anomalies in data by noticing unexpected relationships or by actively searching. Predictive analytics (sometimes called "data mining") provides a powerful adjunct to this: algorithms are used to find relationships in data, and these relationships can be used with new data to "score" or "predict" results.



Predictive analytics software from ADVIZOR don't require enterprises to purchase platforms. And, since all the data is in-memory, the Business Analyst can quickly and easily condition data and flag fields across multiple tables without having to go back to IT or a DBA to prep database tables. The interface is entirely point-and-click, there are no scripts to write. The biggest benefit from the multi-dimensional visual solution is how quickly it delivers analysis, solving critical business questions, facilitating intelligence-driven decision making, providing instant answers to "what if?" questions.

Advantages over Competitors:

• The only product in the market offering a combination of Predictive Analytics + Data Visualisation + In memory data management within one Application.
• The cost of entry is lower than the market leading data visualization vendors for desktop and server deployments.
• Advanced Visualizations like Parabox, Network Constellation in addition to normal bar charts, scatter plots, line charts, Pie charts…
• Integration with leading CRM vendors like Salesforce.com, Blackbaud, Ellucian, Information Builder
• Ability to provide sub-second response time on query against any attribute in any table, and instantaneously update all visualizations.
• Flight recorder that lets you track, replay, and save your analysis steps for reuse by yourself or others.

Update on 5/1/13 (by Andrei): Avizor 6.0 is available now with substantial enhancements: http://www.advizorsolutions.com/Bnews/tabid/56/EntryId/215/ADVIZOR-60-Now-Available-Data-Discovery-and-Analysis-Software-Keeps-Getting-Better-and-Better.aspx

Advizor Visual Discovery, Part 1

If you visited my blog before, you know that my classification of Data Visualization and BI vendors are different from researchers like Gartner. In addition to 3 DV Leaders - Qlikview, Tableau, Spotfire - I rarely have time to talk about other "me too" vendors.

However, sometimes products like Omniscope, Microstrategy's Visual Insight, Microsoft BI Stack (Power View, PowerPivot, Excel 2013, SQL Server 2012, SSAS etc.), Advizor, SpreadshetWEB etc. deserve attention too. However, it takes so much time, so I am trying to find guest bloggers to cover topics like that. 7 months ago I invited volunteers to do some guest blogging about Advizor Visual Discovery Products:

http://apandre.wordpress.com/2012/06/22/advizor-analyst-vs-tableau-or-qlikview/

So far nobody in  USA or Europe committed to do so, but recently Mr. Srini Bezwada, Certified Tableau Consultant and Advizor-trained expert from Australia contacted me and submitted the article about it.  He also provided me with info about how Advizor can be compared with Tableau, so I will do it briefly, using his data and opinions. Mr. Bezwada can be reached at

sbezwada@smartanalytics.com.au , where he is a director at

http://www.smartanalytics.com.au/

Below is quick comparison of Advizor with Tableau. Opinions below belong to Mr. Srini Bezwada. Next blog post will be a continuation of this article about Advizor Solutions Products, see also Advizor's website here:

http://www.advizorsolutions.com/products/

Criteria	Tableau	ADVIZOR	Comment
Time to implement	Very Fast	Fast, ADVIZOR can be implemented within Days	Tableau Leads
Scalability	Very Good	Very Good	Tableau: virtual RAM
Desktop License	$1,999	$ 1,999	$3,999 for AnalystX with Predictive modeling
Server License/user	$1K, min 10 users, 299 K for Enterprise	Deployment license for up to 10 named users $8 K	ADVIZOR is a lot cheaper for Enterprise Deployment $75 K for 500 Users
Support fees / year	20%	20%	1st year included
SaaS Platform	Core or Digital	Offers Managed Hosting	ADVIZOR Leads
Overall Cost	Above Average	Competitive	ADVIZOR Costs Less
Enterprise Ready	Good for SMB	Cheaper cost model for SMB	Tableau is expensive for Enterprise Deployment
Long-term viability	Fastest growth	Private company since 2003.	Tableau is going IPO in 2013
Mindshare	Tableau Public	Growing Fast	Tableau stands out
Big Data Support	Good	Good	Tableau is 32-bit
Partner Network	Good	Limited Partnerships	Tableau Leads
Data Interactivity	Excellent	Excellent
Visual Drilldown	Very Good	Very Good
Offline Viewer	Free Reader	None	Tableau stands out
Analyst's Desktop	Tableau Professional	Advizor has Predictive Modeling	ADVIZOR is a Value for Money
Dashboard Support	Excellent	Very Good	Tableau Leads
Web Client	Very Good	Good	Tableau Leads
64-bit Desktop	None	Very Good	Tableau still a 32-bit app
Mobile Clients	Very Good	Very Good
Visual Controls	Very Good	Very Good
Data Integration	Excellent	Very Good	Tableau Leads
Development	Tableau Pro	ADVIZOR Analyst
64-bit in-RAM DB	Good	Excellent	Advizor Leads
Mapping support	Excellent	Average	Tableau stands out
Modeling, Analytics	Below Average	Advanced Predictive Modelling	ADVIZOR stands out
Predictive Modeling	None	Advanced Predictive Modeling Capability with Built in KXEN algorithms	ADVIZOR stands out
Flight Recorder	None	Flight recorder lets you track, replay, save your analysis steps for reuse by yourself or others.	ADVIZOR stands out
Visualization	22 Chart types	All common charts like  bar charts, scatter plots, line charts, Pie charts are supported	Advizor has Advanced Visualizations like Parabox, Network Constellation
Third party integration	Many Data Connectors, see Tableau's drivers page	ADVIZOR integrates well with CRM software: Salesforce.com, Ellucian, Blackbaud and others.	ADVIZOR leads in CRM area
Training	Free Online and paid Classroom	Free Online and paid via company trainers & Partners	Tableau Leads

Power View in Excel 2013

I doubt that Microsoft is paying attention to my blog, but recently they declared that Power View now has 2 versions: one  for SharePoint (thanks, but no thanks) and one for Excel 2013. In other words, Microsoft decided to have own Desktop Visualization tool. In combination with PowerPivot and SQL Server 2012 it can be attractive for some Microsoft-oriented users but I doubt it can compete with Data Visualization Leaders - too late.

Most interesting is the note about Power View 2013 on Microsoft site: "Power View reports in SharePoint are RDLX files. In Excel, Power View sheets are part of an Excel XLSX workbook. You can’t open a Power View RDLX file in Excel, and vice versa. You also can’t copy charts or other visualizations from the RDLX file into the Excel workbook."

But most amazing is that Microsoft decided to use the dead Silverlight for Powerview: "Both versions of Power View need Silverlight installed on the machine." And we know that Microsoft switched to HTML5 from Silverlight and no new development planned for Silverlight! Good luck with that...

And yes, you can add now maps (Bing of course), see it here:

Advizor Analyst vs. Tableau or Qlikview...

I feel guilty for many months now: I literally do not have time for project I wish to do for a while: to compare Advizor Analyst and other Visual Discovery products from Advizor Solutions, Inc. with leading Data Visualization products like Tableau or Qlikview. I am asking visitors of my blog to volunteer and be a guest blogger here; the only pre-condition here is: a guest blogger must be the Expert in Advizor Solutions products and equally so in on of these 3: Tableau, Qlikview or Spotfire.

ADVIZOR's Visual Discovery™ software is built upon strong data visualization technology spun out of a research heritage at Bell Labs that spans nearly two decades and produced over 20 patents. Formed in 2003, ADVIZOR has succeeded in combining its world-leading data visualization and in-memory-data-management expertise with predictive analytics to produce an easy to use, point and click product suite for business analysis.

Advizor has many Samples, Demos and Videos on its site: http://www.advizorsolutions.com/gallery/ and some web Demos, like this one

http://webnav.advizorsolutions.net/adv/Projects/demo/MutualFunds.aspx but you will need the Silverlight plugin for your web browser installed.

If you think that Advizor can compete with Data Visualization leaders and you have interesting comparison of it, please send it to me as MS-Word article and I will publish it here as a guest blog post. Thank you in advance...

Data Visualization tool is a Presentation Tool

(this is a repost from my other Data Visualization blog: http://tableau7.wordpress.com/2012/05/31/tableau-as-container/ )

Often I used small Tableau (or Spotfire or Qlikview) workbooks instead of PowerPoint, which are proving at least 2 concepts:

• 
  

  Good Data Visualization tool can be used as the Web or Desktop Container for Multiple Data Visualizations (it can be used to build a hierarchical Container Structures with more then 3 levels; currently 3: Container-Workbooks-Views)

  
  


• 
  

  It can be used as the replacement for PowerPoint; in example below I embedded into this Container 2 Tableau Workbooks, one Google-based Data Visualization, 3 image-based Slides and Textual Slide: http://public.tableausoftware.com/views/TableauInsteadOfPowerPoint/1-Introduction

  
  


• 
  

  Tableau (or Spotfire or Qlikview) is better then PowerPoint for Presentations and Slides

  
  


• 
  

  Tableau (or Spotfire or Qlikview) is the Desktop and the Web Container for Web Pages, Slides, Images, Texts

  
  


• 
  

  Good Visualization Tool can be a Container for other Data Visualizations

  
  


• 
  

  Sample Tableau Presentation above contains the Introductory Textual Slide

  
  


• 
  

  Sample Tableau Presentation above  contains a few Tableau Visualization:This Tableau Presentation contains a Web Page with the Google-based Motion Chart Demo

  
  
  
  
  1. 
     

     The Drill-down Demo

     
     
  
  
  2. 
     

     The Motion Chart Demo ( 6 dimensions: X,Y, Shape, Color, Size, Motion in Time)

     
     
  
  
  
  


• 
  

  This Tableau Presentation contains a few Image-based Slides:

  
  
  
  
  1. 
     

     The Quick Description of Origins and Evolution of Software and Tools used for Data Visualizations during last 30+ years

     
     
  
  
  2. 
     

     The Description of Multi-level Projection from Multidimensional Data Cloud to Datasets, Multidimensional Cubes and to Chart

     
     
  
  
  3. 
     

     The Description of 6 stages of Software Development Life Cycle for Data Visualizations

     
     
  
  
  
  

Data Science and Data Visualization

I am planning for a while to respond on Donald Farmer's comments on my recent blog post about Power View.

But instead I found myself thinking about relationship between Data Science and Data Visualization and yesterday Forbes (thanks to Dan Woods) published Donald's "explanation" about "What is a Data Scientist" and it forced me to write about it here before anything else.

Data Science was under radar until 2001, when various people started to use this term as an unifying umbrella for the set of predictive analytics algorithme, statistical methods, data mining technologies etc. like anomaly and trend detection, dependency and correlation modeling, clustering, classification, regression and summarization, pattern discovery etc.

Tableau vs. Qlikview

Some people pushing me to answer on recent Donald Farmer's comments on my previous post, but I need more time to think about it.

Meanwhile today Ted Cuzzillo published an interesting comparison of Qlikview vs. Tableau here:

http://datadoodle.com/2012/04/24/tableau-qlikview/

named "The future of BI in two words" which made me feel warm and fuzzy about both products and unclear about what Ted's judgement is?

Fortunately I had a more "digitized" comparison of these 2 Data Visualization Leaders, which I did a while ago for a different reason. So I modified it a little to bring it up-to-date and you can see it for yourself below. Funny thing is that even I used 30+ criterias to measure and compare those two brilliant products, final score is almost identical for both of them, so it is still warm and fuzzy.

Basically conclusion is simple: each product is better for certain customers and for certain projects, there is no universal answer (yet?):

[googleapps domain="docs" dir="spreadsheet/pub" query="hl=en_US&hl=en_US&key=0AuP4OpeAlZ3PdHJ1eXZyWFJjUVoxbUl0cTlCVDlXU2c&output=html&widget=true" width="510" height="1060" /]

Palettes and Colors

(this is a repost from my Tableau blog: http://tableau7.wordpress.com/2012/04/02/palettes-and-colors/ )

I was always intrigued with colors and their usage, since my mom told me that may be ( just may be, there is no direct prove of it anyway) Ancient Greeks did not know what the BLUE color is - that puzzled me.

Later in my live, I realized that Colors and Palettes are playing the huge role in Data Visualization (DV) and it eventually led me to attempt to understand of how it can be used and pre-configured in advanced DV tools to make Data more Visible and to express the Data Patterns better. For this post I used Tableau to produce some palettes, but similar technique can be found in Qlikview, Spotfire etc.

Tableau published the good article of how to create customized palettes here: http://kb.tableausoftware.com/articles/knowledgebase/creating-custom-color-palettes and I followed it below. As this article recommended, I modified default Preferences.tps file; see it below with images of respective Palettes embedded.

For the first, regular Red-Yellow-Green-Blue Palette with known colors with well-established names, I created even a Visualization in order to compare their Red-Green-Blue components and I even tried to placed respective Bubbles on 2-dimensional surface, even originally it is clearly a 3 dimensional Dataset (click on image to see it in full size):

For the 2nd Red-Yellow-Green-NoBlue Ordered Sequential Palette, I tried to implement the extended "Set of Traffic Lights without any trace of BLUE Color" (so Homer and Socrates will understand it the same way as we are) while trying to use only web-safe colors. Please keep in mind, that Tableau does not have a simple way to have more than 20 colors in one Palette, like Spotfire does.

Other 5 Palettes below are useful too as ordered-diverging almost "mono-chromatic" (except Red-Green Diverging, since it can be used in Scorecards when Red is bad and Green is good). So see below Preferences.tps file with my 7 custom palettes.

<?xml version='1.0'?> <workbook> <preferences>
<color-palette name="RegularRedYellowGreenBlue" type="regular">
<color>#FF0000</color> <color>#800000</color> <color>#B22222</color>
<color>#E25822</color> <color>#FFA07A</color> <color>#FFFF00</color>
<color>#FF7E00</color> <color>#FFA500</color> <color>#FFD700</color>
<color>#F0e68c</color> <color>#00FF00</color> <color>#008000</color>
<color>#00A877</color> <color>#99cc33</color> <color>#009933</color>
<color>#0000FF</color> <color>#00FFFF</color> <color>#008080</color>
<color>#FF00FF</color> <color>#800080</color>

</color-palette>

<color-palette name="RedYellowGreenNoBlueOrdered" type="ordered-sequential" >
<color>#ff0000</color> <color>#cc6600</color> <color>#cccc00</color>
<color>#ffff00</color> <color>#99cc00</color> <color>#009900</color>



</color-palette>

<color-palette name="RedToGreen" type="ordered-diverging" >
<color>#ff0000</color> <color>#009900</color> </color-palette>

<color-palette name="RedToWhite" type="ordered-diverging" >
<color>#ff0000</color> <color>#ffffff</color></color-palette>

<color-palette name="YellowToWhite" type="ordered-diverging" >
<color>#ffff00</color> <color>#ffffff</color></color-palette>

<color-palette name="GreenToWhite" type="ordered-diverging" >
<color>#00ff00</color> <color>#ffffff</color></color-palette>

<color-palette name="BlueToWhite" type="ordered-diverging" >
<color>#0000ff</color> <color>#ffffff</color> </color-palette>
</preferences> </workbook>

In case if you wish to use the colors you like, this site is very useful to explore the properties of different colors: http://www.perbang.dk/rgb/

Free Tableau Reader enables Server-less Visualization!

(this is a repost from http://tableau7.wordpress.com/2012/03/31/tableau-reader/ )

Tableau made a couple of brilliant decisions to completely outsmart its competitors and gained extreme popularity, while convincing millions of potential, future and current customers to invest own time to learn Tableau. 1st reason of course is Tableau Public (we discuss it in separate blog post) and other is a Free Tableau Reader, which provides full desktop user experience and interactive Data Visualization without any Tableau Server (and any other server) involved and with better performance and UI then Server-based Visualizations.

While designing Data Visualizations is done with Tableau Desktop, most users got their Data Visualizations served by Tableau Server to their Web Browser. However in the large and small organizations that usage pattern is not always the best fit. Below I am discussing a few possible use cases, where the usage of Free Tableau Reader can be appropriate, see it here: http://www.tableausoftware.com/products/reader .

1. Tableau Application Server serves Visualizations well, but not as well as Tableau Reader, because Tableau Reader delivers a truly desktop User Experience and UI. Most known example of it is a Motion Chart: you can see automatic motion with Tableau Reader but Web Browser will force user to manually emulate motion. In cases like that user advised to download workbook, copy .TWBX file to his/her workstation and open it with Tableau Reader.

Here is an example of the Motion Chart, done in Tableau, similar to famous Hans Rosling's presentation of Gapminder's Motion Chart (an you need the free Tableau Reader or license to Tableau Desktop to see the automatic motion of the 6-dimensional dataset with all colored bubbles, resizing over time):
http://public.tableausoftware.com/views/MotionChart_0/Motion?:embed=y

Please note that the same Motion Chart using Google Spreadsheets will run in browser just fine (I guess because Google "bought" Gapminder and kept its code intact):
https://docs.google.com/spreadsheet/ccc?key=0AuP4OpeAlZ3PdC14OXU1RGJsV05uaDlxRV9GLXlTZXc#gid=2

2. When you have hundreds or thousands of Tableau Server users and more then couple of Admins (users with Administrative privileges), each of Admins can override viewing privileges for any workbook, regardless of designated for that workbook Users and User Groups. In such situation there is a  risk for violation of privacy and confidentiality of data involved, for example for HR Analytics and HR Dashboards and other Visualizations where private, personal and confidential data used.

Tableau Reader enables additional complementary method of delivering Data Visualizations through private channels like password-protected portals, file servers and FTP servers and in certain cases even by-passing Tableau Server entirely.

3. Due popularity of Tableau and ease of use, many groups and teams are considering Tableau as vehicle to delivering of hundreds and even thousands of Visual Reports to hundreds and may be even thousands of users. That can slow down Tableau Server, decrease user experience and create even more confidentiality problems, because it may expose confidential data to unintended users, like report for one store to users from another store.

4. Many small (and not so small either) organizations trying to save on Tableau Server licenses (at least initially) and they still can distribute Tableau-based Data Visualizations; developer(s) will have Tableau Desktop (relatively small investment) and users, clients and customers will use Tableau Reader, while all TWBX files can be distributed over FTP, portals or file servers or even by email. In my experience, when Tableau-based business will grow enough, it will pay  by itself for buying licenses for Tableau Server, so usage of Tableau Reader in n o way is threat to Tbaleau Software bottom line!

Update (12/12/12) for even more happy usage of Tableau Reader: in upcoming Tableau 8 all Tableau Data Extracts - TDEs - can be created and used without any Tableau Server involved. Instead Developer can create/update TDE either with Tableau in UI mode or using Tableau Command Line Interface and script TDEs in batch mode or programmatically with new TDE API (Python, C/C++, Java). It means that Tableau workbooks can be automatically refreshed with new data without any Tableau Server and re-delivered to Tableau Reader users over ... FTP, portals or file servers or even by email.

Sad News about Andreas Lipphard

Founder of BonaVista Systems (Bona Vista = Good View) Mr. Andreas Lipphard, tragically died: http://www.xlcubed.com/the-news/136-sad-news and BonaVista website said that his products are not available anymore: http://www.bonavistasystems.com/

RIP

Qlikview Prices

Qliktech made its price list public on its website. In a move that calls for "other enterprise software and business intelligence vendors to follow suit, QlikTech is taking the mystery out of purchasing software".

I expanded this post with comments and comparison of pricing from Qlikview and Tableau.

I have to mention that Tableau has pricing on its website for years. I wish Tableau will publish on its website the pricing for Core License (for Tableau Server) and more detail for Tableau Digital and Server pricing, but other than that, Tableau is a few years ahead of Qliktech in terms of "pricing transparency"... Also talking with Qliktech sales people until today was more time consuming then needed and I hope that public pricing will make it more easy.

One note about Qlikview pricing: Qliktech has a very weird requirement to buy a Document License ($350 per named user, per 1 (ONE) document) for each document is a potential time-bomb for Qlikview. But they are very good at sales  (Total Q4 2011 revenue of $108.1 million increases 33% compared to fourth quarter of 2010, see http://investor.qlikview.com/secfiling.cfm?filingID=1193125-12-65355&CIK=1305294) and not me, so I will be glad if Qliktech will prove me wrong!

 Again, for now, just review this:

http://www.qlikview.com/us/explore/pricing

I tried to compare the cost of average Deployment  for Qlikview-based and Tableau-based Data Visualization Systems using currently Published prices of Qlikview and Tableau (I actually have an estimation for Spotfire-based deployment too, but TIBCO did not published its pricing yet). See prices in table below, and comparison of average deploymnet after/below this table:

[googleapps domain="docs" dir="spreadsheet/pub" query="hl=en_US&hl=en_US&key=0AuP4OpeAlZ3PdFJYRUNwOXZ6VHdrczFyeDE3a3k5S2c&output=html&widget=true" width="508" height="440" /]

I took as average the deployment with 46 users (it is my estimate of average Qlikview Deployment), 3 desktop clients, 10 documents/visualizations available to 10 (potentially different) named users each, 1 Application Server and maintenance for 3 years.

.

My estimate of total cost for 3 years came up as about $118K for Qlikview Deployment and $83K for Tableau Deployment (both before discounts and taxes and both do not include any development, training, consulting and IT cost).

Note 3/8/12: you may wish to review this blog post too:

http://i3community.com/blogs/entry/qlikview-user-license-named-client-access-license-cal

16 Data Visualization Vendors are squared

Since Gartner keeps doing its "Magic Quadrant" (MQ; see MQ at the very bottom of this post) for Business Intelligence Platforms every year, it forces me to do my

"Yellow Square for DV, 2012"

for Data Visualization (DV) Platforms too. I did it last year and I have to do it again because I disagreed with Gartner in 2011 and I disagree with it again in 2012. I have a few different (from Gartner) views, but I will mention 3.

1. There is no such thing as Business Intelligence as a software platform. It is a marketing term, used as an umbrella for multiple technologies and market segments. Gartner released its MQ for BI at the same time it had "BI Summit 2012" in London on which it practically acknowledged that BI is not a correct term and suggested to use the term "Business Analytics" instead, see for example this article: http://timoelliott.com/blog/2012/02/what-i-found-interesting-about-gartner-bi-summit-2012-london.html

2. I personally is using - for many years - the term Data Visualization as a replacement for BI, as much more specific. Because of that, I removed from consideration a few vendors present in Gartner's MQ for BI and added a few important DV vendors.

3. I used for my assessment 3 groups of criterias, which I already used on this blog before, for example here:

http://apandre.wordpress.com/2011/12/18/dv-comparison-2011/

and here:

http://apandre.wordpress.com/tools/comparison/

As a result, I got a very different from Gartner the placement of "Data Visualization Platforms and their vendors":

.

For reference purposes please see below the Magic Quadrant for BI, published by Gartner this month. As you can see our lists of Vendors are overlapping by 11 companies, but in my opinion their relative positioning is very different:

Data Visualization page on Google+

I started recently the new Data Visualization Google+ page as the extension of this blog here:

https://plus.google.com/111053008130113715119/posts

.

Internet has a lot of articles, pages, blogs, data, demos, vendors, sites, dashboards, charts, tools and other materials related to Data Visualization and this Google+ page will try to point to most relevant items and sometimes to comment on most interesting of them.

.

What was unexpected is a fast success of this Google+ page - in a very short time it got 200+ followers and that number keeps growing!

.

SpreadheetWEB 3.3 is released

New version 3.3 of SpreadsheetWEB with new features like Data Consolidation, User Groups, Advance Analytics and Interactive Charts, is released this month by Cambridge, MA-based Pagos, Inc.

SpreadsheetWEB is known as the best SaaS platform with unique ability to convert Excel spreadsheets to rich web applications with live database connections, integration with SQL Server, support for 336 Excel functions (see full list here http://wiki.pagos.com/display/spreadsheetweb/Supported+Excel+Formulas ), multiple worksheets, Microsoft Drawing, integration with websites and the best Data Collection functionality among BI tools and platforms.

SpreadsheetWEB supports Scripting (Javascript), has own HTML editor, has rich Data Visualization and Dashboarding functionality (32 interactive Chart types are supported, see http://spreadsheetweb.com/support_charts.htm ),

See the simple Video Tutorial about how to create a Web Dashboard with Interactive Charts by publishing your Excel Spreadsheet using SpreadsheetWEB 3.3 here:

SpreadsheetWEB supports Mapping for a while, see video showing how you can create Map application in less then 4 minutes:

as well as PivotTables, Web Services, Batch Processing, and many other new features, see it here: http://spreadsheetweb.com/features.htm

In order to create a SpreadsheetWEB application, all you need is Excel and free SpreadsheetWEB Add-in for Excel, see many impressive online Demos here: http://spreadsheetweb.com/demo.htm