Has lack of access to data due to security been an issue for you? We’ve been running into this more and more lately. The rules surrounding SOX and HIPAA have been interpreted in such a way by many companies that it makes it nearly impossible to do efficient development. We spend more time jumping through hoops trying to figure out how we can code for data that our development team is not even allowed to see. In some cases, our clients’ legal departments have completely shut down our development team’s ability to see any real data that includes personal information.
There are some options out there to work around this issue. One option is to de-identity data. Another option is to manufacture data to be used for development. While these options can help you stay within legal compliance while still getting development done, they add additional work as well as a layer of complexity and uncertainty. The act of de-identifying data could mask underlying issues with the data that won’t be discovered until the solution is migrated to production and real data is pumped through it. The cost of fixing the solution at that point is 100x the cost of fixing it back at the design stage.
The same can be said about engineering data for development and testing. Clever developers know how to manufacture data in such a way that all the data anomalies that are known about can be tested. But once again, there may be other anomalies in the data that simply won’t be recognized until the real live production data is used. In the mean time, the development teams continue to develop the best they can based on the best information they’re able to get their hands on. And we’re spending a lot more time educating our clients on the need for extra development time based on the lack of quality data that is available during the development and testing phases.
USA Today on Business Intelligence
Last week USA Today actually published an article about business intelligence on the front page. The story started with a discussion of a solar panel manufacturer so you had to read a little more to find out that the story really was about BI. I was pleased that an article like this would be published in such a prominent place. However, it was interesting to see that the story presented BI in a way that many people view BI, even though that view is somewhat inaccurate. The view I’m referring to is the one that implies that BI is all about the front end (e.g., reporting).
The article talked about a few BI reporting and analytic tools (Tableau, Cognos, Tibco Spotfire). While these are all great tools, none of them are a complete end-to-end BI solution. Like most front end reporting, visualization, and analytic tool, they work best when they are connected to high-quality data that has been structured correctly.
That’s the part that I wish the article had covered a little better: that good data is vital. To be fair, the article does mention that it is difficult, especially for large enterprises, to integrate data. My only gripe is that more time was not spent on that particular point.
For those of us that have been in BI for a while, we know that the availability and quality of the data is about 99% of the battle. Given good data, we can do just about anything. However, most of our efforts spent trying to 1) find the data 2) understand the data 3) fix the data, and 4) integrate the data. Once we accomplish that Herculean goal, we can finally think about actually presenting and analyzing the information gleaned from the data.
So even if USA Today didn’t say much about the data as I would have, at least they wrote about BI and produced a very good article. And that’s good to see. The data point aside, it really was well done.