The latest buzz in the Business Intelligence (BI) Data Warehousing (DW) arena is “Big Data”. Visit any BI/DW software vendor website and you’ll see “BIG DATA” splashed everywhere and how their product supports/implements/leverages big data. (Not that long ago, the “cloud” was all the rage).
What is behind the buzz of big data? And more importantly, how do federal agencies assess the value proposition of big data? We’ll start with a definition first.
Big Data = Data Everywhere Now In One Place
Most federal agencies have some sort of data warehouse, analytical data marts and front-end reporting and analysis tools. More than likely, the data in this environment is agency data relating to the mission (e.g., crop insurance, health, etc.) or functional data (e.g., human resources, financial, etc.). The source of this mission and functional data is the agency itself, or more succinctly, the structured data from the agency’s operational systems.
Big data is essentially structured and unstructured data from multiple sources (agency, public, proprietary) integrated in one place to derive deep insights. What makes big data different from ‘regular’ data is that big data includes unstructured data, such as data from social networks (think Twitter, Facebook, and blogs) and data from multiple sources, both internal and external to the agency. All of this data (and lots of it, that’s why it’s called big) is combined in one place to provide insights that would be difficult if not impossible to derive from the disparate sources. Big data essentially takes the DW/BI paradigm to the extreme: data everywhere now in one place.
What Problem Are You Solving with your Data?
Now that you know what big data is, the question now becomes: What problem are you trying to solve by using big data? The commercial sector, driven by the profit-motive, is using big data to ultimately increase revenues. Government agencies don’t have a profit-motive but instead focus on strategic outcomes. So how do federal agencies determine if big data is right for them?
First, check your agency’s strategic plan. What outcomes is your agency trying to achieve? How would big data help achieve those outcomes? Many agency strategic plans are high-level and put the implementation approaches into business and IT plans so check those for guidance.
Second, check your agencies BI Intelligence Roadmap (don’t have one – read my previous post on federal BI roadmaps: ). Big data is a natural extension of BI/DW initiatives and a good BI Roadmap details the business drivers and business outcomes that BI/DW program will help achieve. Big data may be a key technology enabler that could help achieve those goals.
Think Big. Start Small
Working with federal agencies as business intelligence consultants and business performance management analysts, we have found that many agencies struggle with “traditional” BI/DW initiatives (lack of business sponsorship, lack of budget, lack of governance, etc.). Tackling a big data initiative without having your small data initiative in order is not a recommended approach.
You can still investigate big data, perhaps even do a small pilot, but first ensure business alignment. Think big but start small.
By David McIntire, Director of Federal Consulting
© DecisionPath Consulting, 2011