Have you wondered why a lot bigger data businesses are in the private sector and not in the public sector?
Big data is a new practice. While the concept was traced to Fremont Rider, a Wesleyan University Librarian, in 1944, only in 2005 did Yahoo built Hadoop on top of Google MapReduce. Coupled with the use of the words ‘big data’ by Roger on O’Reilly Media in the same year, both form some of the remarkable steps that got us to where we are today in our understand and practice of big data Ramesh Dontha (2017). Even so, approximately 12 years of high-tech experimentation of big data is leaving us with many challenges on how to navigate this cross-cutting resource in the public sector.

As new inventions and innovative ideas like Big Data are known to be risky ventures, the private sector often spearheads the pioneering of new ideas, and when significant progress is realized, the public sector and of course the third sector start to come on board. This was the case with global best practices like performance measurement, monitoring and evaluation, and related techniques in the field of organizational science and technology, which the private sector initiated to improve institutional behavior and resultant outcomes.

A lot more data business resides in the private sector, essentially driven by commercial gains and enabled by the monopoly of data science and technology. For big private businesses like Google, Facebook, Amazon, Yahoo and many others, big data is big business and big money. Some even call it the new oil industry or the 21st-century gold rush. However, the controversies of these gains continue to swell as the bridge to the security of public records and abuse of information privacy lacks adequate global policing.

As of 2017, the public sector is nowhere near the corridors of the gains of big data. Data science or data technology or data architecture (whatever the terminology may be) is not seating well with traditional bureaucratic practices. First, many government agencies and parastatal perceive big data as an added weight to their overburdened culture of civil service – it means new learning, more spending and forfeiting their comfort zones.

A cultural shift by which new technology brings into governance is probably the greatest challenge, especially for core civil service sub-sectors.
However, the few public departments engaged in big data management, still face difficult data challenges deserving careful attention.

1. Paper-based data. At least 60% of government transaction are still paper-based and this is even more for developing countries. We cannot completely rule out the role of paper and hard copy files in this sector, but a substantial part of government operations needs automation for the purposes of saving time, cost and human lives.
2. Inappropriate data infrastructural systems. Government data often resides in big relational database management infrastructures like Oracle, MySQL, Microsoft SQL Server, PostgreSQL, IBM DB2 or a complex combination of these with other smaller applications. The security and services departments including financial and commercialized agencies are some of the ones that can create new technology departments and employ new staff side-by-side with consultancy engagements to solve big data issues. Spending for such departments is probably not a problem because Information Technology (IT) forms a huge annual budgetary allocation. But because government policies and goals are constantly changing, data infrastructure often requires reformation and sometimes outright replacement thereby making data management cumbersome.
3. Inability to transpose data into useful information. Traditionally, government agencies accumulation large and dormant data some of which are seating in static computers and inactive databases. Data can be dormant for many reasons but essentially due to obsolete infrastructures and insufficient incentives to generate high-level insights. In some cases, bureaucracy delays the data ecosystem from catching up with new technologies. However, it’s a painstaking exercise to design Key Performing Indicators (KPI) for civil servants to start with let alone transpose each indicator to a visible gain. The time lag between data mapping, data collection, and data analysis could last longer than expected thereby leaking the gains of big data. Most vendors find a niche across these processes where they provide a win-win situation.
4. Less evidenced based and more political, discretionary reasoning in decision making. An evidence-based policy is not an indicator for measuring political performance at best, it is considered a by-product of good governance. Where decision making is driven by political, ethnic, regional and religious considerations data analytics +is not necessarily perceived as a resource needing attention. Therefore, data is not yet an asset.
5. Segmented and decentralized data sources. It is a natural phenomenon in the public sector to have departments and units operate in silos; some would say ‘that is how it works in government’, but technology can improve information sharing and enhance visibility for transparency and accountability gains. Every government wants to save cost and achieve a return on investments; therefore, information sharing which normally minimizes duplication and replication could help achieve these goals.
6. Inadequate and poor-quality data. At least 95% of human-generated and metadata have errors in them and much of government data are humanly generated. This situation is often compounded by multiple paper-based data collection tools and ‘mix languages’ in data interpretation. If data quality is tantamount to solving real-life situations, then the government needs more quality data than any sector in the world.
7. Database silos. Smaller government departments often store their data in multiple databases ranging from local portals to online and cloud base. The search for cheap or free options has led to a complex mix of technologies leading to higher costs the long run. Unfortunately, these departments have no ceilings and; the knowledge to how these options are linked to each another , making data the newest challenge to information access and utilization. Vendors should not take advantage of this situation.
8. Inadequate human skills at all data levels. There is an acute skill shortage for big data management across government departments. The need for programmers, statisticians, data business experts, data scientists and software developers etc. They often resort to consulting services which are expensive and less sustainable.
9. Lots of information but confusing choices on the whole business of big data. There is no quick fix to these challenges, but the big data industry can minimize the complexities of public sector dependence by cutting down the unnecessary competition that leaves the sector more confused.

Despite the unique challenges of implementing big data in the public sector, its potential gains are remarkable. Developments in data science offer a tremendous opportunity to improve higher levels of accuracy in public policy formulation and decision-making.

Not just in science, technology, and commercial profits, government departments can demonstrate the gains of political parties and democracy through big data management. Where the use of evidence is necessary to check accountability and transparency, governments have little or no choice but to make data business work