Chuck Hollis has posted several times about the emergence of “Big Data.” This post about an analysis of the Big Data market and its practitioners, “Data Scientists,” caught my eye. Here are several observations that jumped out at me:
“I found the ‘wrong organizational structure’ and ‘insufficient executive support’ as two sides of the same coin: about a third of practitioners don’t feel their company is organized for success. Of course, that probably has something to do with both the ‘lack of resources’ and ‘lack of broad-based skills’ observations”
“While many traditional BI analysts are simply functionaries in a larger information-gathering-and-analysis supply chain, the precise opposite seems to be true with data science professionals.
As you can see here, they’re involved from everything from sourcing new data sets (usually from outside the company!) to telling data-driven stories to business stakeholders with the intent of positive change.”
“In many situations, data scientists find themselves working across the entire organization (in addition to other data scientists, of course).
Look at some of the roles they say they work with frequently: graphic designers, HR professionals, marketing, sales, etc. — clearly not just technological professions.”
“The overwhelming number of data science respondents prefer to work in ostensibly smaller settings. Think focused teams, collegial work environment, easy to navigate the organization, and so on.”
“The structure, isolation and inflexibility of most corporate environments appears to be something they’re not warmly embracing.”
This suggests an interaction between the supporting IT infrastructure data scientists depend on and the organization of “data ownership” within the business. IT infrastructure reflects data ownership models. For example, most IT development is defined by business groups who fund projects to build the infrastructure supporting an application important to them. In biological terms, this is a single cell model of data ownership that reflects a similar single cell model for organizing the business functions. Although there are many functions in a business, different functions remain isolated from each other and in many instances expend considerable energy defending their perimeter similar to how single celled organisms behave.
Evolution moved on from single cell organisms to multi-cellular and in time we end up with cells that can network together forming the brain. I have been reading about the brain and its plasticity in a book by Norman Doidge, “The Brain That Changes Itself.” The striking thing about this research is the discovery of ability of the brain to reinforce connections that are in use and interconnect more neurons to improve processing for tasks that are in demand. The brain creates a highly interconnected topology of neurons across large areas. In short, the brain’s network acts like a fabric.
Back to Big Data, IT infrastructure and business organization. If most applications and their data act like self-contained single cell organisms, then Big Data is the new multi-cellular organism. To work efficiently, it requires brain-like connective tissue, neurons, that can dynamically and quickly connect together as needed to process more information. Big Data needs a network that provides a high level of interconnectivity between individual nodes containing data resources. Further, as more access occurs between nodes, network connectivity should increase with minimal effort. Ethernet fabrics inherently provide this type of network flexibility, scalability and policy mobility as workloads migrate. They will be important as Big Data becomes more common.
As Chuck points out, the multi-cellular organism that is Big Data alters the business structure itself. Instead of single celled owners of data (aka, departments) with a defensive perimeter to protect “their” data, a highly connected multi-cellular structure, like the brain, has superior abilities to respond to environmental change. It will dominate for this reason and cause organizational changes. Rigid hierarchy will give way to “on-demand” structures where skills in needed for a particular project can easily connect together to do work. Imagine using “Craig’s List” to advertise a Big Data project and have interested people sign up to work on it. And when the work is done, resources (people with brains) disconnect from one project to connect to a new one. The need for a dynamic network now extends to the campus/LAN not just the data center. The campus network has to enable collaboration, real time desktop-to-desktop video streams, phone conversations and chat. I think Unified Communications promises to provide this and once again, the connective tissue of the network begins to look like a fabric.
It may be that Big Data is the driver for not only new networking constructs but also new business structures. With a fabric for the connective tissue, businesses can operate with the same efficient plasticity as the human brain.
