Vol. 1, No. 3 | October, 2010
Big Data in the wild: lessons learned
Over the course of the past ten months, jStart has been busily working with its clients to identify Big Data challenges facing them, as well as exploring Big Data tools and solutions to better understand how those tools can be most effectively leveraged...all, of course, with an eye towards providing real return and business value. This edition of jStart's Flash will address some of what we've learned...
What is Big Data?
When jStart first started working with Big Data, we had a pretty straight-forward definition of what Big Data meant: copious amounts of data, typically in the petabyte range. But what we've learned over the past ten months is that the size of the data is just one of three factors we've seen as part of the Big Data challenge:
- Very Large Data Sets. Typically, this means data of significant size--petabytes of data. To give some idea of the size we're talking about, a petabyte is equivalent to about 512 of the largest hard drives available on the market today (2 terabyte hard drives). With every one of them filled to the brim with data.
- Very Large Data Flow. Another factor in determining if a company has a "big data challenge" is the rate or volume of data the company is dealing with. Although the amount of data may be smaller (megabytes), the volume may be such that the amount of data that has to be processed immediately is very high. Think of it as a "firehose of data"--an example: analyzing Twitter feeds.
- Very Time Sensistive Data. The last factor we've seen is the time sensitivity of the data. This factor is usually accompanied by one of the other two factors, but basically means that the value of the data has a very short half-life. To extend our previous example, perhaps a company needs to analyze twitter data in real time. This is often referred to as the "speed to insight" challenge: it's not just how fast you can process the analytics, it's how fast you can surface the insights big data is providing your company, whether it be through effective visualizations or deep sophisticated multiple-pass analytics.
Some Innovative Thinking
In the July Edition of jStart's Flash, we discussed three scenarios ranging from law enforcement, to green energy, to leveraging big data to understand customer sentiment for retail. We've also talked about how jStart used Big Data to help a major research institution identify and discover business partners for the research IP it was developing. But what we didn't realize at the time is that Big Data has applicability across virtually every industry vertical that jStart has ever engaged in...and that companies were using Big Data tooling to provide them levels of insight in innovative ways which we hadn't even considered. The upshot? Our eyes were opened to the opportunities that Big Data provided--Big Data challenges were all around us. It was sort of like stumbling in the dark trying to find a dropped penny, only to have someone to turn the lights on and to realize you were standing in a bank vault.
Accelerating Healthcare Research
Using Big Data tooling, raw data from a leading teaching hospital was directly fed into research being done at one of the US's top medical research universities--providing levels of insight previously difficult if not impossible to obtain accelerating cutting edge medical research efforts.
Read the scenario (177KB PDF)
Enabling Startups with Big Data
How did two startups leverage Big Data tooling to extend and enhance their businesses--allowing them to hit the ground running?
Read the scenario (374KB PDF)
Matching Company Skills with Opportunities
How could a British university help match companies with business opportunities by leveraging big data? Learn about jStart's work to do just that...
Read the scenario (132KB PDF)
Feeding the World: A Big Data Opportunity
Could Big Data be used to encourage research collaboration to create better crops? jStart worked with one of the world's leading plant science research businesses to enable collaboration through Big Data discovery.
Read the scenario (274KB PDF)
jStart On The Road
Not content to just talk about Big Data to our clients, jStart will be participating in a series of upcoming conferences and seminars to talk about Big Data, and to illustrate some of the ways we've seen companies leverage Big Data for the benefit. Our next stop in the jStart Big Data Roadshow: the Internet Summit '10, in Raleigh, North Carolina (USA). | Sound interesting? Read more about it.
Using Our Knowledge & Expertise
So, the past ten months have been an exciting time for the jStart team: we've been able to dive into an emerging technology, we've been heavily involved in some of the new technologies coming out of IBM's Research efforts, but also efforts from the Open Source Community. And, perhaps most importantly of all, we've been able to help our clients understand the challenges and opportunities Big Data provides them. What are some of the lessons we've learned? Here are just a few:
- Understand what kind of Big Data Challenge/Opportunity you have. Is it just a huge amount of data? Or is it a significant volume of data? Is it time sensitive? Perhaps "all of the above"?
- Is there a way to leverage a challenge into an opportunity? In almost every case, jStart was brought in to address a problem...but the problem also presented an opportunity (better understand your clients, better anticipate issues/problem areas, enable collaboration in unexpected ways).
- Who needs to understand what your Big Data is trying to tell you? Does an line-of-business professional need to understand the insights/opportunities provided? How sophisticated must be the person interpretting the data analysis? The complexity of a Big Data solution is often inversely proportional to the level of expertise of the person analyzing the results--the higher the level of expertise, the simpler the Big Data implementation, the lower the level of expertise, the more complex the Big Data solution must be in order to provide the information in an intuitive manner.
- Big Data solutions must be thought of strategically. Through our engagements, we soon understood that in order for companies to extract the maximum value out of their Big Data tools, they had to think of Big Data in strategic terms: how would the tools impact the way the company does business? This, in turn, lead to discovery of opportunities that Big Data provided in the face of the challenge originally addressed.
Interested in learning more? Contact us and we'll be happy to have a discussion with you about our Big Data experiences--and how that experience can be leveraged by your particular Big Data needs.
Has this e-zine been useful? You can sign up for future editions of jStart's quarterly Flash:
Keep tabs on jStart
About jStart's Flash
jStart's Flash is a newsletter to keep you informed of the latest with IBM Emerging Technology's client engagement team, jStart. Our mission within IBM software group is to explore emerging internet technologies in partnership with our clients and customers.
Resources + Links
In this edition...
Want more information?
Want to know more about these jStart topics?