Recent technological developments improved the ways in which we can create and manage different types of digital data. As more and more documents go online, better solutions are being offered by cloud service providers. While the simplest apps can generally solve individual necessities for storage space and file transfer, the problems still exist in corporate settings where the amount of data generated daily is much bigger.
The services like Dropbox and SugarSync are primarily intended for individual consumers but there are other cloud services that focus on creating corporate data management solutions. The amount of digital data in companies and organizations grows at a very high rate and now many of them face the challenge of big data processing.
This is why big data is increasingly becoming a focus of cloud industry. Big data is a common name for huge amounts of digital data that cannot easily be processed and transmitted. Before the rise of the Internet and computing resources, such data existed in big scientific projects only. Now that every enterprise deals with huge amounts of digital data it grows more important to solve problems related to their storage, transfer and processing.
One of the most important services built for processing data intensive computing applications is Apache Hadoop which is almost by default related to big data. Through a cluster of computing nodes Hadoop’s Map Reduce breaks data into pieces and processes them separately which is particularly convenient for unstructured data. Many web giants and otherwise big organizations have long shifted to Hadoop support. Amazon, Apple, eBay, Facebook and others use Hadoop to process huge amounts of unstructured data and Yahoo!, Microsoft and IBM have long integrated it in their offerings.
NASA is another major company that relies on Hadoop. NASA uses Hadoop to support data loads in their huge projects such as Square Kilometer Array skyimaging which is expected to produce 700TBps when built in the next decade. Mars Curiosity rover mission is also powered by cloud computing platforms from Amazon Web Services (AWS) that enable easier and faster communication with the rover. On the recently held AWS re: Invent conference NASA administrators said that Jet Propulsion Laboratory staff will probably rely on cloud computing even more.
Gartner Big Data forecast
Big data market is frequently discussed topic and it revolves around the fact that companies increasingly have to manage exabytes of data. Gartner points out that big data makes companies smarter and more productive but also represents a particular challenge to enterprise architecture practitioners. Traditional information architecture shifted from simple data storage to data pooling and analytics. David Newman, research vice president at Gartner says that the task of EA practitioner in the age of big data should be the following: “Design business outcomes that exploit big data opportunities inside and outside the organization.”