Google Breaks Speed Record, Sorting 1PB Data In 6 hours
Published on 11/29/08 at 19:00:01 by On
Internet
15/05/2009: Hosting and registrar fraud by Parava Networks
19/02/2009: Pirate Bay Trial goes LIVE
01/02/2009: Google’s misplaced slash banned the entire Internet
17/05/2007: DreamHost: The Big 5-oh-oh-oh-oh-oh!
InternetGoogle Systems Infrastructure Team has announced that they have been able to sort 1 terabyte of data across one-thousand computers in a mere 68 seconds. (Previously, the same record of 1TB was across 910 computers and took 209 seconds.) For one petabyte of data, it took Google 6 hours and 2 minutes to do the sorting—and the sorted data gets moved to 48,000 hard drives.

Google also acknowledges that they had triple redundancy—the data was backed up on 3 hard drives. Could it be that if you reduced that redundancy, this would all be much quicker? Not necessarily, since storing may not be part of the sorting algorithm that was recorded (and storing the data could have still been ongoing after the sort had been completed).

According to the Google Systems Infrastructure Team during sorting experiments they have followed the rules of a standard terabyte (TB) sort benchmark. The standardized experiments helped understand and compare the benefits of various technologies and also add a competitive spirit. You can think of it as an Olympic event for computations. By pushing the boundaries of these types of programs, one can learn about the limitations of current technologies as well as the lessons useful in designing next generation computing platforms. This, in turn, should help everyone have faster access to higher-quality information.

Read more here: Grzegorz Czajkowski, Systems Infrastructure Team - Sorting 1PB with MapReduce

1 comment, (4950 reads) All Articles by, On
  Printer Friendly version - Google Breaks Speed Record, Sorting 1PB Data In 6 hours   Log in to use this feature 
Sign Up
Comments on this article: Comments on this article:

1. abendkleider katalog Written on 08/02/12 at 22:43:19 by carmencha





The comments are owned by the poster. We aren't responsible for its content.
Only registered members may comment on articles.