Comments on: YouTube’s 45 Terabytes… no big deal? http://kylecordes.com/2006/08/30/youtube-tb/ Kyle Cordes's Software Site Sat, 30 Aug 2008 07:28:07 +0000 http://wordpress.org/?v=2.0.11 by: Bob Caswell http://kylecordes.com/2006/08/30/youtube-tb/#comment-11496 Sat, 08 Sep 2007 16:50:23 +0000 http://kylecordes.com/2006/08/30/youtube-tb/#comment-11496 That story has moved, here’s the updated location: http://www.techconsumer.com/2006/08/30/youtube-stats-revealed-total-viewing-time-9305-years/ That story has moved, here’s the updated location:

http://www.techconsumer.com/2006/08/30/youtube-stats-revealed-total-viewing-time-9305-years/

]]> by: plop http://kylecordes.com/2006/08/30/youtube-tb/#comment-2547 Wed, 28 Mar 2007 14:57:26 +0000 http://kylecordes.com/2006/08/30/youtube-tb/#comment-2547 Kyle just a quick question... don't you think that the 45 TB figure is already taking into account the replicate servers? I've seen quotes stating that they host about 6.1 million videos, if you divide the 45 TB by that, it would make up to about 7.4 MB per video. The average video on youtube according to Chad Hurley is 2 1/2 minutes, 7.4 mb seems a bit too much. Kyle just a quick question… don’t you think that the 45 TB figure is already taking into account the replicate servers? I’ve seen quotes stating that they host about 6.1 million videos, if you divide the 45 TB by that, it would make up to about 7.4 MB per video. The average video on youtube according to Chad Hurley is 2 1/2 minutes, 7.4 mb seems a bit too much.

]]> by: Amit Patel http://kylecordes.com/2006/08/30/youtube-tb/#comment-59 Thu, 14 Sep 2006 05:05:56 +0000 http://kylecordes.com/2006/08/30/youtube-tb/#comment-59 I agree with you that 45 TB is not that much data, and I agree with you that you don't need high-end drives. YouTube's videos are low resolution so that they can stream them. One streaming video uses at most 100 kilobytes/sec (~ 768k DSL). I'd guess they use a lot less to save bandwidth. A modern SATA hard drive can probably stream at least 35 megabytes/sec. Why would you need 10k rpm drives for this? Just read the entire 30 second video (3 MB?) into cache in 90ms, then spend 10ms seeking to the next video. You should be able to read 300 30-second clips each 30 seconds, serving 300 users from each drive. If they have 1000 drives, that's 300,000 simultaneous viewers. It's likely that videos aren't evenly distributed though, so popular videos can be cached in RAM and served to lots of users without more disk seeks, and they can probably serve even more users with those disks. I agree with you that 45 TB is not that much data, and I agree with you that you don’t need high-end drives.

YouTube’s videos are low resolution so that they can stream them. One streaming video uses at most 100 kilobytes/sec (~ 768k DSL). I’d guess they use a lot less to save bandwidth. A modern SATA hard drive can probably stream at least 35 megabytes/sec. Why would you need 10k rpm drives for this? Just read the entire 30 second video (3 MB?) into cache in 90ms, then spend 10ms seeking to the next video. You should be able to read 300 30-second clips each 30 seconds, serving 300 users from each drive. If they have 1000 drives, that’s 300,000 simultaneous viewers. It’s likely that videos aren’t evenly distributed though, so popular videos can be cached in RAM and served to lots of users without more disk seeks, and they can probably serve even more users with those disks.

]]> by: kyle http://kylecordes.com/2006/08/30/youtube-tb/#comment-45 Sat, 02 Sep 2006 12:49:26 +0000 http://kylecordes.com/2006/08/30/youtube-tb/#comment-45 Aurash commented that they would need high-end drives, much higher cost per drive... but I disagree. As I pointed out above, they are much better off per dollar, with more "normal" drives than with faster drives, for an application like this. HIgh end (fast) drives are for busy DBMSs and the like, not for highly parallelizable static data serving. Of course, the math works out rough the same way if you assume a smaller number of high-end drives, so even if the architect chooses a solution with high-end hard drives, 45 TB is still not all that much data. Aurash commented that they would need high-end drives, much higher cost per drive… but I disagree. As I pointed out above, they are much better off per dollar, with more “normal” drives than with faster drives, for an application like this. HIgh end (fast) drives are for busy DBMSs and the like, not for highly parallelizable static data serving. Of course, the math works out rough the same way if you assume a smaller number of high-end drives, so even if the architect chooses a solution with high-end hard drives, 45 TB is still not all that much data.

]]> by: Aurash http://kylecordes.com/2006/08/30/youtube-tb/#comment-44 Sat, 02 Sep 2006 08:25:12 +0000 http://kylecordes.com/2006/08/30/youtube-tb/#comment-44 not necessarily true, remember that YouTube hosts a service, so you can be pretty damn sure that those drives are AT LEAST 10k rpm, so toss that 200$ / 300GB drive figure out the window, its more likely to be 3x or more than that. per drive. not necessarily true, remember that YouTube hosts a service, so you can be pretty damn sure that those drives are AT LEAST 10k rpm, so toss that 200$ / 300GB drive figure out the window, its more likely to be 3x or more than that. per drive.

]]>