Comments on: YouTube Scalability Talk

by: Frank Malina

Thu, 03 Jul 2008 14:09:59 +0000

Very concise reading.
I’ve just wrote a similar article on the video streaming applications, although it’s from the top of my head.

by: nelix

Sun, 22 Jul 2007 22:29:55 +0000

A good read, psyco is makes multiple versions of each block for different types (python is dynamically typed, which is slow) so the type bottlenecks are removed at run time, to some extent anyway. At least that’s how I understand it.

by: nsfun

Wed, 18 Jul 2007 13:31:23 +0000

and youtube uses netscaler … a small detail

by: Charles Iliya Krempeaux

Tue, 17 Jul 2007 14:42:26 +0000

Alot of this sounds like the scalability techniques we used to build an ad network… although we were using PHP and not Python.

We were doing about 1000 hits per second on average… although it could be very spikey traffic… so it could shoot way up, beyond that, on a spike. (It almost like a getting DDoS’ed all the time.)

Believe it or not… there’s scalability problems (and solutions) beyond those listed here. Interesting stuff to work on, IMO.

The more and more you have to scale… things like database accesses become to “expensive”, and you have to remove them. Table JOIN’s need to be taken out too (… to “expensive”).

In-memory databases sometime become a solution.

And logging things in a file… instead of INSERT’ing into or UPDATE’ing a database also become necessary.

ReiserFS was a help.

Also… sometimes you need data centers spread around the world… with slightly different versions of your system optimized for those locations.

– Charles

by: metale

Mon, 16 Jul 2007 16:33:26 +0000

Simply Superb…!!

by: geekmaster grok

Mon, 16 Jul 2007 06:33:17 +0000

such an excellent read. thanks.

by: spect

Mon, 16 Jul 2007 05:55:45 +0000

its a rare pleasure to gain this kind of architecture and engineering insight from some of the larger web based companies. used to be easier, but i dont live in the bay area anymore, wish i could find more info like this.

thanks

by: Adarsh

Mon, 16 Jul 2007 04:48:20 +0000

Excellent Discovery !

by: Seo Sanghyeon

Mon, 16 Jul 2007 04:32:49 +0000

psyco is not a Python->C compiler. It’s Python JIT (although not exactly).

by: Chris

Mon, 16 Jul 2007 01:23:08 +0000

This is awesome. Thanks for taking the time to expose the underpinnings!

Very interesting to see that they’re using Lighttpd. I have to agree with your comment on nginx … perhaps they’ll move to it in the future.