Facebook rolls out storage system to wrangle massive photo stores

Homegrown system, Facebook Haystack, built to handle multiplying files of user photos

Needing to better deal with 50 billion files worth of photos, engineers at Facebook are installing a new photo storage system they say is 50% faster than traditional systems.

The storage system, dubbed Haystack, has been under development in-house for the past couple of years, and Facebook has been rolling it out in limited test versions to parts of the network for the past few months. The company expects to use Haystack to store all Facebook photos by next week, according to Bobby Johnson, director of engineering at Facebook.

And Jonathan Heiliger, vice president of technical operations, toldComputerworld today that based on tests, Haystack is more than 50% faster than traditional photo storage systems.

"In terms of cost, if it's twice as efficient, we can have 50% less hardware," said Johnson. "With 50 billion files on disk, the cost adds up. It's essentially giving us some [financial] headroom."

Johnson and Heiliger said they began building the new storage system to better handle the growing number ofphotos Facebook has to store. Many of their 175 million and 200 million users share photos of everything from their pets to vacations, weddings and days at the beach. That means users are posting and calling up their own photos, as well as their friends' and family members' photos. Keeping the system running efficiently was a growing challenge.

Johnson noted that Facebook deals with 15 billion photos - not including all of the replications. User data grows by 500GB per day. And Facebook has 50 million requests per second to its back-end servers.

A spokesman for Facebook said more specifics about the new system will be released in a few weeks.

Johnson, though, said the system is so much faster than the previous one because of changes made to its setup. Haystack is tailored for small files that don't change very often, instead of for a small number of large files that are changing all the time. Traditional file directories also need file names, and a lot of resource cost goes to just finding the files. The new system uses ID numbers instead of names; that mapping is very small and doesn't involve directory structures or file names.

Johnson said that so far, the rollout of the new system has gone very smoothly.

Five-year-old Facebook's user base passed one-time leader MySpace last year, according to a recent report.

Facebook, once regarded as the up-and-coming social network, had almost 222 million unique visitors last month, while MySpace came in at 125 million, according to online researcher comScore Inc. That's a dramatic change, since the Facebook-MySpace race for unique visitors was a near dead heat in April 2008.

The company is closing in on a big milestone -- 200 million users, executives said today.

Massive computer spy network uncovered

Massive computer spy network uncovered; Google downplays Google Docs security concerns; and Intel to release faster low-power laptop chips.

Subscribe to this podcast in iTunes!

A 10-month cyberespionage investigation has found that 1,295 computers in 103 countries and belonging to international institutions have been spied on, with some circumstantial evidence suggesting China may be to blame. The 53-page report, released on Sunday, provides some of the most compelling evidence and detail of the efforts of politically-motivated hackers while raising questions about their ties with government-sanctioned cyberspying operations. It describes a network which researchers have called GhostNet, which primarily uses a malicious software program called gh0st RAT to steal sensitive documents, control Web cams and completely control infected computers. The analysts did say, however, they have no confirmation if the information obtained has ended up being valuable to the hackers or whether it has been commercially sold or passed on as intelligence.

Google Docs users shouldn't lose sleep over the security concerns a security analyst has raised about the hosted suite of office productivity applications, Google said late Friday. In an official blog posting, Jonathan Rochelle, Google Docs' product manager, details why the company has determined that the issues included in the analyst's report are far from critical. Google's conclusions aren't a surprise. Hours after Ade Barkah published his report last Thursday, Google responded with a preliminary statement saying it was investigating the matter but that it didn't believe there were significant security issues with Docs. Nonetheless, Google evidently sees some merit in Barkah's report. Google has added information regarding Barkah's observations to its Docs "help" pages about creating drawings and about adding viewers and collaborators to documents.

Intel is expected to refresh its line of laptop chips Monday with new ultra-low-voltage processors that should make ultraportable laptops operate faster without sacrificing battery life. Intel currently offers ultra-low-voltage processors for fully functional thin and light laptops, such as Apple's expensive MacBook Air and Lenovo's ThinkPad X300. These chips fit into small spaces and draw less power than conventional laptop chips. The chips are about the size of a dime, or 60 percent smaller than mainstream laptop chips. The chips will run at clock speeds of up to 1.6GHz, a speed bump from earlier chips that ran at up to 1.40GHz.

The U.S. Library of Congress has begun uploading its audio archives to iTunes, and it will soon begin to post videos on YouTube, in an effort to make its materials easier for the public to access. The library already offers the materials at its own Web site but the expansion to YouTube and Apple's iTunes is part of the library's efforts to make its 15.3 million digital items more accessible. The decision to post audio and video on iTunes and YouTube follows a successful launch early last year of a library photo archive on Flickr. Since January 2008, the library's photos on Flickr have been viewed about 15.7 million times

...And those are the top stories from the IDG Global IT News Update, brought to you by the IDG News Service. I'm Sumner Lemon in Singapore. Join us again later for more news from the world of technology.