Join Transform 2021 this July 12-16. Register for the AI event of the year.


LinkedIn today announced that it has open-sourced Ambry, a piece of software it built to store and serve up media files like photos, videos, and PDFs. The system is available on GitHub under an open source Apache license.

LinkedIn previously relied on a complex architecture involving closed source technology that was not cheap to scale, even as user numbers and data have both kept increasing. It wasn’t easy to expand, either.

“This convinced us that we needed to find a better approach,” LinkedIn engineering manager Sriram Subramanian wrote in a blog post. “We took it as an opportunity to look at different end-to-end solutions. We looked at many possible solutions including distributed file systems, storage appliances, cloud, and an in-house implementation. We thought deeply about our design goals, understood the tradeoffs and were convinced to build an in-house solution that fit our needs better.”

The system has been in production for more than a year, wrote Subramanian, who has previously worked on Kafka and Samza open source software during his time at LinkedIn.

Of course, other big technology companies, like Facebook and Google, regularly open-source different types of software. But object storage isn’t the biggest area of innovation in the open source world.

Many popular Web services, like Airbnb, Pinterest, and Spotify, look to cloud services like Amazon Web Services’ (AWS) S3 for object storage. It’s not trivial. But LinkedIn is more like Facebook and Google in the sense that it operates its own infrastructure for its services. Hence the need for full software that can operate across multiple LinkedIn data centers. Of course, there are representational state transfer (REST) application programming interfaces (APIs) for working with Ambry, which is written completely in Java.

The system is complex. For a full rundown of its functionality, check out the full blog post and the Ambry wiki.

VentureBeat

VentureBeat's mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact. Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:
  • up-to-date information on the subjects of interest to you
  • our newsletters
  • gated thought-leader content and discounted access to our prized events, such as Transform 2021: Learn More
  • networking features, and more
Become a member