Jon Corbet's Linux Forecast, Netflix and More from Collaboration Summit

Afternoon sessions at The Linux Foundation’s Collaboration Summit featured Netflix, Intel, Red Hat, Linux Foundation collaborative projects and the Linux weather forecast by Jon Corbet. Here are some of the highlights.

Netflix

As of November, the Neflix streaming video service comprised about one third of the downstream web traffic in the U.S., said Adrian Cockcroft, Director of Architecture for the cloud systems team at Netflix.

“Thanks for building the Internet so we can fill it with movies,” Cockcroft said. 

Incredibly, there is no data center behind Netflix. The application is “cloud native”, running on 500 to 1,000 Linux-based web machines and distributed across three Amazon service zones, an infrastructure that provides a highly agile and available service, Cockroft said. If something goes wrong, Netflix can continue to run the entire service on two out of three zones — a scenario it tests often with its open source Chaos Gorilla software. 

“We go around trying to break things to prove everything is resistant to it,” he said. 

This year, Netflix is also working on multi-region availability so that if anything goes wrong in one zone they can easily switch traffic back and forth across regions. This will prove a challenge, though, as DNS vendors all have wildly different APIs and are designed to be hand-managed by an engineer, Cockroft said.

Still, the company won’t be satisfied until it has created a system that will work as a complete, cloud-native open source platform. It has the functionality and scale they desire, but they are adding more features in an effort to make it more portable and easier to deploy. They’re also seeking contributions from end users and vendors such as Eucalyptus, Cloudstack and OpenStack that are interested in using NetflixOSS in private enterprise clouds.

The Netflix open source program is growing and the company is making an effort now to come up with a set of best practices; hire and retain top engineers; and build up the Netflix technology brand and the benefits of a shared ecosystem.

Linux Foundation Collaborative Projects Panel

With two recent announcements of new collaborative projects at the Linux Foundation, OpenDaylight and Xen, it was an ideal time for executive director Jim Zemlin to lead a panel on taking a project open source and the benefits of working with the foundation. Beyond the standard challenges of creating great code, choosing a license and governance model, open source projects must also attract new community members — that are often competitors — and create shared resources for marketing, training and other programs. 

It’s our mission to “spread the collaborative DNA of Linux” to other industries and projects, Zemlin said.

In creating the openMAMA project, a market data API for the financial industry, the New York Stock Exchange was seeking broad adoption and started to explore open source as an option. The company thought originally it would only open source the header files but involving the Linux Foundation, they learned that wouldn’t be enough to attract participants, said Michael Schonberg, Director of Market Data Engineering at Quincy Data and an openMAMA founder. Working with the foundation also helped NYSE bring its legal team up to speed quickly on open source practices and lended credibility to the project with its competitors.

“We ended up open sourcing more than we thought we would,” Schonberg said. 

The Yocto Project was founded on open source projects such as Open Embedded, said Tracey Erway, a Senior Marketing Manager at Intel and Advocacy lead for the Yocto Project. But similar to the NYSE, Intel was seeking a solution for industry-wide problems, not just its own. They want to grow the number of projects compatible with Yocto, further develop the community, attract more users and do developer training.

“The Linux Foundation made the process easier for us in a hundred million different ways,” Erway said. “It’s not anything Intel could have pulled off on its own. The results are really showing how well a project can be kicked off and be highly successful.”

Unlike Yocto or openMAMA, the Xen Project had been around for a decade before coming onto The Linux Foundation as a collaborative project. It is, however, trying to expand from a project sponsored primarily by one company, Citrix, to having all 30 contributing organizations come together to collectively drive the project forward. 

“As a project we haven’t worked and engaged the users as much. The Linux Foundation is really focused on that,” said Lars Kurth, community manager for the Xen Project. The project aims to grow its contributor base by working with Linux Foundation members, he said.

Linux weather forecast 

The Linux kernel has made great strides since the 3.3 kernel was released in March 2012. Some 3,172 developers working for more than 370 companies merged 68,000 changesets, released 5 kernels and added 1.53 million lines of code. Big changes in the past year include bufferbloat fixes, an improved networking subsystem and adding 64-bit ARM architecture, among many others.  

“We’ve been busy, as usual,” said Jonathan Corbet, a Linux kernel contributor and co-founder and editor of LWN.net, in his annual Linux weather forecast. 

The 3.9 release could come as soon as this weekend, poised to set a record for the most developers involved in a single kernel release. Intel is also set to replace Red Hat as the No. 1 contributor to the kernel, behind volunteer developers. Mobile and embedded companies such as Samsung, TI and Linaro have also increased their contributions to the kernel, Corbet said.

3.9 will bring improvements such as KVM virtualization support on ARM, the addition of PowerClamp for managing power consumption of processiors in a datacenter setting, and dm-cache for increasing storage performance.

As for future releases, Corbet predicts kernel developers will continue to tackle NUMA scheduling and memory management as well as power-aware scheduling and big.LITTLE. In networking he focused on multipath TCP that would allow devices to use both wi-fi and cellular interfaces together. One other area of development is namespaces, a piece of the containers issue, Corbet said. The problem is that namespaces fixes right now create security issues and therefore won’t be enabled in distributions anytime soon, he said.

In conclusion, Corbet said the Linux and open source communities can no longer be accused of following in the footsteps of proprietary software. 

“We’ve now reached a point where interesting things tend to be done on Linux first,” Corbet said.

But, this also means the Linux community is navigating in the dark. Without a clear vision of what it will become, different notions of the Linux system have emerged, including the Android and Ubuntu architectures, with disputes that have sometimes become public. 

“Everybody is trying to distinguish themselves in a way that we’re getting back to a point that wrecked Unix,” Corbet said. It hasn’t happened yet, he said, “but we need to keep this in mind.”

Intel 

Open source enables freedom and innovation. This was the message of the afternoon’s final two speakers: Dirk Hohndel, Chief Linux and Open Source Technologist at Intel and John Mark Walker, Gluster Community Leader at Red Hat.

Intel has seen a dramatic shift toward open source, Hohndel said, and is now advocating for the use of HTML5 as an open web platform for application development.  The web is the one thing that binds together client facing compute devices, whether it’s your phone, watch, glasses or soon, an implant, he said. This common platform can help developers easily code across a diverse ecosystem of mobile devices. 

“Why have a native application that provides the same information (as a website)?” Hohndel said. “I want a platform… that’s easy for the developer.. and (provides) an application-like experience for the end user.” 

To accomplish this, developers must shift their focus from designing for compute to making applications power and performance aware, he said. Intel is working with the W3C to create standards that allow access to platform features like bluetooth, camera, and audio that are traditionally hard to get to from Javascript and HTML.

“The measure of success will be web applications that run like native applications,” Hohndel said. 

Red Hat/ Gluster

Total victory has already been achieved by the open source software community, Red Hat’s Walker argued. No new piece of proprietary software is likely to gain ubiquity these days, he said, Open source is where innovation is happening. 

The next step for open source victory, then, is in the data center. Ironically, “the new lock-in,” he said, is proprietary services and data sitting on top of open source software. The old argument against using proprietary software is that you wouldn’t buy a car with the hood welded shut. Similarly, in the cloud the question is “would you drive on a road system that didn’t let you choose your route?” 

Walker proposed five tests of an open hybrid storage system:

1. A single namespace. Your data appears the same regardless of context or platform, and it’s presented the same way regardless of the application presenting it.

2. Unified protocols. The ability to store data regardless of format and the ability to access it by a variety of protocols.

3. Elasticity across platforms. It expands where and when you need it.

4. Storage-aware applications. They work with compute to decide where data can go.

5. Open source software-led. There’s no other proprietary mix that can acheive the same level of agility.