OpenDaylight and OpenStack: Now With Helium!

This is just a quick post to note that the devstack support for OpenDaylight was recently updated to use the Helium release of OpenDaylight. For anyone who wants to pull down devstack and have it spin-up Neutron with OpenDaylight, you will now get the latest and greatest OpenDaylight release as a part of this. My blog post on how to use this is still relevant, so if you’re looking for instructions please look there.

Happy SDN’ing!

Peer Reviews for Neutron Core Reviewers

As I was recently given the chance to serve as Neutron PTL for a second cycle, I thought it would be a good idea for me to share some insight into what I’m hoping to achieve upstream in Kilo. I’ll have some upcoming posts on what we’re planning on accomplishing, but I wanted to first start with a post about the actual people who are allowed to merge code into Neutron, the core reviewers.

Core Reviewers

Neutron has 14 total core reviewers. You can see a list of these and also some notes around this on our wiki. Cores are responsible for reviewing code submissions submitted to Neutron’s gerrit, as well as merging those code submissions. They are also responsible for other things, most of which fall in a grey zone and aren’t documented that well. We’ll come back to this part in a bit.

The Neutron team has added a small handful of cores as the project has gone on, and we’ve also lost a small handful of people. But for the most part, once you’re a core, you remain a core forever. While this approach has served it’s purpose, there are issues with it.

Improving the Core Experience

OpenStack is always trying to improve things around not only code quality but also governance, so members of the Neutron community have taken it upon themselves to improve the process by which we understand a core’s responsibilities, and also a process under which we can judge how cores are performing up to that standard. The idea is to allow for the constant and consistent evaluation of existing Neutron cores. The long-term goal is to use the mechanism to also vet potential new cores. It’s an ambitious goal, and it takes into account more than just reviews, but also the grey zone aspects of being core which are hard to document.

This grey zone includes things such as community leadership, how you interact with the rest of the community, and participation in things like weekly meetings, IRC chats, and mailing list conversations. It includes mentoring of new contributors (something we haven’t recognized officially, but which happens). It also includes interactions with other OpenStack projects, and the leadership around being a liaison from Neutron into these other projects. It even includes evangelism for Neutron. All of these things are done by Neutron core team members.

Neutron Core Team Peer Review

The result of this has led us to begin to implement a Peer Review process for Neutron core team members. This is currently documented on an etherpad, and we’re in the process of collecting one more round of feedback. I’m highlighting this process here so people can provide input for this process. The goal is to keep this lightweight at first, and collect a good amount of actionable information to relay back to the existing core reviews. See the etherpad link for more info.

The end result of this is that we as a Neutron core team hope to better define what it means to be a Neutron core team member. We also hope to provide actionable feedback to existing members who may have strayed from this definition. A stronger Neutron core team benefits the entire OpenStack ecosystem. I’m proud of the work our Neutron core team is doing, and I hope we can continue to grow and evolve Neutron cores in the future by using this new Peer Review process.

What’s New in Neutron for OpenStack Juno

As of today, we just published the second Juno release candidate for Neutron. The expectation is this will be the final RC candidate and will become the official 2014.2 release of OpenStack Neutron. I thought I would take a moment to highlight some of the awesome work done by our community during the past 6 months.

Distributed Virtual Router

By far one of the largest, if not the largest, features we added as a team was the addition of Distributed Virtual Router (DVR) functionality. The team working on this has spent multiple cycles iterating on this code, and it finally landed in Juno. This is an exciting development because in prior versions of Neutron, the default L3 routing behavior was to send all traffic to L3 network nodes for routers. Clearly this presents issues around single points of failure on those L3 network nodes, not to mention the issue around having those L3 nodes become traffic bottlenecks. With DVR, routing functionality is distributed to each compute node, removing the need for a central L3 network node for this functionality. NAT functionality is also distributed in a similar manner. SNAT, however, is still centralized. The reason for this is due to the requirement of burning an IPV4 address on each compute node in a distributed SNAT environment. The wiki listed above has many more details on the DVR architecture.


The Juno release of Neutron will close the gaps on IPV6 support for tenant networks, allowing full IPV6 for tenant networks. We’ve added the capability for Neutron to manage the RADVD daemon to serve IPV6 RAs when required. For those looking to deploy Neutron in a pure IPV6 environment, you’ll find Juno allows for such deployments.

Security Group Enhancements

There are some well known issues around security group scaling with previous versions of OpenStack Neutron. In Juno, we’ve addressed these issues with two very important blueprints: The addition of ipset in lieu of iptables to manage security group rules on compute nodes, and the refactoring of the security_group_rules_for_devices RPC call. Both of these additions are meant to scale and dramatically improve the performance of the security groups implementations of Neutron.

L3 Agent Improvements

We made some serious performance improvements in the L3 agent during Juno, as well as added the capability to have HA for the L3 agent. Both of these will help with users deploying the Neutron L3 agent. The HA work in particular means you can now have redundancy for L3 agents. As I mentioned earlier, the L3 agent is still required for SNAT traffic for DVR. So this improvement means there is no longer a single point of failure for that traffic when using the DVR solution. For deployers who choose not to use DVR, now you can have HA for all your L3 routing and NAT traffic as well.


I’ve just highlighted some of the many improvements we’ve made in Juno for Neutron. For information on more features, including new plugins and drivers, checkout the release notes. We’re excited by the work the Neutron community has done for Juno, and we’re looking forward to an exciting development cycle for Kilo as well!

Trust the Upstream Community

Note: This post was co-authored by my good friend Thomas Graf.

My last post was around how to effectively contribute to an Open Source community. This post received a fair amount of traction, and I was happy with the conversation it created around Open Source contributions. There are plenty of people who hopefully benefited from this discussion.

Recently, I’ve been having some discussions with my friend Thomas Graf around upstream community engagement. Thomas recently shared with me the below, which are his words on how to engage and trust the upstream community. Thomas was kind enough to let me share these thoughts with everyone. I won’t summarize his words, but instead share them with you below.

1. Trust the upstream community

    Trust leads to openness. Openness leads to respecting all
    opinions. Respect leads to collaboration. Collaboration leads
    to give and take. The taking multiplies over time. Profit.

    Mistrust leads to mistrust.

    Before any interaction I ask myself: Would I say/do that to
    somebody I trust?

 2. Upstream does not care about the product roadmap

    Use cases and vision make up an excellent upstream motivation.
    Product roadmap driven agenda will drive contributors away.
    How I derive a downstream product from upstream is *my* problem,
    not the problem of the community.

    Open source is a long term investment into a sustainable model
    that if done right, can lead to maximum short term gains as well.

 3. Public discussions and public decision making

    Opening the circle of decision making makes a project attractive.
    Trust and respect whoever writes *good* code!

    Having all discussions on the public mailing list or other
    asynchronous communication channels allows new contributors to
    grow into the project by gradually spending more time on it and
    by reading up on previous discussions.

    Only exception: Classified internal material.

 4. Upstream reputation and community building is key

    Feature completeness, upstream merge, and packaging will come
    naturally, driven by magic community pixie dust, faster than
    anybody can imagine.

I agree with everything Thomas says above. Open Source is about transparency and trust. Due to the fact that many projects have disparate developers from not only different companies but also different cultures, locales, and skill levels, developing trust is important to the very essence of Open Source projects. Trust is both in the form of giving as well as receiving trust. Consider trust to be the currency of Open Source development.

I’d like to thank Thomas again for sharing these words with me, and I am grateful to be able to share them with you as well.

How to Effectively Contribute to An Open Source Project Such As OpenStack Neutron

Since being elected as the OpenStack Neutron PTL, I’ve been mostly heads down working to ensure the Neutron project has a successful Juno release. Increasingly, and especially near OpenStack Juno milestone deadlines, I’m seeing frustration from new contributors around their contributions to Neutron. I sent an email to the openstack-dev mailing list this morning addressing this in a terse form, this blog is an attempt to expand upon that email.

An increasing concern I see from people who are new contributors is the perceived issues in getting their code merged into Juno. The common concerns come from one of the following possible ideas:

  1. OpenStack Neutron is a “closed development” environment where only a select few are allowed to make changes.
  2. OpenStack Neutron is part of some grand conspiracy by vendor X, and as I work for vendor Y, my changes never get merged.
  3. OpenStack Neutron needs more core developers, this would solve all the velocity problems.

Addressing the above concerns in order, OpenStack Neutron is not a “closed development” environment. We do all of our work in the open by using mailing lists andIRC (for both meetings and discussions). Anyone can submit a patch to OpenStack Neutron once you’ve signed the CLA and followed the process here. We welcome new contributors and do our best to work with them! My advice for new contributors, echoed below, is to start small, listen and learn.

The second concern around conspiracy theories comes from people who are frustrated that their specification or blueprint isn’t prioritized for inclusion into a specific Juno release. They may also work for a competing company of some existing Neutron core developer, and think there is some political machinations going on here. While it’s true upstream developers work for corporations (we like to get paid and feed our families, it’s true!), it’s also true the core team in particular works together as a team. This means that people who work for Cisco do in fact work with people from VMware, as an example. We’re all working to drive stability and innovation upstream, but we do it in an open and collaborative manner.

The third concern is a tricky one. While adding more core reviewers would certainly help spread the load, we’re generally picky about adding new ones and have a policy in place around how we do this. The reason we’re picky is because being a core reviewer is a big responsibility, and we want dedicated developers who can contribute time upstream for the role. We’re in the process of coming up with a mentoring program to groom Neutron core developers, which will hopefully help here.

I’d like to reiterate from the email a list of effective ways to contribute to an upstream Open Source project:

  1. Get involved in the ways the project uses to communicate, including IRC, mailing lists, Hangouts, phone calls, whatever they use.
  2. Work upstream to fix some bugs. Documentation bugs and low-hanging fruit bugs are a good place to start.
  3. Understand how the existing team works. It’s best to understand this before suggesting any changes.
  4. Come to weekly meetings. Make sure you spend a little while listening to understand how the existing team works before jumping in with your own agenda.
  5. Build relationships. Doing the 4 points above will naturally lead you to this point.

An Open Source project is really a large development project being done in the open. To effectively join this project, you need to learn how the project operates and contribute in small ways initially. As in close sourced development, gaining the trust of the existing contributors is key to having influence over the direction of the project. The existing developers for a project such as Neutron (both core and non-core developers) all spend varying amount of time upstream working. But the key point is that they have taken the time to develop their contributions as well as grow relationships with existing upstream developers.

Upstream effectiveness is really about the time you put into your contributions. As existing developers upstream, we’re responsible for the current and future state of a project like OpenStack Neutron. While we have policies in place dictating how we work, we’re also responsible for things like gate failures, bug regressions, packaging issues, broad community interaction, and core developer grooming, among other things.

This post isn’t meant to discourage anyone from being an upstream developer. Quite the contrary, actually. I’m hoping to highlight effective ways to work upstream, which is beneficial for both new contributors as well as existing contributors. At the end of the day, a project like OpenStack Neutron involves people from across a wide spectrum of companies and interests. As the PTL, it’s my job to lead these people to a common goal. Growing the base of contributors is part of this responsibility as well. I look forward to seeing new contributors in Neutron and OpenStack in general!

ONUG 2014: My impressions

I was asked to present a workshop at the Spring 2014 Open Networking Users Group (ONUG)  this past Monday. The workshop was focused on OpenStack and OpenDaylight. My slides are embedded below. The room I was given held about 40 people, but it was overflowing with attendees and was standing room only. It was great to see all of the interest in these two technologies! I’m a firm believer in Open Source, so seeing this level of attendance is always a welcome sight.

I was very happy with the questions asked by attendees as I went over my material. There is a healthy interest by the enterprise in deploying OpenStack and finding a way to scale their networking into a cloud environment. One very interesting data point, which should be no surprise, is the fact enterprise customers have requirements and use cases which are vastly different from the typical service provider users of OpenStack today. Since the OpenStack Summit is next week, I look forward to spending some time talking with both SP and enterprise deployers next week to see how we can provide a robust IaaS platform for both sets of use cases.

Another interesting discussion point was around the work being done in the community around Group Based Policy and OpFlex. Interest in learning more about both technologies was very high, and a good number of questions were asked. Both GBP and OpFlex are being worked in the open as part of both OpenStack (GBP) and OpenDaylight (GBP and OpFlex). OpFlex was recently accepted as an incubated project into OpenDaylight, and work is ramping up. GBP has been incubated in ODL since January. And on the Neutron side, GBP has been actively developed since the Hong Kong Summit last fall. Expect big things from both projects in the coming months.

ONUG was a great event. Open Networking is clearly the future, and Open Source projects including OpenStack Neutron and OpenDaylight are a huge part of this future. I look forward to attending and presenting at future ONUG events!


OpFlex Is Not An OpenFlow Killer

There has been a flurry of press around Cisco’s release of OpFlex. If you want the nitty gritty details, please read the IETF draft available here. What exactly is OpFlex? The IETF draft sums it up nicely:

The OpFlex architecture provides a distributed control system based on a declarative policy information model. The policies are defined at a logically centralized policy repository (PR) and enforced within a set of distributed policy elements (PE). The PR communicates with the subordinate PEs using the OpFlex Control protocol. This protocol allows for bidirectional communication of policy, events, statistics, and faults.

It’s clear that Cisco intends to make OpFlex an open standard together with it’s partners in the vendor, provider and Open Source communities. We’re working hard to make that a reality. On the Open Source front, I’m leading a team of people who are working hard on the code around this new OpFlex Policy Agent. We intend to fully Open Source the OpFlex Policy Agent under the Apache 2.0 license. We’re excited to build an Open Community around this work, and in the coming months we’ll be looking to get more companies and individuals involved as we move this out into the Open. As you can imagine, starting an Open Source project from scratch takes time and planning, and we’re doing our best on all of these fronts.

What Exactly Is OpFlex?

So, now that you know what OpFlex is, what exactly is it not? Well, if you’ve read the article here, you may think it’s a, quote, “OpenFlow Killer.” Well a headline such as this may make for good SEO, it’s not true. OpFlex is meant to be embedded in the device or host via the Policy Agent. This could be a virtual switch such as Open vSwitch, a whitebox switch, a load balancer, a firewall, or any other element which can enforce policy. To show how this all fits together, the following diagram can be used:

|              |                                
|  Policy      |                                
|  Authority   |                                
|              |                                
|              |                                
|              |                                
       | Policy                                 
       | Messaging                              
+––––––+–––––––+                   +––––––––––––––+
|              |                   |              |
| OpFlex       |                   |  Host or     |
| Policy       +––––––––––––––––---+  Device      |
| Agent        |  Device           |              |
|              |  Programmability  |              |
|              |  Functions        |              |
+––––––––––––––+                   +––––––––––––––+

As you can see from the diagram, the key pieces of OpFlex are the Policy Authority and the Policy Agent. Policies defined in the Policy Authority are resolved asynchronously by the Policy Agent. The policies are then rendered in the system specific implementation into the programmability functions provided by the device or system. Examples of programmability functions could be the OpenFlow and OVSDB protocols supported by Open vSwitch, APIs provided by Arista’s EOS, the onePK APIs supported on Cisco gear, or even a CLI interface to a firewall device. OpFlex doesn’t replace the programmability mechanisms provided by the system, it works in tandem with them to enforce the policy. This is the key piece which many have missed.

In Conclusion

We’re excited about the potential of OpFlex and the Policy Agent. OpFlex is not meant to replace Open vSwitch, nor any other host or system programmability layer. Open vSwitch is a great Open Source project for which Cisco is a contributor. We plan to continue working closely with the Open vSwitch community, as well as projects such as OpenCompute, OpenDaylight, and the ONF. And just as these are all vibrant Open Source projects and communities, we hope to get the OpFlex Policy Agent into a similar state over the coming months and build a community for people who want to enable Open Source policy.

Workaround for ODL in Neutron

With the Icehouse release of Neutron impending, we’ve unfortunately uncovered a bug which is affecting ODL integration with Neutron. This bug was introduced by this commit, and the reality is better CI for the ODL plugin would have caught this. I’m going to work to enable this better CI in the near future. The workaround for this is to add the following in your nova.conf:

vif_plugging_timeout = 10
vif_plugging_is_fatal = False

I’m working on fixing this right now. Implementing the fix above in your nova.conf will get you running again with ODL and OpenStack Neutron. Thanks to Simon Pasquire for finding this bug and noting the workaround as well!

Engineering Artifacts Themselves Are No Longer the Source of Sustainable Advantage and/or Innovation

My good friend Dave Meyer just wrote a great blog post at SDN Central available here. A key point which Dave makes is this:

Now, if we step back and ask what is implied by these three observations, you begin to see an important and profound macro trend: Engineering artifacts themselves are no longer the source of sustainable advantage and/or innovation. Rather, sustainable advantage is achieved through engineering systems, organizational culture, and the people and process that comprise the community (and/or organization). Open-source community, code, and associated engineering systems are coming together in a way that is fundamentally transforming the network industry.

Dave is spot on with his analysis here. We need to stop thinking about artifacts as being only things such as installable software, firmware images, or even pieces of hardware. In the Open Source world, the community is an artifact. Vibrant mailing lists filled with discussion are artifacts. The tool chains are artifacts. The CI/CD systems become artifacts. All of these things become equally as important as the final result, because they each can substantially affect the final result, and without them working in harmony, you can’t get the final result.

When you grasp this concept, it really blows your mind.

OpenDaylight Integration with OpenStack has merged into Icehouse!

As OpenStack marches towards it’s Icehouse release this spring, some work I’ve been doing has finally merged upstream. This week, both the OpenDaylight ML2 MechanismDriver and devstack support for OpenDaylight merged upstream. This was a huge effort which spans the efforts of many people. This was the first step in solidifying the integration of OpenDaylight with OpenStack Neutron, and we have many additional things we can do. To get a first taste of running the two together, please see the video of the OpenDaylight Summit presentation myself, Madhu Venugopal, and Brent Salisbury did in early February.

Taking OpenDaylight For a Test Run With OpenStack Neutron

Now that the patches have merged upstream, trying this out is extremely simple. If you’re running a single node, you can simple setup the Neutron portion of your local.conf as follows and OpenDaylight will be downloaded and run as a top-level devstack service:

enable_service odl-server odl-compute

That’s all that’s required. When you run “”, you will see a screen called “odl-server” which will be OpenDaylight. And Neutron will use OpenDaylight to satisfy the requirements of virtual tenant networks.

If you’re trying this with a multi-node setup, you can use the above for your controller. And on your compute nodes, try this addition to local.conf:

enable_service odl-compute

That will configure the host to use Open vSwitch and set it up to point at OpenDaylight. It will also ensure Nova is setup to use Neutron for networking API calls.

Running OpenDaylight Outside of devstack

If you’re running OpenDaylight outside of devstack, you can configure your control node like this:

enable_service odl-compute

Just replace x.x.x.x in the ODL_MGR_IP line with the IP of your running OpenDaylight instance.

For compute hosts, all you need to add is this:

enable_service odl-compute

Again, just replace x.x.x.x in the ODL_MGR_IP line with the IP of your running OpenDaylight instance. Your control and compute nodes will now utilize an external OpenDaylight controller.

Additional Configuration Values

There are some additional things you can configure for OpenDaylight. They are:

  • ODL_ARGS: This value can be set to options to pass OpenDaylight. The default is “-XX:MaxPermSize=384m”. An example would be like this:ODL_ARGS=”-Xmx1024m -XX:MaxPermSize=512m”
  • ODL_BOOT_WAIT: This value indicates how long to sleep after starting OpenDaylight before proceeding with the rest of devstack. The default value is 60 seconds. An example is like this:

The Future of Open Source SDN

OpenDaylight is progressing at a very fast pace. The current release being worked on (Helium) is going to stabilize and scale the platform even more. With features like Group Policy being added, the future looks increasingly awesome for OpenDaylight. And now you can use it to scale your Neutron networks as well. How cool is that?