Trust the Upstream Community

Note: This post was co-authored by my good friend Thomas Graf.

My last post was around how to effectively contribute to an Open Source community. This post received a fair amount of traction, and I was happy with the conversation it created around Open Source contributions. There are plenty of people who hopefully benefited from this discussion.

Recently, I’ve been having some discussions with my friend Thomas Graf around upstream community engagement. Thomas recently shared with me the below, which are his words on how to engage and trust the upstream community. Thomas was kind enough to let me share these thoughts with everyone. I won’t summarize his words, but instead share them with you below.

1. Trust the upstream community

    Trust leads to openness. Openness leads to respecting all
    opinions. Respect leads to collaboration. Collaboration leads
    to give and take. The taking multiplies over time. Profit.

    Mistrust leads to mistrust.

    Before any interaction I ask myself: Would I say/do that to
    somebody I trust?

 2. Upstream does not care about the product roadmap

    Use cases and vision make up an excellent upstream motivation.
    Product roadmap driven agenda will drive contributors away.
    How I derive a downstream product from upstream is *my* problem,
    not the problem of the community.

    Open source is a long term investment into a sustainable model
    that if done right, can lead to maximum short term gains as well.

 3. Public discussions and public decision making

    Opening the circle of decision making makes a project attractive.
    Trust and respect whoever writes *good* code!

    Having all discussions on the public mailing list or other
    asynchronous communication channels allows new contributors to
    grow into the project by gradually spending more time on it and
    by reading up on previous discussions.

    Only exception: Classified internal material.

 4. Upstream reputation and community building is key

    Feature completeness, upstream merge, and packaging will come
    naturally, driven by magic community pixie dust, faster than
    anybody can imagine.

I agree with everything Thomas says above. Open Source is about transparency and trust. Due to the fact that many projects have disparate developers from not only different companies but also different cultures, locales, and skill levels, developing trust is important to the very essence of Open Source projects. Trust is both in the form of giving as well as receiving trust. Consider trust to be the currency of Open Source development.

I’d like to thank Thomas again for sharing these words with me, and I am grateful to be able to share them with you as well.

How to Effectively Contribute to An Open Source Project Such As OpenStack Neutron

Since being elected as the OpenStack Neutron PTL, I’ve been mostly heads down working to ensure the Neutron project has a successful Juno release. Increasingly, and especially near OpenStack Juno milestone deadlines, I’m seeing frustration from new contributors around their contributions to Neutron. I sent an email to the openstack-dev mailing list this morning addressing this in a terse form, this blog is an attempt to expand upon that email.

An increasing concern I see from people who are new contributors is the perceived issues in getting their code merged into Juno. The common concerns come from one of the following possible ideas:

  1. OpenStack Neutron is a “closed development” environment where only a select few are allowed to make changes.
  2. OpenStack Neutron is part of some grand conspiracy by vendor X, and as I work for vendor Y, my changes never get merged.
  3. OpenStack Neutron needs more core developers, this would solve all the velocity problems.

Addressing the above concerns in order, OpenStack Neutron is not a “closed development” environment. We do all of our work in the open by using mailing lists andIRC (for both meetings and discussions). Anyone can submit a patch to OpenStack Neutron once you’ve signed the CLA and followed the process here. We welcome new contributors and do our best to work with them! My advice for new contributors, echoed below, is to start small, listen and learn.

The second concern around conspiracy theories comes from people who are frustrated that their specification or blueprint isn’t prioritized for inclusion into a specific Juno release. They may also work for a competing company of some existing Neutron core developer, and think there is some political machinations going on here. While it’s true upstream developers work for corporations (we like to get paid and feed our families, it’s true!), it’s also true the core team in particular works together as a team. This means that people who work for Cisco do in fact work with people from VMware, as an example. We’re all working to drive stability and innovation upstream, but we do it in an open and collaborative manner.

The third concern is a tricky one. While adding more core reviewers would certainly help spread the load, we’re generally picky about adding new ones and have a policy in place around how we do this. The reason we’re picky is because being a core reviewer is a big responsibility, and we want dedicated developers who can contribute time upstream for the role. We’re in the process of coming up with a mentoring program to groom Neutron core developers, which will hopefully help here.

I’d like to reiterate from the email a list of effective ways to contribute to an upstream Open Source project:

  1. Get involved in the ways the project uses to communicate, including IRC, mailing lists, Hangouts, phone calls, whatever they use.
  2. Work upstream to fix some bugs. Documentation bugs and low-hanging fruit bugs are a good place to start.
  3. Understand how the existing team works. It’s best to understand this before suggesting any changes.
  4. Come to weekly meetings. Make sure you spend a little while listening to understand how the existing team works before jumping in with your own agenda.
  5. Build relationships. Doing the 4 points above will naturally lead you to this point.

An Open Source project is really a large development project being done in the open. To effectively join this project, you need to learn how the project operates and contribute in small ways initially. As in close sourced development, gaining the trust of the existing contributors is key to having influence over the direction of the project. The existing developers for a project such as Neutron (both core and non-core developers) all spend varying amount of time upstream working. But the key point is that they have taken the time to develop their contributions as well as grow relationships with existing upstream developers.

Upstream effectiveness is really about the time you put into your contributions. As existing developers upstream, we’re responsible for the current and future state of a project like OpenStack Neutron. While we have policies in place dictating how we work, we’re also responsible for things like gate failures, bug regressions, packaging issues, broad community interaction, and core developer grooming, among other things.

This post isn’t meant to discourage anyone from being an upstream developer. Quite the contrary, actually. I’m hoping to highlight effective ways to work upstream, which is beneficial for both new contributors as well as existing contributors. At the end of the day, a project like OpenStack Neutron involves people from across a wide spectrum of companies and interests. As the PTL, it’s my job to lead these people to a common goal. Growing the base of contributors is part of this responsibility as well. I look forward to seeing new contributors in Neutron and OpenStack in general!

ONUG 2014: My impressions

I was asked to present a workshop at the Spring 2014 Open Networking Users Group (ONUG)  this past Monday. The workshop was focused on OpenStack and OpenDaylight. My slides are embedded below. The room I was given held about 40 people, but it was overflowing with attendees and was standing room only. It was great to see all of the interest in these two technologies! I’m a firm believer in Open Source, so seeing this level of attendance is always a welcome sight.

I was very happy with the questions asked by attendees as I went over my material. There is a healthy interest by the enterprise in deploying OpenStack and finding a way to scale their networking into a cloud environment. One very interesting data point, which should be no surprise, is the fact enterprise customers have requirements and use cases which are vastly different from the typical service provider users of OpenStack today. Since the OpenStack Summit is next week, I look forward to spending some time talking with both SP and enterprise deployers next week to see how we can provide a robust IaaS platform for both sets of use cases.

Another interesting discussion point was around the work being done in the community around Group Based Policy and OpFlex. Interest in learning more about both technologies was very high, and a good number of questions were asked. Both GBP and OpFlex are being worked in the open as part of both OpenStack (GBP) and OpenDaylight (GBP and OpFlex). OpFlex was recently accepted as an incubated project into OpenDaylight, and work is ramping up. GBP has been incubated in ODL since January. And on the Neutron side, GBP has been actively developed since the Hong Kong Summit last fall. Expect big things from both projects in the coming months.

ONUG was a great event. Open Networking is clearly the future, and Open Source projects including OpenStack Neutron and OpenDaylight are a huge part of this future. I look forward to attending and presenting at future ONUG events!


OpFlex Is Not An OpenFlow Killer

There has been a flurry of press around Cisco’s release of OpFlex. If you want the nitty gritty details, please read the IETF draft available here. What exactly is OpFlex? The IETF draft sums it up nicely:

The OpFlex architecture provides a distributed control system based on a declarative policy information model. The policies are defined at a logically centralized policy repository (PR) and enforced within a set of distributed policy elements (PE). The PR communicates with the subordinate PEs using the OpFlex Control protocol. This protocol allows for bidirectional communication of policy, events, statistics, and faults.

It’s clear that Cisco intends to make OpFlex an open standard together with it’s partners in the vendor, provider and Open Source communities. We’re working hard to make that a reality. On the Open Source front, I’m leading a team of people who are working hard on the code around this new OpFlex Policy Agent. We intend to fully Open Source the OpFlex Policy Agent under the Apache 2.0 license. We’re excited to build an Open Community around this work, and in the coming months we’ll be looking to get more companies and individuals involved as we move this out into the Open. As you can imagine, starting an Open Source project from scratch takes time and planning, and we’re doing our best on all of these fronts.

What Exactly Is OpFlex?

So, now that you know what OpFlex is, what exactly is it not? Well, if you’ve read the article here, you may think it’s a, quote, “OpenFlow Killer.” Well a headline such as this may make for good SEO, it’s not true. OpFlex is meant to be embedded in the device or host via the Policy Agent. This could be a virtual switch such as Open vSwitch, a whitebox switch, a load balancer, a firewall, or any other element which can enforce policy. To show how this all fits together, the following diagram can be used:

|              |                                
|  Policy      |                                
|  Authority   |                                
|              |                                
|              |                                
|              |                                
       | Policy                                 
       | Messaging                              
+––––––+–––––––+                   +––––––––––––––+
|              |                   |              |
| OpFlex       |                   |  Host or     |
| Policy       +––––––––––––––––---+  Device      |
| Agent        |  Device           |              |
|              |  Programmability  |              |
|              |  Functions        |              |
+––––––––––––––+                   +––––––––––––––+

As you can see from the diagram, the key pieces of OpFlex are the Policy Authority and the Policy Agent. Policies defined in the Policy Authority are resolved asynchronously by the Policy Agent. The policies are then rendered in the system specific implementation into the programmability functions provided by the device or system. Examples of programmability functions could be the OpenFlow and OVSDB protocols supported by Open vSwitch, APIs provided by Arista’s EOS, the onePK APIs supported on Cisco gear, or even a CLI interface to a firewall device. OpFlex doesn’t replace the programmability mechanisms provided by the system, it works in tandem with them to enforce the policy. This is the key piece which many have missed.

In Conclusion

We’re excited about the potential of OpFlex and the Policy Agent. OpFlex is not meant to replace Open vSwitch, nor any other host or system programmability layer. Open vSwitch is a great Open Source project for which Cisco is a contributor. We plan to continue working closely with the Open vSwitch community, as well as projects such as OpenCompute, OpenDaylight, and the ONF. And just as these are all vibrant Open Source projects and communities, we hope to get the OpFlex Policy Agent into a similar state over the coming months and build a community for people who want to enable Open Source policy.

Workaround for ODL in Neutron

With the Icehouse release of Neutron impending, we’ve unfortunately uncovered a bug which is affecting ODL integration with Neutron. This bug was introduced by this commit, and the reality is better CI for the ODL plugin would have caught this. I’m going to work to enable this better CI in the near future. The workaround for this is to add the following in your nova.conf:

vif_plugging_timeout = 10
vif_plugging_is_fatal = False

I’m working on fixing this right now. Implementing the fix above in your nova.conf will get you running again with ODL and OpenStack Neutron. Thanks to Simon Pasquire for finding this bug and noting the workaround as well!

Engineering Artifacts Themselves Are No Longer the Source of Sustainable Advantage and/or Innovation

My good friend Dave Meyer just wrote a great blog post at SDN Central available here. A key point which Dave makes is this:

Now, if we step back and ask what is implied by these three observations, you begin to see an important and profound macro trend: Engineering artifacts themselves are no longer the source of sustainable advantage and/or innovation. Rather, sustainable advantage is achieved through engineering systems, organizational culture, and the people and process that comprise the community (and/or organization). Open-source community, code, and associated engineering systems are coming together in a way that is fundamentally transforming the network industry.

Dave is spot on with his analysis here. We need to stop thinking about artifacts as being only things such as installable software, firmware images, or even pieces of hardware. In the Open Source world, the community is an artifact. Vibrant mailing lists filled with discussion are artifacts. The tool chains are artifacts. The CI/CD systems become artifacts. All of these things become equally as important as the final result, because they each can substantially affect the final result, and without them working in harmony, you can’t get the final result.

When you grasp this concept, it really blows your mind.

OpenDaylight Integration with OpenStack has merged into Icehouse!

As OpenStack marches towards it’s Icehouse release this spring, some work I’ve been doing has finally merged upstream. This week, both the OpenDaylight ML2 MechanismDriver and devstack support for OpenDaylight merged upstream. This was a huge effort which spans the efforts of many people. This was the first step in solidifying the integration of OpenDaylight with OpenStack Neutron, and we have many additional things we can do. To get a first taste of running the two together, please see the video of the OpenDaylight Summit presentation myself, Madhu Venugopal, and Brent Salisbury did in early February.

Taking OpenDaylight For a Test Run With OpenStack Neutron

Now that the patches have merged upstream, trying this out is extremely simple. If you’re running a single node, you can simple setup the Neutron portion of your local.conf as follows and OpenDaylight will be downloaded and run as a top-level devstack service:

enable_service odl-server odl-compute

That’s all that’s required. When you run “”, you will see a screen called “odl-server” which will be OpenDaylight. And Neutron will use OpenDaylight to satisfy the requirements of virtual tenant networks.

If you’re trying this with a multi-node setup, you can use the above for your controller. And on your compute nodes, try this addition to local.conf:

enable_service odl-compute

That will configure the host to use Open vSwitch and set it up to point at OpenDaylight. It will also ensure Nova is setup to use Neutron for networking API calls.

Running OpenDaylight Outside of devstack

If you’re running OpenDaylight outside of devstack, you can configure your control node like this:

enable_service odl-compute

Just replace x.x.x.x in the ODL_MGR_IP line with the IP of your running OpenDaylight instance.

For compute hosts, all you need to add is this:

enable_service odl-compute

Again, just replace x.x.x.x in the ODL_MGR_IP line with the IP of your running OpenDaylight instance. Your control and compute nodes will now utilize an external OpenDaylight controller.

Additional Configuration Values

There are some additional things you can configure for OpenDaylight. They are:

  • ODL_ARGS: This value can be set to options to pass OpenDaylight. The default is “-XX:MaxPermSize=384m”. An example would be like this:ODL_ARGS=”-Xmx1024m -XX:MaxPermSize=512m”
  • ODL_BOOT_WAIT: This value indicates how long to sleep after starting OpenDaylight before proceeding with the rest of devstack. The default value is 60 seconds. An example is like this:

The Future of Open Source SDN

OpenDaylight is progressing at a very fast pace. The current release being worked on (Helium) is going to stabilize and scale the platform even more. With features like Group Policy being added, the future looks increasingly awesome for OpenDaylight. And now you can use it to scale your Neutron networks as well. How cool is that?

Vote for my OpenStack Talks!

I am very lucky to be a part of six great presentation submissions for the upcoming OpenStack Summit in Atlanta. The OpenStack Foundation uses voting to help decide which of these talks, panels, and tutorials will be scheduled. I would appreciate your vote for my submissions! I’ll highlight them below.

  • Using OpenStack Within An OpenStack Environment: This is a talk which will be similar to the tutorial Madhu, Brent, Ryan and I did at the OpenDaylight Summit, except it will be less tutorial focused and more presentation based. We’ll show you how to use the latest OpenDaylight Hydrogen release with OpenStack Neutron. I’m hoping to add Madhu and Brent to present alongside Dave Meyer and myself.
  • OpenStack Integration with OpenDaylight: State of the Union and Future Directions: In this talk, Dave Meyer and I will go over OpenDaylight for people who are new to this Open Source SDN controller. We’ll cover a bit of history and go over what’s supported in the Hydrogen release. And we’ll cover a bit about the future direction. This session will be great for people new to OpenDaylight.
  • The Future of OpenStack Networking: This is a panel discussion we put together to discuss future directions of OpenStack Networking. Some great people are proposed on this panel with me: Lew Tucker, who leads Cloud Computing at Cisco and is one of the sharpest minds in the industry; Dan Dumitriu, CEO of Midokura and a trailblazer in the SDN industry; Chris Wright, Technical Director of SDN at Red Hat, and one of the best known names in Open Source networking; and Nils Swart from Plexxi, who is leading their Open Source strategy. A great group of people, this will be a fun panel if it’s accepted.
  • Network Policy Abstractions in Neutron: There is work going on in Neutron to add a new Policy API layer here. This will make it easier for application developers and deployers to work with the networking subsystem in OpenStack. This is a talk which will go into details on this. I’m happy to be joined by Mohammad Banikazemi from IBM and Stephen Wong from Midokura. Anyone interested in advanced policy discussions will want to attend this discussion should it be approved.
  • An Overview of Open Source Backends for Neutron: This talk will walk people through the options available in the Open Source world for OpenStack Neutron networking. Salvatore Orlando, who is one of the original Neutron core team members, will be my co-presenter, should this talk get approved. This talk would be great for people looking at deployment options which are pure open source.
  • Hackathon Syllabus: This is a great talk submitted by Sean Roberts from Yahoo!, and we plan to cover how to hold a Hackathon Style user group meeting. We’ll be joined by Colin McNamara from Nexus IS and Mark Voelker from Cisco. This will be great for people who are looking at forming User Groups focused on hacking!

Please consider any or all of my talks above. I appreciate your votes and look forward to continuing to share my knowledge around Open Source networking!

TimeMachine on a QNAP TS-659

For the last few months, TimeMachine has been failing for me between my Macs and a QNAP TS-659 NAS I have. The NAS is running firmware 4.0.3, and I would consistently get the error “Cannot connect” when trying to connect. I was able to work around this with the following command run manually:

sudo tmutil setdestination -ap afp://TimeMachine@

Running that in a terminal window allowed me to then use the GUI to select that disk for backups and things started rolling again. In the above, make sure to replace “” with your IP or hostname of the QNAP. It will ask you for your password when you run the command. I found all of this on the forum thread here for reference.

Getting Started With OpenDaylight and OpenStack

If you’re a fan of networking, you are no doubt very excited by all of the recent excitement in the industry as of late. And there is no larger area of innovation in networking at the moment than Open Source networking. Two of the projects at the forefront of Open Source networking innovation are OpenStack Neutron and OpenDaylight. OpenStack Neutron is driving an API around networking for Infrastructure as a Service Clouds, and has been very successful at driving mindshare in this area. There are a large number of Plugins and ML2 MechanismDrivers for Neutron in existence already. However, so far, there is no OpenDaylight integration with OpenStack, at least upstream. I am pleased to announce that a team of us are working on making this happen, however. We have a blueprint filed and we are actively working towards the support in OpenDaylight required to support the Neutron APIs. What I’m going to show you in this blog post is how to take what we currently have for a test run and try it out yourself.

OpenDaylight Integration with OpenStack: The Details

OpenDaylight is a highly scalable controller written in Java. It is designed from the start to be modular. Perhaps the best way to understand the Modular nature of OpenDaylight is to look at an architecture diagram of it:

OpenDaylight Hydrogen Release

OpenDaylight Hydrogen Release Architecture Diagram

You can see all the pieces of OpenDaylight, and there are quite a few. Because of the modular nature of OpenDaylight it makes heavy use of the OSGI framework. I’m not going to go into extreme details of how this works, but suffice to say it allows for anyone to write a bundle which can run and interact with other bundles in OpenDaylight.

As part of this, there exists a few bundles which are relevant to the OpenStack integration efforts:

Each of those bundles provides a necessary component in the OpenStack integration. The NeutronAPIService provides an abstraction of the Neutron APIs into OpenDaylight. It caches all of the Neutron objects inside of OpenDaylight providing access to this information to anything in OpenDaylight which requires it. The OVSDB and OpenFlow OSGI bundles in OpenDaylight provide the code which actually programs things on each compute host. They allow for the creation and deletion of tunnel ports, flow programming for ports as they come and go, and bridge creation and deletion on the host.

The main benefit of the above is that each compute host no longer needs an Open vSwitch Agent running on each host. The combination of OpenFlow and OVSDB provide the equivalent functionality as the Open vSwitch Agent.

OpenDaylight and OpenStack: Getting Started

To test out the latest OpenDaylight Modular Layer2 MechanismDriver, you will need the following:

  • A machine to run the OpenDaylight Controller
  • A machine to run the OpenStack control software
  • At least one machine to run the OpenStack Compute service to run virtual machines

Now, you can combine some of the things above, and you should most certainly run all of the above as virtual machines. I personally run all of the above as virtual machines on VMware Fusion and have one VM in which I run OpenDaylight, one VM in which I run the OpenStack control software, and 3 other VMs in which I run OpenStack compute services. A fairly minimum setup would be 3 VMs, however: One to run the OpenDaylight controller, one to run OpenStack control and compute services, and another one to run only OpenStack compute services.

In either case, your topology will look very similar to the following diagram:


OpenStack and OpenDaylight Integration

OpenDaylight and OpenStack: Building and Installing OpenDaylight

Lets get started with the actual configuration of the system now. The first piece is your OpenDaylight VM. To build and install this, follow the steps below. I should note a much larger view of building the controller is on the wiki page here, the instructions below are mostly meant to get you going very fast without having to read that wiki page in detail.

mkdir ~/odl
cd odl
git clone
cd opendaylight/distribution/opendaylight/
mvn clean install
cd ~/odl
git clone
cd ovsdb

At this point, you can cut and paste the script below as “” and use that to build OVSDB and copy the bundles over to the controller:

git pull
cd neutron
echo "Refreshing ovsdb/neutron.."
mvn clean install
cd ../northbound/ovsdb/
echo "Refreshing northbound/ovsdb.. "
mvn clean install
cd ../../ovsdb
echo "Refreshing ovsdb/ovsdb.."
mvn clean install
cd ..
cp ~/odl/ovsdb/neutron/target/ovsdb.neutron-0.5.0-SNAPSHOT.jar ~/odl/controller/opendaylight/distribution/opendaylight/target/distribution.opendaylight-osgipackage/opendaylight/plugins/
cp ~/odl/ovsdb/northbound/ovsdb/target/ovsdb.northbound-0.5.0-SNAPSHOT.jar ~/odl/controller/opendaylight/distribution/opendaylight/target/distribution.opendaylight-osgipackage/opendaylight/plugins/
cp ~/odl/ovsdb/ovsdb/target/ovsdb-0.5.0-SNAPSHOT.jar ~/odl/controller/opendaylight/distribution/opendaylight/target/distribution.opendaylight-osgipackage/opendaylight/plugins/
echo "done!"

Once you’ve created the script, simply make sure it has execute permissions (chmod +x and run it and you will have the OVSDB bundles created and installed into the plugins directory. To verify they are there, look in the following location:

  • odl/controller/opendaylight/distribution/opendaylight/target/distribution.opendaylight-osgipackage/opendaylight/plugins

The next step is to modify the “of.address” variable in the “configuration/config.ini” file. This file is relative to the odl/controller/opendaylight/distribution/opendaylight/target/distribution.opendaylight-osgipackage directiory. Fire up vi and add the management IP address for your ODL instance as the value for of.address.

Now it’s time to fire up your controller! To do that, execute the following:

cd ~/odl/controller/opendaylight/distribution/opendaylight/target/distribution.opendaylight-osgipackage/opendaylight

Once the controller is running, you will want to disable the SimpleForwarding Application, so do the following:

  • In the OSGI console, run “lb | grep simple” to find the bundle ID of the simpleforwarding application.
  • Run “stop <bundle ID>” to disable simpleforwarding.
  • Run “lb | grep simple” to verify it is in the “Resolved” state.

The entire thing looks like below:

osgi> lb | grep simple
 132|Active | 4|samples.simpleforwarding (0.4.1.SNAPSHOT)
 osgi> stop 132
 osgi> lb | grep simple
 132|Resolved | 4|samples.simpleforwarding (0.4.1.SNAPSHOT)

OpenStack and OpenDaylight: Readying the devstack nodes

At this point, you have an OpenDaylight controller running. Now it’s time to fire up your devstack nodes. You will need at least two virtual machines ready for this. They can run anything which devstack supports. I am ardent user of Fedora Linux, so that’s what I use, but Ubuntu works fine as well. Note if you’re using Ubuntu 12.04 LTS, that particular variant of Ubuntu is using OVS 1.4, which is quite a bit old. Fedora 19 uses a much newer version of OVS.

One thing to note is that you should make sure you have passwordless “sudo” access setup for the account you’re running devstack as.

So, the next thing to do on each node is to checkout devstack:

cd ~/
git clone git://
cd devstack
git remote add opendaylight
git fetch opendaylight
git checkout opendaylight

Run the above on each devstack node. It will checkout the customer OpenDaylight devstack branch. Now to configure your local.conf files.

On the control node, your local.conf will look like the below:


# Only uncomment the below two lines if you are running on Fedora
disable_service rabbit
enable_service qpid
disable_service n-cpu
enable_service n-cond
disable_service n-net
enable_service q-svc
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service quantum
enable_service tempest








You should note in the above you will want to change the following:

  • HOST_IP: This is the management IP of the control host itself.
  •  VNCSERVER_PROXYCLIENT_ADDRESS: The management IP address of the control node itself.
  • HOST_NAME: The host name of the control node.
  • SERVICE_HOST: The management IP of the control node.
  • The “url” parameter in the ml2_odl section near the bottom: Make sure the url and credentials match your OpenDaylight configuration. If you didn’t change the default username password for ODL, you can leave those bits alone.

Once you have that done, the next step is to setup your local.conf for the compute nodes:


enable_service neutron nova n-cpu quantum n-novnc qpid







Again, the parts to edit above on the compute nodes are:

  • HOST_NAME: The host name of each compute node.
  • HOST_IP: The management IP address of each host.
  • SERVICE_HOST_NAME: The hostname of the control node.
  • SERVICE_HOST: The management IP of the control node.
  • ml2_odl: Modify the IP address there for the ODL controller.

Each local.conf file should be saved in the ~/devstack directory on each control and/or compute host.

Now you should be able to run “” on all of the nodes (control and each compute) by doing this:

  • cd ~/devstack
  • ./

Once that completes, you should have a functioning OpenStack setup with OpenDaylight.

Possible Issues With devstack on Fedora

One possible issue you may hit if you’re using a fresh VM on Fedora is mysql errors. You will see keystone errors and mysql access errors in the run. To get around this, follow the workaround listed in this post here. It’s worked for me every time I hit this error running devstack on Fedora. One other issue with Fedora is that the latest devstack fails to kill all the nova processes when you run “” To workaround this, simply run the following after “”:

  • killall nova-api nova-cert nova-scheduler nova-consoleauth nova-api nova-conductor

OpenStack and OpenDaylight: Verifying The Install

At this point, you should have the entire system up and running. To verify this, you can do the following:

  • Point your web browser at the OpenStack Horizon GUI:
    • http://<control node IP>/auth/login/
    • Login using “admin/admin” and you can see you OpenStack install.
  • Point your web browser at the OpenDaylight GUI:
    • http://<odl IP>:8080/
    • Login using “admin/admin”

You can play around in the GUIs, launch VMs, etc. As you launch VMs, you will see ODL create tunnel ports and links between compute hosts, which will become visible with a refresh in the OpenDaylight GUI.

OpenStack and OpenDaylight: Getting Help

The most appropriate place to get help at this early stage is on #opendaylight-ovsdb on Freenode. A long list of OpenStack Neutron and OpenDaylight developers hang out there and can provide help. Besides myself (IRC nick “mestery”), you can also expect to find the following people online:

  • Madhu Venugopal (IRC Nick “Madhu”)
  • Brent Salisbury (IRC Nick “networkstatic”)
  • Keith Burns (IRC Nick “alagalah”)
  • Florian Otel (IRC Nick “FlorianOtel”)

You can ping any of us and we should be able to help you debug any issues. Florian in particular has some VM images which may expedite the process above for folks trying this out for the first time. The instructions above were meant to walk through all of the steps necessary to get this up and running from scratch.

OpenStack and OpenDaylight: What’s Next

In a future post, I will walk through debugging this setup and using it see flows and how different pieces interact. In particular, I’ll walk through debugging this system so you understand exactly how things are done when networks, subnets, ports, routers and other OpenStack Neutron API objects are created and how OpenDaylight handles programming them onto each host.