OpenStack Liberty Design Summit

Sandvine is a telco network equipment vendor supplying 100+Gbps devices on custom hardware. As we are retooling for NFV, we have adopted OpenStack. In order to maximise our return from it, we decided to 'dogfood' it, and have rolled out a private cloud to our staff, partners, and lead customers, and are working at replacing our current lab 'pxe-boot' + 'vlan setup' automation with Openstack and Heat. This has become our reference platform for NFV.

Along the way we ran into a number of problems and challenges. In this talk I will describe some of the challenges we hit, and how we overcame them. I will also go through some of the adoption cycle problems, and how we got the various groups to align from 'virtualisation is bad' to 'I want more OpenStack' with strong champions.

Some of the specific challenges and solutions we implemented involve:

* Using layer-2 transparent networking in Neutron

* Having many more users and instances than public IPv4 addresses, a method to make this transparent via VPN + SSH

* Dealing with the rapid scale up associated with big groups adopting hard and fast

* Having very dynamic, short-lived instances, and the, problems that showed with race conditions in nova, and neutron

* Best practises for snapshots, and how we avoided snapshot proliferation with an 'auto-configuration' mechanism that stored changed in git

* Integration of keystone to Active Directory + Salesforce.com (for customer support and training usage)

* a desire to have an arbitrary remote machine, on remote network, join a heat-stack 

I will describe some of the best practises we came up with to accomodate a large and diverse community of users who showed a remarkable propensity to break things or find shortcomings.