Extending a flatdhcp network the hard way

The title may make you think there’s an easy way. No such luck. Nova has no facility for extending a flatdhcp network, and as far as I can tell Quantum also has no facility for doing so.

Extending the flatdhcp network can be kind of a pain in the ass, so here’s how I handled it:

Assumptions

  • Network before extension:
    • Network CIDR: 10.4.0.0/24
    • Broadcast: 10.4.0.255
    • Netmask: 255.255.255.0
    • Network ID: 2
  • Network after extension:
    • Network CIDR: 10.4.0.0/21
    • Broadcast: 10.4.7.255
    • Netmask: 255.255.248.0
    • Network ID: 2

Modify the network

First modify the network via the database:

mysql nova -e "UPDATE networks SET netmask=\"255.255.248.0\",cidr=\"10.4.0.0/21\",broadcast=\"10.4.7.255\" WHERE id=2;"

Add the fixed IPs

Now it’s necessary to add all of the IP addresses in the range into the fixed_ips table. Additionally, the broadcast address in the original range should be modified so that it’s no longer reserved, and the new broadcast address should be marked as reserved.

for i in {1..7}
do
    for j in {0..255}
    do
        mysql nova -e "INSERT INTO fixed_ips SET created_at=\"2012-10-01 19:24:21\",updated_at=\"2012-10-01 19:24:21\",deleted=0,address=\"10.4.${i}.${j}\",network_id=2,allocated=0,reserved=0,leased=0"
    done
done
mysql nova -e "UPDATE fixed_ips SET reserved=0 WHERE address=\"10.4.0.255\""
mysql nova -e "UPDATE fixed_ips SET reserved=1 WHERE address=\"10.4.7.255\""

Restart nova-network and nova-compute services

I tried launching some instances after making this change, and got the following error popping up in my logs:

2012-10-01 20:04:48 TRACE nova.rpc.amqp DetachedInstanceError: Parent instance <Instance at 0x46fa150> is not bound to a Session; lazy load operation of attribute 'instance_type' cannot proceed

In the #openstack channel, zynzel mentioned that it’s because I needed to restart my nova-network service. Actually, I needed to restart all nova-network and all nova-compute services.

I ran into a likely unrelated issue during this as well. My nova-compute services were deadlocked. I’ve actually noticed this in the past as well. Clearing the lock files from /var/lock/nova then restarting the services fixed that issue, though I still need to trace this issue down.

Remove the old gateway addresses

The old gateway addresses, with the /24 CIDR, need to be removed from the bridge and from the routing table on the network nodes.

Restart dnsmasq and nova-network

After removing the addresses, it’s necessary to restart dnsmasq. Kill the processes, then restart nova-network again.

Why doesn’t Nova and Quantum have this functionality?

Neither Nova, nor Quantum seem to have operations to modify a network. It’s not a completely abnormal task to need to extend a network, to re-vlan it, or to change IP ranges. Hell, I still need to add IPv6 to my network and I need to make it multi-network-node; when I created the network neither of these features existed and there’s no way simple way to enable them now.

You can delete and re-create a network, but what happens to IP address assignments? What happens to DNS entries that were created via the DNS plugin for Nova? We really need the ability to modify networks, not just create and delete them.

A user-unfriendly experience

The above steps don’t really seem very difficult, but the actual steps involved assume fairly involved knowledge of the code. When in the middle of doing this, things are stressful and things are failing. It’s pretty user-unfriendly.

Additionally, when I asked about this in the #openstack channel, I was treated like I was stupid for not wanting to muck around with the database. I was told that it was my fault for creating a network that was too small and that it isn’t Nova’s job to fix my mistakes. I was told that I should “use Windows”; meaning that I’m expecting the software to hold my noob-ish hand.

I can muck in the database and trace code, but I think if users are forced to do that then we’re failing from a usability perspective.

5 Comments

  1. Both melange and the quantumv2 api support multiple subnets on the same L2 segment. Unfortunately in the legacy nova networks system, you have to muck with the db due to its design, and in quantumv2 you are limited to what the plugin you run supports.

    But assuming you’re running a quantumv2 plugin that supports it, all you have to do is add 3 more /24’s (10.4.1.0/24, 10.4.2.0/24, and 10.4.3.0/24) and 1 /22 (10.4.4.0/22), this does have the side effect that 5 total instances of dnsmasq will be running if you’re using the quantum dhcp agent.

    In our installation, we use flat only (routing is handled by dedicated hardware) and static assignment in vm’s though the agent (or as some call it rootkit ;), and routinely add/expand networks (although though melange since we deployed prior to any of the quantumv2 work being started).

    Reply

    1. That’s great to hear! Would we be able to have those subnets on different vlans?

      Reply

      1. Different vlans would require different l2 networks. But you will/can be able to specify what l2 network to attach to on server build depending on what network manager/network api is running. Much of this will be improved in Grizzly to make it easier to have custom networks on a per instance basis.

        Reply

        1. I’m eagerly awaiting my Quantum overlords. I’m glad to hear this is all being worked out!

          Reply

  2. […] By Ryan Lane: Extending a flatdhcp network the hard way […]

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>