Configuration Management on SwitchesCumulus and the Salt Minion

We love Debian Linux. And we love Configuration Management with Salt. All good, you might think. Installing the Salt Minion on our switches running on Cumulus Linux as an operating system, though, turned out to be quite the brainteaser. But first things first.

Once upon a time, in a land where switches were made by Cisco or Juniper or another producer - you didn't have the choice when it came to operating systems, vendor lock-in being the operative word. Today, things have changed - and luckily so: so-called bare metal switches enable us to equip and to configure the bare, neutral hardware of equal quality with a suitable operating system ourselves.

Switch = Server

Debian-based Cumulus Linux is one of the best-known Linux distributions for the operation of bare metal switches. When it comes to servers, we mainly work with Debian, so using Cumulus Linux as a switch operating system seemed like a no-brainer.

Thus, we can administer our switches just like we administer our servers. Actually, thanks to Cumulus Linux, a switch to us is nothing but a "server" with many network interfaces. The switching part, though, is done on hardware (ASIC), which makes our bare metal switches at least as high-performance as those made by traditional producers.

Lazyness is king

As you rarely have just a handful of switches and we, like all IT pros, are really lazy people and thus try to automate anything and everything remotely suited for automation. That's why, obviously, when it comes to network infrastructure, we trust our esteemed configuration management system Salt.

As mentioned above, Cumulus Linux is based in Debian, or rather its current oldstable Wheezy. Installing further packages shouldn't be a problem, not even via third party repositories. On Cumulus's website and in the relevant documentation, too, Salt is listed as one of the potential configuration management systems - even if Cumulus, when it comes to automation, supposedly puts its eggs in Ansible's basket.

Two kinds of CPU

If you decide to use Salt, on the other hand, you will encounter two problems. The first one has to do with the two CPU architectures that we use. The second one - and here, I had to dig down all the way to the source code level for a solution - with Salt's Python package modules.

On our bare metal switches, we use AMD64 and PowerPC as CPU architectures. With AMD64, everything went as expected: installed the packages, linked the Salt specific package repository, chose and installed the Salt Minion (version number 2015.5.3) in the package administration - worked like a charm.

Exception: libzmq3

With PowerPC - not so much. Salt, as well as the additional Python Modules necessary, are architecture independent and thus can be installed and applied on PowerPC, too. The ZeroMQ messaging framework libzmq3, which uses Salt for communication, is the exception. Its compiled C-library must be installed along with it, so as for the Salt Minion to work. The Salt repository, though, does not provide it for PowerPC.

libzmq does exist in another repository, though, namely in Wheezy Backports. If you embed that into the package administration, libzmq3 should be available. The solution? Yes and no.

Saltstack repo unusable

The Salt packages from Saltstack's own repository can be installed, because due to their being written in Python code, they are architecture independent; the architecture dependent packages for PowerPC can be served from the Wheezy Backports.

When trying to choose the Salt Minion for installation, the Wheezy Backports repository turns out to  provide only older versions (0.18). Saltstack's own repository, on the other hand, can't be used as it doesn't contain any PowerPC packages:

http://debian.saltstack.com/debian/dists/wheezy-saltstack/Release

 0240f5f0a2cb6b177671304ffff9cf59 21238 main/binary-i386/Packages
 d417b04bf0648192c616d34ff7cf7c63 5027 main/binary-i386/Packages.gz
 5e55a9eafa39c3bd3eb9aeeea84fcf7d 145 main/binary-i386/Release
 f73621e46c1a72f8b5f473c11e068aa1 21248 main/binary-amd64/Packages
 741bd25688aa898d3fb3a2f5c031239b 5021 main/binary-amd64/Packages.gz
 a292412b342085353d25082a8c15ffdb 146 main/binary-amd64/Release
 ce0dd3da9c7fd3f17e23b34fe739918a 15609 main/binary-i686/Packages
 79f8a39310194aaad46fc8b6ccf4ab06 3739 main/binary-i686/Packages.gz
 5d04a4d1b5f5a702db3cad04539d607e 145 main/binary-i686/Release
 3c505c425043bb091866f010c1659c89 21260 main/binary-armhf/Packages
 2885d061390e982df4594777451a89e4 5034 main/binary-armhf/Packages.gz
 66a8b8ff451f98dbc4e704f2833cf129 146 main/binary-armhf/Release
 ce0dd3da9c7fd3f17e23b34fe739918a 15609 main/binary-mips/Packages
 79f8a39310194aaad46fc8b6ccf4ab06 3739 main/binary-mips/Packages.gz
 29fa1d124d1f9e1eac42664d65aeafec 145 main/binary-mips/Release
 c37c1f9bfc7356bc080464a998a65c1d 20948 main/binary-armel/Packages
 6c5333158b4d4cdcf36e2f11541f2719 5200 main/binary-armel/Packages.gz
 8a1f309ad7070859f6b66adf2706a085 146 main/binary-armel/Release
 3b402db5eb4865fb721664e4df274cc6 6744 main/source/Sources
 b42833c6d8f9a544e8b64fcb6e2a1d80 2362 main/source/Sources.gz
 8c304f10f5dfbc5f2b0734fe8d1f6983 147 main/source/Release.

The easy way would be to simply download the Salt packages yourself and manually install them on every switch, a less than ideal process, as it would have to be repeated after every update - and we are lazy, after all (see above).

The right way, on the other hands, leads through your own repository. Ideally, you are already running one, because you, like us, are already providing your own software and packages (as software, we use Reprepro, by the way). In our case, it was enough to add the PowerPC architecture to our repository.

/srv/apt-repository/rep-debian-wheezy/conf/distributions
Origin: aiticon
Label: aiticon
Suite: stable
Codename: wheezy
Description: aiticon Repository for squeeze
Architectures: i386 amd64 powerpc source
Components: main contrib non-free
Log: /srv/apt-repository/rep-debian-wheezy/logs/reprepro.log
SignWith: B3C7B376

We have the Salt packages built automatically by a dedicated build job in Atlassian Bamboo and finally have them imported into our own repository.

In short: We simultaneously embed the Wheezy Backports repository and our own one, where the Salt packages are stored (and, along with them, the package information for PowerPC, as they are what's missing in Saltstack's own repository) - and finally, the packages can be chosen in the package administration and the dependencies can be solved and installed successfully.

Salt had been installed, the test function

 (salt 'our-switch*' test.ping) 
had been executed successfully. When we wanted to apply the first states to the switch via state.highstate, things suddenly came to a halt. Our states failed, because the execution of package installations within our states ran into an error:

 

our-switch:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1161, in _thread_return
        return_data = func(*args, **kwargs)
      File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 523, in highstate
        whitelist=kwargs.get('whitelist')
      File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2968, in call_highstate
        return self.state.call_high(high)
      File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2067, in call_high
        ret = dict(list(disabled.items()) + list(self.call_chunks(chunks).items()))
      File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1623, in call_chunks
        running = self.call_chunk(low, running, chunks)
      File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1769, in call_chunk
        self._mod_init(low)
      File "/usr/lib/python2.7/dist-packages/salt/state.py", line 612, in _mod_init
        self.states['{0}.{1}'.format(low['state'], low['fun'])]  # pylint: disable=W0106
      File "/usr/lib/python2.7/dist-packages/salt/utils/lazy.py", line 90, in __getitem__
        raise KeyError(key)
    KeyError: 'pkg.installed'

Strangely enough, this problem didn't only appear on the PowerPC switches, but on the AMD64 ones, too. The assumption that this time, it was not related to the CPU architecture, seemed obvious. But as the problem was confined to switches - our servers were running flawlessly - it had to be related to Cumulus.

What followed were tenacious hours of analyzing logs and source code - which finally resulted in the solution to this particular riddle. The crucial point of source code is this one here:

https://github.com/saltstack/salt/blob/2015.5/salt/modules/aptpkg.py#L85

def __virtual__():
'''
Confirm this module is on a Debian based system
'''
if __grains__.get('os_family', False) == 'Kali':
return __virtualname__
elif __grains__.get('os_family', False) == 'Debian':
return __virtualname__
return False

The pkg module for apt is available only if, within the system, either Kali or Debian are recognized by the Grain os_family. Unfortunately, the Debian distribution Cumulus doesn't identify itself as Debian, but as Cumulus instead.

salt 'our-switch*' grains.item os_family
our-switch0:
    ----------
    os_family:
        Cumulus

What to do? We overwrite the grain OS_family by means of an additional configuration file (/etc/salt/grains).

os_family: Debian

We stop the Salt Minion, clear the cache, just to be sure (/var/cache/salt/minion) and restart the salt Minion. Now, we have the correct grain:

salt 'our-switch0' grains.item os_family
our-switch0:
    ----------
    os_family:
        Debian

And finally, our states and formulas are running on our Cumulus Linux switches, too. Voilà.

Configuration Management of bare metal switches using Salt is no rocket science after all. Once you have overcome the two pitfalls as described above, Configuration Management takes over the rest of the work: creating additional administrative users, optimizing kernel settings or installing and configuring additional services like BIRD (Internet Routing Daemon for BGP). This is where the combination of Salt and Cumulus Linux plays to its strengths. By now, nobody will shed any more tears over the old Cisco IOS shell.

In the end this is one more reason why we are 100 percent Open Source advocates: When push comes to shove, one is able to analyze the situation and find solutions on one's own and to share the knowledge with the community - which we hereby gladly do.

Are you interested? Please get in touch!