If you have been working with container virtualization and orchestration software like Docker and Kubernetes, then you probably have heard of network namespace.
Recently, I started exploring the Linux ip
command. In this post, I will show you how to use the command to connect processes in two different network namespaces, on different subnets, over a pair of veth
interfaces.
About Network Namespace
Container runtime uses the namespace kernel feature to partition system resources to achieve a form of process isolation, such that changes to the resources in one namespace do not affect that in other namespaces. Example of such resources include process IDs, hostnames, user IDs, file names, and network interfaces.
Network namespace, in particular, virtualizes the network stack. Each network namespace has its own set of resources like network interfaces, IP addresses, routing tables, tunnels, firewalls etc. For example, iptables
rules added to a network namespace will only affect traffic entering and leaving that namespace.
Command Syntax
Some of the ip
commands in the rest of this post may look complex initially. But they are actually quite simple.
In general, all ip
commands take the form of:
For example,
- To add a new network interface, use
ip link add <interface-name> type <interface-type> <interface-arguments>...
- To allocate a new IP address range to an interface (device), use
ip addr add <ip-address-range> dev <device-name>
- To delete a route entry from the route table, use
ip route del <route-ip-range> dev <device-name>
The -n
option can be used to switch the target namespace. For example, to allocate the 10.0.1.0/24 IP address range to the interface veth0
within the vnet0
network namespace, use ip -n vnet0 addr add 10.0.1.0/24 dev veth0
.
💡 The
-n
option is the shortened form ofip netns exec
.
Configure the 1st Network Namespace
Executing all subsequent commands requires sudo privileges and have been tested on Ubuntu 16.04.6 LTS.
Our first task is to create a new pair of veth
interfaces, veth0
and veth1
, by using the ip link add
command:
The veth
interfaces are usually created as interconnected pairs, where data transmitted on one end is immediately received on the other end. This type of interfaces is commonly used in container runtime to transfer packets between different network namespaces.
Let’s create our first network namespace, vnet0
. Then we can assign the veth0
interface to this network namespace, and allocating the 10.0.1.0/24 IP address range to it:
What happens if we try to ping
the veth0
interface from both the host and vnet0
network namespaces?
Notice that veth0
is no longer reachable from the host network namespace.
Before moving on to the next step, let’s look at the last ip netns exec
command in the above code snippet. This command allows us to execute arbitrary commands in network namespaces.
It is comprised of two parts:
ip netns exec vnet0
identifies the target network namespaceping -c10 10.0.1.0
is the command to be executed in target namespace
Later we will see an example where this command is being used to run tcpdump
to debug some routing issues between the network namespaces.
Configure the 2nd Network Namespace
We will reuse the above commands to create our second network namespace, vnet1
. Then we assign the veth1
interface to this network namespace, and allocate the 10.0.2.0/24 IP address range to this interface:
Note that we deliberately use a different subnet IP range for veth1
, to set up a route debugging session later.
Similar to the veth0
interface, the veth1
interface is no longer reachable from the host network namespace. ping
only works from within the vnet1
network namespace:
Configure The Routes Between The Subnets
However, we can’t ping
either of the veth
pairs from their peer network namespace 😟😟😟…
Since both interfaces are up
, and ping
works from within the network namespaces, the issues are likely related to routing.
Let’s use the ip
command to do some debugging!
We can determine the route that a packet takes by using the ip route get
command:
Let’s examine the route tables in both network namespaces:
Can you spot the problem?
The route tables in both network namespaces only have route entries for their respective subnet IP range. They have no routes to other subnets. We can insert new route entries into the route tables using the ip route add
command:
If we tried to ping
the veth
interfaces again from the peer network namespace…
... it works!! 🎉🎉🎉
We can also use tcpdump
to capture the packets transmitted between the two network namespaces:
Let’s finish this section by testing with TCP connections.
Use the nc
command to start a TCP server at port 7096 in the vnet0
namespace. Then initiate a TCP handshake connection from the vnet1
namespace:
Once the TCP connection is established, we can send a test message from vnet1
to vnet0
:
We should see the test message appear in thevnet0
network namespace.
tcpdump
will also pick up all the packets that are transmitted between the two network namespaces:
Conclusion
In this post, we looked at the different ip
subcommands that could be used to create and configure network namespaces, interfaces and routes. We created a pair of veth
interfaces. The interfaces were assigned to two different network namespaces, with different subnet IP address range. The route tables in the network namespaces were configured with additonal routes to enable communication between the two subnets.
Both veth
interfaces were not reachable from the host network namespace. And changes to their IP address ranges and route tables were isolated to their own network namespace.
We used the ip netns exec
command to run tools likeping
and tcpdump
to debug connection issues between the network namespaces.
Connecting the veth
interfaces to the host network namespace via bridge interfaces will be content for a future post.