Mark Needham

Thoughts on Software Development

Archive for the ‘Networking’ Category

netcat: Strange behaviour with UDP – only receives first packet sent

with 5 comments

I was playing around with netcat yesterday to create a client and server which would communicate via UDP packets and I rediscovered some “weird” behaviour which I’d previously encountered but not explained.

I started up a netcat server listening for UDP packets on port 9000 of my machine:

$ nc -kluv localhost 9000

We can check with lsof what running that command has done:

$ lsof -Pni :9000
COMMAND   PID         USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
nc      63289 markhneedham    5u  IPv6 0xc99222a54b3975b5      0t0  UDP [::1]:9000

We can see that the netcat process is listening on port 9000 so let’s send it a UDP packet, using another netcat process:

$ echo "mark" | nc -vvu localhost 9000 -4
Connection to localhost 9000 port [udp/cslistener] succeeded!

We can see the sending of the UDP packet was successful and it shows up on the netcat server’s terminal as well.

$ nc -kluv localhost 9000
XXXXmark

If we ‘Ctrl-C’ the netcat client and run it again we’ll notice that it’s still able to connect to the port but we don’t see the message in the netcat server’s terminal:

$ echo "mike" | nc -vvu localhost 9000 -4
Connection to localhost 9000 port [udp/cslistener] succeeded!
$ nc -kluv localhost 9000
XXXXmark

I wasn’t sure what was going on but eventually came across an explanation by David Schwartz:

When nc is listening to a UDP socket, it ‘locks on’ to the source port and source IP of the first packet it receives.

As soon as it received its first datagram (from port 52832), it issued a connect system call ‘connecting’ it to the 127.0.0.1:52832. For UDP, a connect rejects all packets that don’t match the IP and port in the connect.

If we use lsof again we can see that we have exactly the same problem:

$ lsof -Pni :9000
COMMAND   PID         USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
nc      63508 markhneedham    5u  IPv6 0xc99222a54b394c5d      0t0  UDP [::1]:9000->[::1]:63732

In our case the netcat server has created a connection between port 9000 and port 63732 and rejects packets coming in from any other IP/port combination.

I thought ‘Ctrl-C’ on the netcat client would kill the connection but it doesn’t.

At this stage I wasn’t sure what to do but Nathan pointed out that if I made use of timeouts on the client and server then it would be possible to send multiple UDP packets.

I restarted the netcat server but this time with a 1 second timeout:

$ nc -kluvw 1 localhost 9000

And then started sending UDP packets from a netcat client also with a 1 second timeout:

$ echo -e "all" | nc -vvuw 1 localhost 9000
Connection to localhost 9000 port [udp/cslistener] succeeded!
$ echo -e "the" | nc -vvuw 1 localhost 9000
Connection to localhost 9000 port [udp/cslistener] succeeded!
$ echo -e "udp" | nc -vvuw 1 localhost 9000
Connection to localhost 9000 port [udp/cslistener] succeeded!
$ echo -e "packets" | nc -vvuw 1 localhost 9000
Connection to localhost 9000 port [udp/cslistener] succeeded!

And the netcat server now receives all of them:

$ nc -kluvw 1 localhost 9000
XXXXall
XXXXthe
XXXXudp
XXXXpackets

If we increase the client timeout a bit so that we can run lsof while it’s connected we can see that the connection exists for the duration of the timeout:

$ lsof -Pni :9000
COMMAND   PID         USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
nc      64317 markhneedham    5u  IPv6 0xc99222a54b394c5d      0t0  UDP [::1]:9000
nc      64541 markhneedham    5u  IPv6 0xc99222a54cc0101d      0t0  UDP [::1]:55792->[::1]:9000

But then once it’s finished it goes back to generally listening for UDP packets on that port which is exactly what we want:

$ lsof -Pni :9000
COMMAND   PID         USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
nc      64317 markhneedham    5u  IPv6 0xc99222a54b394c5d      0t0  UDP [::1]:9000

Written by Mark Needham

July 30th, 2013 at 6:01 am

Posted in Networking

Tagged with

Java: Testing a socket is listening on all network interfaces/wildcard interface

without comments

I previously wrote a blog post describing how I’ve been trying to learn more about network sockets in which I created some server sockets and connected to them using netcat.

The next step was to do the same thing in Java and I started out by writing a server socket which echoed any messages sent by the client:

public class EchoServer {
    public static void main(String[] args) throws IOException {
        int port = 4444;
        ServerSocket serverSocket = new ServerSocket(port, 50, InetAddress.getByAddress(new byte[] {0x7f,0x00,0x00,0x01}));
        System.err.println("Started server on port " + port);
 
        while (true) {
            Socket clientSocket = serverSocket.accept();
            System.err.println("Accepted connection from client: "  + clientSocket.getRemoteSocketAddress() );
 
            In  in  = new In (clientSocket);
            Out out = new Out(clientSocket);
 
            String s;
            while ((s = in.readLine()) != null) {
                out.println(s);
            }
 
            System.err.println("Closing connection with client: " + clientSocket.getInetAddress());
            out.close();
            in.close();
            clientSocket.close();
        }
    }
}
 
public final class In {
    private Scanner scanner;
 
    public In(java.net.Socket socket) {
        try {
            InputStream is = socket.getInputStream();
            scanner = new Scanner(new BufferedInputStream(is), "UTF-8");
        } catch (IOException ioe) {
            System.err.println("Could not open " + socket);
        }
    }
 
    public String readLine() {
        String line;
        try {
            line = scanner.nextLine();
        } catch (Exception e) {
            line = null;
        }
        return line;
    }
 
    public void close() {
        scanner.close();
    }
}
 
public class Out {
    private PrintWriter out;
 
    public Out(Socket socket) {
        try {
            out = new PrintWriter(socket.getOutputStream(), true);
        } catch (IOException ioe) {
            ioe.printStackTrace();
        }
    }
 
    public void close() {
        out.close();
    }
 
    public void println(Object x) {
        out.println(x);
        out.flush();
    }
}

I ran the main method of the class and this creates a server socket on port 4444 listening on the 127.0.0.1 interface and we can connect to it using netcat like so:

$ nc -v 127.0.0.1 4444
Connection to 127.0.0.1 4444 port [tcp/krb524] succeeded!
hello
hello

The output in my IntelliJ console looked like this:

Started server on port 4444
Accepted connection from client: /127.0.0.1:63222
Closing connection with client: /127.0.0.1

Using netcat is fine but what I actually wanted to do was write some test code which would check that I’d made sure the server socket on port 4444 was accessible via all interfaces i.e. bound to 0.0.0.0.

There are actually some quite nice classes in Java which make this very easy to do and wiring those together I ended up with the following client code:

public class EchoClient {
    public static void main(String[] args) throws IOException {
        Enumeration<NetworkInterface> nets = NetworkInterface.getNetworkInterfaces();
        for (NetworkInterface networkInterface : Collections.list(nets)) {
            for (InetAddress inetAddress : Collections.list(networkInterface.getInetAddresses())) {
                Socket socket = null;
                try {
                    socket = new Socket(inetAddress, 4444);
                    System.out.println(String.format("Connected using %s [%s]", networkInterface.getDisplayName(), inetAddress));
                } catch (ConnectException ex) {
                    System.out.println(String.format("Failed to connect using %s [%s]", networkInterface.getDisplayName(), inetAddress));
                } finally {
                    if (socket != null) {
                        socket.close();
                    }
                }
            }
        }
 
    }
}

If we run the main method of that class we’ll see the following output (on my machine at least!):

Failed to connect using en0 [/fe80:0:0:0:9afe:94ff:fe4f:ee50%4]
Failed to connect using en0 [/192.168.1.89]
Failed to connect using lo0 [/0:0:0:0:0:0:0:1]
Failed to connect using lo0 [/fe80:0:0:0:0:0:0:1%1]
Connected using lo0 [/127.0.0.1]

Interestingly we can’t even connect via the loopback interface using IPv6 which is perhaps not that surprising in retrospect given we bound using an IPv4 address.

If we tweak the second line of EchoServer from:

ServerSocket serverSocket = new ServerSocket(port, 50, InetAddress.getByAddress(new byte[] {0x7f,0x00,0x00,0x01}));

to:

ServerSocket serverSocket = new ServerSocket(port, 50, InetAddress.getByAddress(new byte[] {0x00,0x00,0x00,0x00}));

And restart the server before re-running the client we can now connect through all interfaces:

Connected using en0 [/fe80:0:0:0:9afe:94ff:fe4f:ee50%4]
Connected using en0 [/192.168.1.89]
Connected using lo0 [/0:0:0:0:0:0:0:1]
Connected using lo0 [/fe80:0:0:0:0:0:0:1%1]
Connected using lo0 [/127.0.0.1]

We can then wrap the EchoClient code into our testing framework to assert that we can connect via all the interfaces.

Written by Mark Needham

July 14th, 2013 at 2:31 pm

Posted in Networking

Tagged with

Learning more about network sockets

with one comment

While reading through some of the neo4j code a few weeks ago I realised that I didn’t have a very good understanding about the mechanics behind network ports/sockets so I thought I’d try to learn more.

In particular I’d not considered what binding a socket to different network interfaces meant so I decided to setup a few examples using netcat to help me understand better.

To start with let’s list the network interfaces that I have on my machine using ifconfig:

$ ifconfig -u
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
	inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	ether xxxxxxxxxxxx
	inet6 fe80::9afe:94ff:fe4f:ee50%en0 prefixlen 64 scopeid 0x4 
	inet 192.168.1.89 netmask 0xffffff00 broadcast 192.168.1.255
	media: autoselect
	status: active
p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304
	ether xxxxxxxxxxxx
	media: autoselect
	status: inactive

‘p2p0’ isn’t active so we’ll only be able to use the other two to make a socket connection on the machine.

We’ll use netcat to setup a server socket on port 4444 listening on the loopback interface:

$ nc -l -k 127.0.0.1 4444

We can use lsof to see how this has been setup:

$ lsof -Pni :4444
COMMAND   PID         USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
java    34178 markhneedham   35u  IPv6 0x114b98295506482d      0t0  TCP 127.0.0.1:4444 (LISTEN)

Let’s try and connect to that port using netcat:

$ nc -v 127.0.0.1 4444
Connection to 127.0.0.1 4444 port [tcp/krb524] succeeded!

That works as we’d expect but if we try to connect using the IP of the ‘en0’ interface we’ll get an error:

$ nc -v 192.168.1.89 4444
nc: connect to 192.168.1.89 port 4444 (tcp) failed: Connection refused

However, if we setup netcat to listen on the wildcard interface (0.0.0.0) then we would be able to connect to 4444 regardless of the interface:

$ nc -l -k 0.0.0.0 4444
$ nc -v 192.168.1.89 4444
Connection to 192.168.1.89 4444 port [tcp/krb524] succeeded!
$ nc -v 127.0.0.1 4444
Connection to 127.0.0.1 4444 port [tcp/krb524] succeeded!

If we check lsof it will confirm that we are listening on 4444 on all interfaces:

$ lsof -Pni :4444
COMMAND   PID         USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
nc      37725 markhneedham    3u  IPv4 0x114b98294ba8dfd5      0t0  TCP *:4444 (LISTEN)

One thing I hadn’t realised is that you can set up two different processes listening on the same port but on different interfaces. e.g.

$ nc -l -k 127.0.0.1 4444
$ nc -l -k 0.0.0.0 4444

We can see from lsof how this has been setup:

$ lsof -Pni :4444
COMMAND   PID         USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
nc      38306 markhneedham    3u  IPv4 0x114b98294ba8dfd5      0t0  TCP *:4444 (LISTEN)
nc      38331 markhneedham    3u  IPv4 0x114b98294cc53fd5      0t0  TCP 127.0.0.1:4444 (LISTEN)

If we make a socket connection to 127.0.0.1 it goes to our first netcat and if we use 192.168.1.89 the second one is used.

Finally, if we try to bind a server socket on an interface that we don’t own netcat will fail in a fairly predictable way:

$ nc -l -k 192.168.1.90 4444
nc: Can't assign requested address

Written by Mark Needham

July 14th, 2013 at 9:52 am

Posted in Networking

Tagged with

Network Address Translation

without comments

I’ve often heard people talking about Network Address Translation (NAT) but I never really understood exactly how it worked until we started configuring some virtual data centres on my current project.

This is an attempt at documenting my own current understanding so I won’t forget in future.

In our case we’ve been provisioning a bunch of machines into different private networks, and each machine therefore has an IP in the range of IPv4 addresses reserved for private networks:

We’ve been using IP addresses from the 10.*.*.* allocation so all our machines have an IP address in that range.

As I understand it private IP addresses were initially allocated because there are a limited number of IPv4 addresses available and because we don’t necessarily want every machine to be directly accessible to the internet.

NAT becomes necessary when we want machines outside the private network to access our machine (e.g. we want to host a web server) or if we want to access things outside the network from our machine (e.g. we want to run an ‘apt-get update’).

In the first case we’d already have a publicly accessible IP address but creating a connection to it would only take us to a router/firewall.

If we want the connection to make its way to our web server we’d need to create a Destination NAT (DNAT) rule which will translate the destination IP address and port in the IP header of the IP packet accordingly.

Dnat

We’ve created a little DSL which allows us to define these rules:

dnat :original =>   { :ip => "217.191.90.72", :port => 80 }, 
     :translated => { :ip => "10.0.0.2", :port => 80 }

As far as I understand it there would be a table which keeps track of these rules and if it sees request coming in on 217.191.90.72 on port 80 it will forward that on to 10.0.0.2 port 80.

If it’s for any other port then there will will be no such forwarding.

If we want to access the internet from our 10.0.0.2 machine we will need to put in a Source NAT (SNAT) rule to translate our source IP address and port otherwise any machines we send a request to will have no way of routing the response back to us.

In this case we wrote a more encompassing rule since we want all the machines in any of our networks to be treated the same way:

snat :original =>   { :ip => "10.0.0.0/8" }, # this means 10.*.*.*
     :translated => { :ip => "217.191.90.72" }

In this example 217.191.90.72 would be a public IP address that we have available and any requests coming from machines within our network will look like they came from there.

If we make a request to google (74.125.230.142) from 10.0.0.2 we might create a TCP connection using port 23479 so our NAT software would at least need to translate the source IP from 10.0.0.2 to 217.191.90.72 and possibly the port too.

The NAT software creates some sort of internal state in a NAT session to remember the translations that have been done so that when the response comes back from google it can be routed back to 10.0.0.2 on port 23479.

Snat

There is more going on behind the scenes to ensure that the checksums are changed accordingly but conceptually this is what’s happening.

The TCP/IP red book has a section where it explains this in more detail but I found the chapter in TCP/IP Illustrated: The Protocols an easier read.

If I’ve got anything wrong please feel free to point it out in the comments and I’ll update the post.

Written by Mark Needham

September 23rd, 2012 at 7:23 pm

Posted in Networking

Tagged with , ,

tcpdump: Learning how to read UDP packets

with 4 comments

Phil and I spent some of Friday afternoon configuring statsd:

A network daemon that runs on the Node.js platform and listens for statistics, like counters and timers, sent over UDP and sends aggregates to one or more pluggable backend services

We configured it to listen on its default port 8125 and then used netcat to send UDP packets to see if it was working like so:

echo -n "blah:36|c" | nc -w 1 -u -4 localhost 8125

We used tcpdump to capture any UDP packets on port 8125 like so:

tcpdump -i lo udp port 8125 -vv -X

To briefly explain the options we passed to it:

This is what one of the messages received by tcpdump looks like:

13:16:40.317636 IP (tos 0x0, ttl 64, id 58103, offset 0, flags [DF], proto UDP (17), length 37)
    localhost.48358 > localhost.8125: [bad udp cksum 7c8f!] UDP, length 9
	0x0000:  4500 0025 e2f7 4000 4011 59ce 7f00 0001  E..%..@.@.Y.....
	0x0010:  7f00 0001 bce6 1fbd 0011 fe24 626c 6168  ...........$blah
	0x0020:  3a33 367c 63                             :36|c

The last three lines of this output detail the IP header, UDP header and the data in the packet. The following diagram is quite useful for understanding what each part of the IP header is defining:

IP Header

Source: WTCS.org

This diagram defines things in terms of bits whereas the tcpdump output is in hexidecimal. Each block of 4 hexidecimal digits is equivalent to 16 bits.

There are a couple of parts of the IP header that might be interesting to us in this case.

The first 4 bits/1 digit define the IP version which is 4 in this case since we’re using IPv4.

The next 4 bits define the Internet Header length – the number of 32 bit words in the header. In this case the value is 5 so we know the total length of the IP header will be 160 bits (5 * 32 = 160).

The next few bits aren’t that interesting but we can see the source IP address at an offset of 96 bits and covers the next 32 bits:

0x0000: 4500 0025 e2f7 4000 4011 59ce 7f00 0001

We know this is going to represent an IPv4 address so it will be represented as ‘x.x.x.x’ where the maximum value of x is 255.

In this case since we’re just sending packets locally it translates to 127.0.0.1:

The next 32 bits are the destination IP address which has now gone onto the next line but is exactly the same:

0x0010: 7f00 0001 bce6 1fbd 0011 fe24 626c 6168

We’ve now covered 160 bits which means that the IP header is complete and we can move onto the IP payload which starts with the UDP header:

Udp header

Source: ciscoskills.net

We start with the source port which is ‘bce6’ or 48358 in decimal. We can see that value referenced in the 2nd line of the tcpdump output as well.

The next 16 bits/4 digits are the destination port which is ‘1fbd’ or 8125 in decimal – exactly what we’d expect.

The next 32 bits/2 blocks of 4 digits define the length and checksum but after that we reach the data part of the packet which should contain ‘blah:36|c’.

The word ‘blah’ is defined like so:

626c 6168

00×62 is 98 in decimal and we can use a UTF-8 encoding table to see that 98 maps to the letter ‘b’.

00x6c is 108 or the letter ‘l’, 00×61 is 97 or the letter ‘a’ and 00×68 is 104 or the letter ‘h’

We wrap onto the last line for the rest of the data we wanted to send to statsd:

0x0020:  3a33 367c 63

It follows the same pattern though where 00x3a is 58 or the ‘:’ character and so on.

And now I have a slightly better idea of how to read tcpdump’s output than I did when I started writing this! As usual any tips or hints are welcome.

—–

I found this article useful for initially understanding how to read the output but I think the diagrams above work best! TechRepublic’s ‘anatomy of a data packet‘ also provides a good explanation.

Written by Mark Needham

July 15th, 2012 at 1:29 pm

Posted in Networking

Tagged with

netcat: localhost resolution not working when sending UDP packets

with one comment

As part of some work we were doing last week Phil and I needed to send UDP packets to a local port and check that they were being picked up.

We initially tried sending a UDP packet to localhost port 8125 using netcat like so:

echo -n "hello" | nc -w 1 -u localhost 8125

That message wasn’t being received by the application listening on the port so Phil decided to try and send the same packet from Ruby which worked fine:

require 'socket'
udp = UDPSocket.new
udp.send("hello", 0, "localhost", 8125)

We eventually worked out that ‘localhost’ wasn’t being resolved by netcat which we thought was weird since it is correctly mapped:

~$ ping localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.019 ms

We dumped the packets sent to port 8125 using tcpdump and noticed that the structure of the packets received when using localhost was massively different than when we used 127.0.0.1 but we couldn’t work out what exactly was going wrong.

A better way would actually have been to listen on the port with netcat like so:

nc -l -u 8125

In this case netcat doesn’t output anything when we use the netcat command but does with the Ruby one.

I came across this post which explains exactly what’s going on:

netcat is well behaved in that it will make the getaddrinfo() system call based on the arguments given to it.

In the case of giving it the name “localhost”, a port number and telling it to use UDP, netcat is not aware of the socket family to use for the getaddrinfo() system, so it will leave at it’s default 0.

getaddrinfo() will return all possible results for the query. In this case, where you aren’t either specifying an actual IPv4 address (127.0.0.1), getaddrinfo() is also returning the IPv6 result, which is showing up first and therefore being used.

If we try that out in Ruby we can see what’s going on:

> Socket.getaddrinfo("localhost", 10002, nil, :DGRAM)
=> [["AF_INET6", 10002, "::1", "::1", 10, 2, 17], ["AF_INET", 10002, "127.0.0.1", "127.0.0.1", 2, 2, 17]]

netcat picks the first result which is “::1” – the IPv6 version of ‘localhost’ – and sends a UDP packet to that address.

If we were using TCP then netcat would initially send a packet to that address, realise it had failed and then try the next address which is the IPv4 one.

However since UDP is connectionless netcat can fire the packet and then forget about it, which is exactly what happens!

We can get around the problem by passing the -4 flag to netcat which forces it to use IPv4 addresses only.

The ‘getaddrinfo’ lookup would presumably then look like this:

Socket.getaddrinfo("localhost", 10002, :INET, :DGRAM)
=> [["AF_INET", 10002, "127.0.0.1", "127.0.0.1", 2, 2, 17], ["AF_INET", 10002, "127.0.0.1", "127.0.0.1", 2, 2, 17]]

Our final netcat command to send UDP packets therefore looks like this:

echo -n "hello" | nc -w 1 -u -4 localhost 8125

Written by Mark Needham

July 15th, 2012 at 8:14 am

Posted in Networking

Tagged with