Mnet, our TCP/IP stack in OCaml

Unikernels as services

This is surely one of the major differences between an executable and a unikernel: networking. In the case of a simple executable, using the POSIX interface, it is possible to interact with the kernel's TCP/IP stack in order to initiate and manage TCP/IP connections. This ensures that the kernel alone acts as the network manager (hence a suite of related tools such as the firewall with iptables).

As for unikernels, they cannot access the kernel's TCP/IP stack. This is why unikernels have their own implementations of the TCP/IP stack. These implementations then use what are known as TAP interfaces to connect to the network. A TAP interface can really be compared to an Ethernet socket.

TAP interfaces

A TAP interface is therefore a virtual device that simulates an Ethernet port. It is through this device that the unikernel can join a network. Here, we will look at how to allocate and manage a TAP interface.

$ sudo ip tuntap add name tap0 mode tap
$ sudo ip link set tap0 up

Next, it is generally accepted that a TAP interface is part of a network. To do this, we will create a virtual bridge and connect our TAP interface to that bridge.

$ sudo ip link add name service type bridge
$ sudo ip link set service up

$ sudo ip link set tap0 master service

Finally, it may be worth setting up a local IP network and ensuring that our bridge also acts as a router.

$ sudo ip addr add 10.0.0.1/24 dev service

From this point onwards, we have everything we need to deploy multiple unikernels on the same network and communicate with them. So we're going to start using mnet.

A simple echo server

mnet is a library that implements the TCP/IP protocol. The TCP layer is itself implemented by the [utcp][utcp] project, which was created by manually extracting a proof of the TCP state machine from [HOL4][hol4]. The project implements other protocols, but here we’ll focus on a fairly simple service: an echo server. Its purpose is simply to repeat whatever is sent to it.

A brief aside on random numbers

mnet is a project that sometimes requires the generation of random numbers. Here too, our unikernel cannot use the functionality provided by the kernel (and therefore has no access to /dev/urandom or getrandom(2)). We therefore also need to initialise our own random number generator:

let ( let@ ) finally fn = Fun.protect ~finally fn

let rng =
  let fn () = Mirage_crypto_rng_mkernel.(initialize (module Pfortuna)) in
  Mkernel.map fn Mkernel.[]

let () = Mkernel.(run [ rng ]) @@ fun rng () ->
  let@ () = fun () -> Mirage_crypto_rng_mkernel.kill rng in
  Fmt.pr "%s\n%!" (Ohex.encode (Mirage_crypto_rng.generate 16))

Allocate a TCP/IP stack

So we're going to not only start our random number generator but also allocate a TCP/IP stack from a net device:

let _2s = 2_000_000_000
let ( let@ ) finally fn = Fun.protect ~finally fn
let rec forever () = Mkernel.sleep _2s; forever ()

let rng =
  let fn () = Mirage_crypto_rng_mkernel.(initialize (module Pfortuna)) in
  Mkernel.map fn Mkernel.[]

let () =

  let cidrv4 = Ipaddr.V4.Prefix.of_string_exn "10.0.0.2/24" in
  let stack = Mnet.stack ~name:"service" cidrv4 in
  Mkernel.(run [ rng; stack ]) @@ fun rng (stack, _tcp, _udp) () ->
  let@ () = fun () -> Mirage_crypto_rng_mkernel.kill rng in
  let@ () = fun () -> Mnet.kill stack in

  forever ()

Here we see the daemon concept introduced in the chapter on Miou. It is also worth noting that we always try to adhere to the rule of not forgetting our tasks. Whether it is our rng daemon or our stack daemon, they initiate background tasks that we must properly kill at the end of our unikernel (using our let@ operator).

We can now reproduce our build by adding mnet and mirage-crypto-rng-mkernel as new dependencies for our unikernel.

- (libraries mkernel fmt)
+ (libraries mkernel fmt mirage-crypto-rng-mkernel mnet)

$ unic infer -r . -x vendors -x _build > _mfetch
$ mfetch
bin                              ok
cstruct                          ok
digestif                         ok
mirage-crypto                    ok
mkernel                          ok
mnet                             ok
utcp                             ok
$ dune build

$ solo5-hvt --net:service=tap0 -- _build/solo5/main.exe --solo5:quiet &
[1] xxx
$ PID=$!

$ ping -c3 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=38 time=0.246 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=38 time=0.157 ms
64 bytes from 10.0.0.2: icmp_seq=3 ttl=38 time=0.144 ms

--- 10.0.0.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2076ms
rtt min/avg/max/mdev = 0.144/0.182/0.246/0.045 ms
$ kill -9 $PID
[1]  + xxx killed     solo5-hvt --net:service=tap0 -- _build/solo5/main.exe

At this stage, our unikernel can communicate with other executables (as well as other unikernels)! So let's start implementing our service.

Our echo server

mnet offers an interface very similar to the Unix module when it comes to sockets. Users will therefore not find it too difficult to communicate via TCP/IP. We will therefore implement an echo server, which simply involves sending back whatever the client sends us.

let handler flow =

  let rec go () = match Mnet.TCP.get flow with
    | Ok sstr ->
      let fn = Mnet.TCP.write flow in
      List.iter fn sstr;
      go ()
    | Error (`Eof | `Refused) -> () in
  let finally = Mnet.TCP.close in

  let resource = Miou.Ownership.create ~finally flow in
  Miou.Ownership.own resource;
  let@ () = fun () -> Miou.Ownership.release resource in
  go ()

let rec clean_up orphans =
  match Miou.care orphans with
  | Some None | None -> ()
  | Some (Some prm) ->
    let _ = Miou.await prm in clean_up orphans

let () =
  let cidrv4 = Ipaddr.V4.Prefix.of_string_exn "10.0.0.2/24" in
  let stack = Mnet.stack ~name:"service" cidrv4 in
  Mkernel.(run [ rng; stack ]) @@ fun rng (stack, tcp, _udp) () ->
  let@ () = fun () -> Mirage_crypto_rng_mkernel.kill rng in
  let@ () = fun () -> Mnet.kill stack in

  let listen = Mnet.TCP.listen tcp 123 in
  let rec go orphans =

    clean_up orphans;
    let flow = Mnet.TCP.accept tcp listen in
    let _ = Miou.async ~orphans @@ fun () ->
      handler flow in
    go orphans in
  go (Miou.orphans ())

We can now test our unikernel.

$ dune build
$ solo5-hvt --net:service=tap0 -- _build/solo5/main.exe --solo5:quiet &
$ PID=$!

$ nc -q0 10.0.0.2 123
Hello World!
Hello World!
^D
$ kill -9 $PID

Let's communicate with Internet

We can communicate with our unikernel, but can it communicate with us? If we’re on the local network, yes. But if the unikernel wants to connect to the Internet, we still need to configure our network and our kernel (we'll assume here that the interface connected to the Internet is wlan0):

$ sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o wlan0 -j MASQUERADE

$ sudo iptables -A FORWARD -i service -o wlan0 -j ACCEPT
$ sudo iptables -A FORWARD -i wlan0 -o wlan0 -m state \
  --state RELATED,ESTABLISHED -j ACCEPT
$ sudo sysctl -w net.ipv4.ip_forward=1

A simple HTTP client

The simplest example is to send an HTTP request to a web server. This demonstrates that our unikernel will be able to communicate with the outside world. We’re simply going to try to retrieve the content of our favourite website: https://robur.coop.

let () =
  let cidrv4 = Ipaddr.V4.Prefix.of_string_exn "10.0.0.2/24" in
  let stack = Mnet.stack ~name:"service" cidrv4 in
  Mkernel.(run [ rng; stack ]) @@ fun rng (stack, tcp, _udp) () ->
  let@ () = fun () -> Mirage_crypto_rng_mkernel.kill rng in
  let@ () = fun () -> Mnet.kill stack in

  let daemon, happy_eyeballs = Mnet_happy_eyeballs.create tcp in
  let@ () = fun () -> Mnet_happy_eyeballs.kill daemon in

  let dns = Mnet_dns.create (udp, he) in
  Mnet_happy_eyeballs.inject happy_eyeballs (getaddrinfo dns) in
  let uri = "https://robur.coop/" in
  let fn _meta _req _resp buf = function
    | Some str ->
      Buffer.add_string buf str
      buf
    | None -> buf in

  let result = Mhttp_client.request ~happy_eyeballs ~fn ~uri
    (Buffer.create 0x7ff) in
  match result with
  | Ok (resp, buf) ->
    let str = Buffer.contents buf in
    Fmt.pr "@[<hov>%a@]\n%!" (Hxd_string.pp Hxd.default) str
  | Error err -> Fmt.failwith "%a" Mhttp_client.pp_error err

This example is interesting because it shows all the steps required to make an HTTP request. In this case, several protocols are involved:

• there is, of course, TCP/IP via mnet
• but there is also DNS for resolving domain names; you can see the OCaml implementation here: ocaml-dns
• there's also happy-eyeballs, which is a standard for connecting to a service. Its implementation is available here: happy-eyeballs
• although it isn't necessarily visible, TLS is involved to encrypt the connections. Its implementation is available here: ocaml-tls
• And of course, there’s HTTP, the implementation of which is available here: ocaml-h1 (its version 2 and its integration with Miou and Mkernel here)

At this stage, you can truly consider yourself a full-stack developer!

What else can we do?

At this stage, we've achieved quite a lot. It may seem fairly straightforward to simply make an HTTP request, but the key point here is that absolutely everything is done in OCaml (from the Ethernet frame right through to the HTTP parser). Our cooperative is actively working on developing formats and protocols in OCaml, particularly so that we can use these implementations in our unikernels. It may therefore be worth taking a look at our packages at this stage to explore the OCaml unikernel ecosystem and help you develop your own unikernel.

Of course, we've shown the most visible part, but this ultimately represents only a fraction of our ecosystem. In this case, when it comes to protocols, we can mention our implementation of DHCP or SSH. We also implement protocols such as NTP and OpenVPN. We can even have a bit of fun creating our own protocol, such as spoke. In short, anything is possible!

In the next part, we'll show you just how far you can go when it comes to developing your own unikernel. However, you might also now be interested in the deployment of unikernels - and in particular, the deployment of the ones we're offering here! Unikernels sit at the intersection of development (as we’ve just seen here) and deployment. It’s time to show you our tools, such as Albatross and Aussi!