Understanding Linux Internals for Data Transfer – Part 1

This would be a series of 3 blog posts in which we are going to talk about:

1. Basics of Networking and System calls involved in client-server communication
2. Understanding and benchmarking different data download techniques from AWS-S3
3. Potential Bottlenecks in the current downloading scheme from AWS-S3 and how to mitigate those

In this blog we are specifically going to talk about how does linux currently implements and expose APIs or system calls for transferring of data from one system to another with focus being on downloading and how can we improve the download time.

In this modern era with the advent of big data and distributed systems, its becoming more and more difficult to cache all the data you have onto production machines. So for storing this vast majority of data , we have options like Amazon S3 or Google Cloud Storage. But with this approach comes another set of problems, that being how can we quickly download this data and make users experience better.

But first we need to have a good understanding about how does this data gets downloaded onto our machine.  To understand this problem , we will just have a look into this simple piece of code which makes a simple get request on a specific url on a server.

This piece of codes makes a get request / URI on google.com:80

int main(int argc , char *argv[])
{
  int socket_desc;
  struct sockaddr_in server;
  char *message , server_reply[10];

  //Create socket
  socket_desc = socket(AF_INET , SOCK_STREAM , 0);

  //ip address of www.google.com (get by doing a ping www.google.com at terminal)
  server.sin_addr.s_addr = inet_addr("216.58.220.164");
  server.sin_family = AF_INET;
  server.sin_port = htons( 80 );

  //Connect to remote server
  connect(socket_desc , (struct sockaddr *)&server , sizeof(server));

  //Send some data
  message = "GET / HTTP/1.1\r\nHost: google.com:80\r\n\r\n";
  send(socket_desc , message , strlen(message) , 0)

  recv(socket_desc, server_reply , 6000 , 0)
  puts(server_reply);
}

Lets understand each of the system calls in

Socket

This system call is used initialise the data structures which are essential for carrying out the communication between a client and a server.   This system call returns a file descriptor on which we can use generic file descriptor system calls like read and write which internally gets mapped onto send or recvfrom system. This is done by the kernel itself.

int socket(int domain, int type, int protocol)

where

domain: Specifies the Protocol Family which would be used 
        for creating the socket.
        e.g LocalCommunication , IPv4 Internet protocols
type: Specifies the communication semantics which needs to be 
      followed while creating the socket.
      e.g. SOCK_STREAM, SOCK_DGRAM
protocol: Particular protocol to be used with the socket. Normally 
          there is only a single protocol which can be used 
          with a given socket.
          e.g. Most of the time 0, as there is only a single protocol

It returns the file descriptor of the socket on which subsequent 
system calls have to be made.

Connect

This system call is responsible for creating the connection between a client and a server. This system call is used by clients to connect to the server running on a particular port. This connect system call is responsible for making the 3-way handshake for a TCP connection and also changing the state for the kernel level data structures to reflect the connection state change i.e. ESTABLISHED

This is a generic flow for a connect system call in case of TCP connection.

int connect(int sockfd, const struct sockaddr *addr, socklen_t addr)

where

sockfd: File Descriptor representing a socket
addr: Address which contains the details of server to which we 
      make the connection
len: Size of the socket address data structure

If the connection is successfully established 0 is returned otherwise
-1 is returned.

Recv

This is a generic flow for a generic recv system call.

ssize_t recv(int sockfd, void *buf, size_t len, int flags);

where

sockfd: File Descriptor
buf: Pointer of User space Memory Buffer for received data
len: Amount of data to be received

It returns the amount of data or bytes which are actually returned
or copied in the user space buffer

But there is a single caveat in this diagram that being, that not always recvfrom system call result into process getting blocked. This happens when we already have some data to read from the socket buffer into the user space buffer of the process.

Send

This is a generic flow for a send system call

size_t send(int sockfd, const void *buf, size_t len, int flags);

where

sockfd: File Descriptor
buf: Pointer of User space Memory Buffer in which data is contained
len: Amount of data to be send

It returns the amount of data or bytes which are actually send.

As with the case with recv in this system call as well we will send out the data to the socket buffer and it might block us if and when we have the socket buffer full or socket is in non blocking mode.

More on this you can read out here:

1. TCP IP Network Stack
2. Linux TCP IP Presentation
3. Tcp System Calls