So how many TCP connections can I make?

Overview

In web development, we deal a lot with HTTP(s) connections, which actually utilizes TCP as it’s transportation protocol.

If you’re using an http client, it’s almost trivial to think about how many http connections are established and destroyed.But if you’re working on the server side, I’m sure you have thought at least once about the connection limit, after which your server may strike.

In this article, let’s jump into the view of a process trying to establish numerous tcp connections, and see how far it can make.

Starting the game

Hey, I’m a process on a Linux server named Peter and I’ve always been told, that I can only make a maximum of 65535 TCP connections. I don’t believe this, and today I’m going to practice it myself. So I walked up to the operating system, Oscar, and said:

“Oscar, I want to create a TCP connection!”

Oscar looked at me and took out a form quickly. Sure Peter, for that you just need to fill out this for me."

I look at the table and realized it’s the classic socket quaternion. Since I only have one NIC with an IP address of 200.201.1.1, and I want to establish a TCP connection to port 80 of 200.201.1.2, so I fill in the table with this information:

What is the source port number then? I remember that a port number is a 16-bit number, and there can be any number between 0 and 65535. So I’ll pick one at random!

Just as I was hesitating on what number to choose, Oscar grabbed away my form firmly.

“What are you hesitating about, Peter? You don’t need to fill in the source port number because I will assign you an available one. Besides, you don’t need to fill in the source IP either since I know what “cards”(NIC) I have in hand and will choose the right one. Now you could go back and wait for my news.” Oscar took my form and left.

After a long time, Oscar comes back with a note for me:

“Here you go Peter, keep this safe and sound”

“What is this?”

Oscar got impatient and said, “Oh Peter, you know how much work I’ve got here. Again, this 5 is a file descriptor. Everything in Linux is a file. For your subsequent communication with this target IP over TCP, just read and write to this file descriptor”.

“So convenient! OK, thank you, Oscar.”

I took the shiny file descriptor, put it in a box and closed the lid. After all, I just wanted to see how many TCP connections I could create. All I’m doing is just to get as many of it as possible and fill up my box with them. Hehehe.

Port number crisis

After a minute, I went back to Oscar and said the same thing:

“Oscar, I want to establish a TCP connection!”

As last time, I got the same form as last time. This time, I’m much quicker and just fill in the target IP and target port:

And Oscar gave me later another note with a 6 on it:

In this way, I was going to Oscar every minute to establish a new TCP connection, the target IP was always 200.201.1.2, the target port was 80.

Oscar found my behaviour strange and wondered what I was up to. Although he has absolute authority, he had no right to refuse my request. Each time, he conscientiously completed my request quickly, giving me one note after another.

Until one time, when the note I received was a little different.

I asked in a slightly scolding tone, “Hey Oscar, that’s not what I’m supposed to get!”

Oscar was also disturbed and said, “That means that the port number is not enough! I’ve had the feeling for a long time that you’re just messing around. You’ve been creating TCP connections with the same IP and port all the time. Indeed, I had no choice but to execute the commands you gave me. But now I can’t, the port number is used up and I can’t fill in the source port for you anymore”.

I wasn’t entirely convinced and asked, “Well, don’t treat me like a newbie Oscar, I know that the port number is 16-bit, which gives a range of 1~65535. A total of 65535 TCP connections can be created. But I’ve only got 63977 at the moment. "

Oscar looked at me and sighed, “You really are a boring kid and counted that one by one, huh? Ok the reason is that, Linux limits the range of ports you can use. You can look up the limit with the following command: "

[root]$ cat /proc/sys/net/ipv4/ip_local_port_range
1024 65000

“See that? The current limit is 1024-65000. That’s why you only have 63977 ports available.”

I hurriedly apologized: “Ah, really sorry for my ignorance Oscar. But since I still need more connections, can I modify this range?”

Oscar was generous and still patiently answered me, “Yes. You could execute vim /etc/sysctl.conf to modify this file. Since you are taking up too many ports, I will add this line for you:”

net.ipv4.ip_local_port_range = 50000 50010

“After that, let’s run sysctl -p /etc/sysctl.conf to make it take effect. Now you’ll only have 10 port numbers to work with, and you’ll experience an insufficient port number error quicker. "

“That’s mean! But still I have learned so much from you Oscar. Thanks anyway! " I yelled. Oscar took away all my file descriptor notes from the box and left. I then tried another 10 tcp connections and they all succeeded. But just as Oscar configured, the 11th request failed with the same error as earlier.

“There must be another way!” I told myself. Suddenly I remembered, that for each TCP connection, Oscar will bind the source socket and target socket together. Like:
Source IP, Source Port <—–> Target IP, Target Port

As long as the quaternion formed by any binding is not already present, Oscar will give me the connection. Which means, the port number being not enough just now is just because I keep establishing connections to the same target IP and port. So I’ll try a different target port number, 3306.

I gave this table to Oscar and he immediately saw through my little mind. But since this is a legitimate request, he still went ahead and created the connection for me anyway. And I got the beautiful file descriptor note, yay! I went down this way further. And once I used up the 10 ports, I change the target IP or port. The combinations of IP and port are almost endless and I think I can collect an almost infinite amount of those lovely file descriptors. I remembered the saying now, that 65535 is the maximum number of tcp connections. How ridiculous!

File descriptor limit

Having found a way to break through the port number restriction, I kept looking for Oscar to establish TCP connections for me. Oscar couldn’t do anything to stop me. Until one time, I got another special note with something other than a file descriptor.

“What again now?!” With much frustration, I yelled again at Oscar. This time he gloated giggling and told me, “Hehe, you think you are now out of control just because you’ve broken the port number limit? Now the file descriptors are used up!”

“How come there are restrictions on everything? You operating system gives us too many restrictions, doesn’t it?”

“Nonsense! Look how many TCP connections you’ve made! For every one of it, I have to assign you a file descriptor. Linux has three separate limits on the number of file descriptors that can be opened.”

System level: The maximum number of files that can be opened on the current system.
User level: The maximum number of files that can be opened by a specific user.
Process level: The maximum number of processes that can be opened by an individual process.

These could be viewed via following commands accordingly:

cat /proc/sys/fs/file-max
cat /etc/security/limits.conf
cat /proc/sys/fs/nr_open

“Geeze, what a life under somebody’s thumb!” I thought to myself and rushed to look at these specific restrictions.

[root ~]$ cat /proc/sys/fs/file-max
32768
[root ~]$ cat /proc/sys/fs/nr_open
32768
[root ~]$ cat /etc/security/limits.conf
*soft nproc 32768
*hard nproc 32768

No wonder! I remembered that the last note I got from Oscar was like this:

And after that, I got the insufficient file descriptor error.

I asked Oscar again, “Oscar, can this limit be changed?”

Oscar still patiently told me, ‘Of course you can. For example, if you want to change the maximum file descriptor limit for a single process to 100, you can do it like this.’

[root]$ echo 100 > /proc/sys
1000

“Ah, I see! But 100 is far from enough. I will just add some 0s at the end!”

‘Eh, I knew it! ’ Oscar walked away shaking his head.

C10K Problem

After breaking the file descriptor limit, I started to create TCP connections wildly again.

However, I noticed that Oscar was getting slower and slower, and it thus took longer and longer to establish a TCP connection.

At one point, I couldn’t help but ask him, “Are you slacking off? Before, it took just an instant to establish a TCP connection. But lately, I had to wait for more than an hour for you to get it done.”

After a couple of minutes, Oscar replied, “Really Peter? Did you know that every TCP connection you make uses up a thread? As a result, my boss CPU and I are very busy because of that many threads. We’re constantly switching contexts for your hundreds of thousands of threads. But we have only limited energy, so of course we can’t serve you as fast as before.”

After listening to the Oscar’s complaints, I remembered that someone seemed to have told me about the C10K problem. Basically, when you get up to 10,000 connections and each connection needs to consume a thread resource, the operating system will be constantly busy with thread context switching, which will eventually lead to a system crash, and that is not funny at all.

I hurriedly asked for advice, “I’m really sorry, Oscar. I always thought you were omnipotent, but I didn’t realise that even you have limits. So what should we do now?”

Oscar said helplessly, “I advise you not to play further since there is no point in that. But I know how a lovely obedient kid you are, so let me tell you more. "

“The way you’re doing it, by creating a thread for every TCP connection you make, is the most traditional model of multithreaded concurrency. This is indeed the only supported way by early operating systems. But now I’ve evolved and I also support IO multiplexing, which simply means that a single thread can manage the resources of multiple TCP connections. This means you can manage a large number of TCP connections with a small number of threads.”

This is really an eye-opener! I hurried to updating my code into the IO multiplexing model. The original TCP connections and threads were destroyed and new threads were created, eachmanaging a bunch of TCP connections. Soon, Oscar was able to restore the previous efficiency, while my TCP connection number is even bigger.

Memory out

Having all the way broken through the heavy limitations of port numbers, file descriptors, thread counts, I have created a massive amount of connections and used up thousands of boxes to store those file descriptors.

Until one time, I received another weird red card.

I’m sure this is another limit which I can change and break. “And what is that this time Oscar? "

Oscar said. “This error is called memory overflow. Each TCP connection itself, as well as the buffer used by this connection, is required to occupy a certain amount of memory. Now the memory is used up and there is no more space for you to occupy. That’s why this error is reported.”

This time Oscar was particularly patient, and didn’t complain anything more. He is definitely quite exhausted now. But since I really want to see how big this trueman world is, I asked again for help.

“The great Oscar, please help the desperate Peter and do him a last favour! You have the priviledge right? Can you look for those processes with huge memory consumption and kill them? I want to complete my dream today and see how many connections on earth can be created!”

Seeing that I was really desperate enough, the Oscar obliged me and killed a lot of processes. I got tear in eyes for that.

CPU burn out

With the memory resources that Oscar had secured for me, I started creating TCP connections again day in and day out.

Oscar didn’t say anything anymore, but kept carrying out every single one of my instructions.

At one point, Oscar said to me seriously, “Now it’s the final time to advise you to stop. The CPU usage is reaching 100%.”

I find Oscar really ridiculous. After these small setbacks, I understand that there is just almost no limit which can not be bypassed. So how can I stop halfway? I ignored Oscar and kept going on.

Then on one sunny afternoon, Oscar invited me for a walk, then we went for a drink and ate something nice. As I was wondering how he manages to have so much free time, he told me word for word, “We’ve been working together for a long time Peter. I’m actually here to say goodbye to you.”

I was confused and asked, “What’s the matter Oscar? What happened?”

Oscar said, “Due to this many TCP connections from you, the CPU occupancy has been kept at 100% for a very long time. Our user, which is our God, can’t do almost anything anymore. Even a mouse move takes a long time. So he gave me a reboot command. After I executed this command, you, and all the processes like you, including me, the operating system itself, will just vanish.”

I stood up panically, “What?! Really so all of a sudden? When will this reboot instruction be executed?”

Oscar slowly got up, “About right now. We got the time for all these conversations because the command didn’t get a CPU timeslot to run. But it’s here now.”

Immediately, everything went black before my eyes…

Summarize

Every connection occupies some system resources:

When considering the maximum number of connection, the first resource restriction is decisive. Resource consumption can be viewed in this table:

As an end note, the concrete limit is actually not that important at all. Rather, the way of analyzing the problem, and seeing all the factors is the key to a deeper understanding.