Ping Calculation Improvements

Hello everyone! I am Bart Hawthorne, a Senior Networking Engineer on the engine team here at Epic, and I’d like to discuss our method for displaying ping values in Fortnite, the new method we tried in 7.30 and how it went wrong, and finally our future plans for improving the accuracy of our display.

First though, a quick introduction on some terms I’ll be using a lot for those that are unfamiliar with them:

Client and server: Each console, mobile device, PC, or Mac is a client, and clients connect to a dedicated server that hosts the game. The server maintains the overall game state (players’ locations, the status of the circle, which Chests are active, and a whole lot more). The server is also responsible for updating all the clients when there are game state changes.

Packet: A packet is a block of data sent over the network from either the client to the server, or the server to a client. It is how clients and the server communicate with each other. In Fortnite (and UE4), we use these packets in our ping calculation method

Frame: On the server, a frame is how often the server will perform all its tasks - checking for new packets, updating the game state, and sending out updates to clients. On the client, a frame is how often the image on the screen is updated. The frame rate, also known as FPS, or frames per second, is how many frames will be processed on the server or client in one second. So when we say a client is running at 60FPS, that means that each second has 60 frames in it.

With that out of the way, let’s talk about ping!

Our Old Ping Packet Calculation Method

When most people think of “ping”, they think of the amount of time it takes for a client to send a packet to the server and for the server to send a reply back, or vice-versa. This is also referred to as the round-trip time, or RTT. This provides the information most players are interested in - how long will it take for them to receive updates from the server and how long it will take for the server to receive their updates. The higher a player’s ping value is, the more likely they will see in-game artifacts like players teleporting, being eliminated unexpectedly, or delays interacting with chests or other objects.

We display ping in milliseconds, and the lower the better. If you happen to live right next to a data center that hosts our servers, you might have a ping in the single digits! However, if you’re playing on a server that’s located on the other side of the world, your ping might be 100ms or more.

However, there’s more involved in client-server communication than just how long packets take to travel back and forth. In Fortnite, we generally check for new packets once per frame near the start on both the server and client. This means that there is a potential delay of up to a full frame between when a packet arrives and when the client or server actually checks for it if a packet arrives right after we check for new packets. Additionally, it also takes some time to read the packet and update the game state based on the data it contains.

Here is a diagram to help illustrate:

The black client and server arrows represent time. In this example, both the server are completely in sync and running at the same framerate. The black vertical lines represent the beginning of each frame, the blue notches indicate when the client or server checks if there are any packets to process, and the orange lines are when the client or server sends packets. The purple dashed lines represent a packet traveling over the internet.

Here is the sequence of events:

1. The client sends a packet to the server
2. The server receives the packet the client sent.
3. The server checks for any new packets - this is when the server sees the packet the client sent. Notice that the packet has been sitting there for nearly a full frame because it arrived just after the server checked for packets.
4. The server sends its reply to the client saying it received it.
5. The server’s reply arrives at the client.
6. The client reads the server’s reply.

We only want to display the round trip time, shown by the green arrows, and subtract out the time shown by the red arrows. So, we do our best - when we send a packet from the server to the client, it includes the server frame time, so for a specific packet, we subtract out that server frame time as well as the client frame time up to the point that we’re processing the packet. Our ping calculation looks like this:
For the value we display, we average this calculated value from packets over several seconds to produce a more consistent value.

In practice though, this isn’t as accurate as we would like. It doesn’t take into account the time when packets arrived and are waiting to be processed so the displayed ping value is often too high. The challenge is that most platforms don’t offer a way to find out when a specific actually arrived, so we don’t know exactly how much time to subtract.

Our Ping Packet Calculation Method in 7.30

For 7.30, we had an idea and wanted to try something new - what if we only used the packets that took the least amount of time to travel from the client to the server and get a response back? The theory was that the packets that took the least the amount of time would be ones that got processed right away, so that would allow us to ignore any waiting time because it should be close to zero. We ended up using the fastest 25% of our packets for our ping value calculation because it seemed like a good balance of choosing the fastest packets while also having enough data to get a realistic value.

What Went Wrong?

After we release 7.30, we noticed lots of reports of players having low single digit ping when it should be more, or even 0. If this were true, we’d be sending packets faster than the speed of light, and while UE4 is awesome, it’s not there yet! So clearly something’s wrong.

The problem involves part of our ping calculation. Here it is again for reference:
The problem is subtracting out the server frame time. When we only consider the fastest packets, we are subtracting out way too much - if a packet is handled on the server as soon as it arrives, it might only take a few milliseconds to process its contents and send a reply. The whole server frame time might be 5 times that!

Here is a diagram of what that looks like:

This is a client that has a framerate much higher than the server. Since we’re only considering the green packets, those are arriving right before the server checks for them. In this case, the server is running faster than it’s locked framerate, so it sleeps (does nothing for a while) at the start of the frame before it starts checking for packets.
 

The red line is what we’re subtracting out as the server framerate, but what would be correct is the orange line. This is clearly way too much time, which is how some clients started displaying a 0 ping.

Using our old method where we consider a wider variety of packets, this is actually still a problem. However, we have a higher chance of considering packets that waited a long time to be processed, so they can cancel out the packets where we subtracted out too much time.

Future Plans

We have improvements we plan on making in future releases to make the displayed ping value more accurate. I said the following earlier:

"The challenge is that most platforms don’t offer a way to find out when a specific actually arrived, so we don’t know how much time to subtract."

Fortunately, Linux does! Since our servers run exclusively on Linux, we can change the ping calculation to the following:

Ping = Packet time to travel to server + Packet time to travel to client-time between packet arriving on the server and sending a reply - client frame time so far.

This removes the guesswork we were trying to do by subtracting out the server frame time. We anticipate this to provide much more accurate values, but we still have lots more testing to do.

We are also investigating checking for packets more frequently on the client, which would allow us to remove the “client frame time so far” part of the calculation.

These two changes combined will hopefully give us a much more accurate ping value to display.

Conclusion

I hope this was a helpful explanation of how we calculate ping and how we’re working to improve it. When we display information like this, it’s crucial for it to be accurate - displaying inaccurate values can be worse than showing no information at all. It’s important for players to know when they miss a shot, or if the game feels “laggy” if it’s due to poor ping so that they can correct it.

I’ll be watching the comments for any questions and I would be happy to answer whatever I can. Thanks for reading!