Troubleshooting Backend Systems

We dive into details about the DevTools architecture and components especially the networking tab, we learn the different columns, the waterfall and what each latency metric means.

Backends process request differently depending on the nature of the request, some backends might take time to process a request. Some backends might not have enough workers to process the request and might queue them, or there might be enough workers to process the request but the host is too busy and the architecture was built in a non-scalable manner where many threads running and processes are starving for CPU and ram.

We can have a C web server accept the connection but doesn't write anything to the client until 5 seconds. and close

The server might process and serialize the response very quickly but downloading the response and deserializing it on the client might be what's taking time. Slow network or client side logic or just client might be busy might be the reason.

We can use the web server to accept the connection, immediately write content-length 10 bytes but only write 9. Then sleep for 5 seconds then write the final byte.

The cost of establishing the TCP connection can be high due to latency or simply because the server listener thread is not accepting connections fast enough. We will show how to use DevTool to identify that. Note that just the TCP might be slow but the TLS is normal.

We can simulate a slow connection establishment adding a sleep 5 to the accept and make the backlog set to 1, we can fill the 1 with a curl connection then immediately try to connect from the browser.

There are other factors that slows down requests during connection establishment, first we need to perform a DNS to find the IP address of the host. Second after creating the connection we may need to encrypt it with TLS. I will demonstrate examples of when this can be slow. for this I'll use a VPN in Hong Kong to increase latency as devtools simulated latency (alas) does not apply on connection establishment. 

We take a deep look at everything Twitter has to offer through the lens of DevTools

There are cases where the app is not supported on the browser and is a desktop or mobile app even if the protocol is HTTP. For that we can't use DevTools and we need a slightly different way to sniff requests and connection establishment. We will be adding a proxy layer where the app is instructed to use the proxy to direct all HTTP traffic to the proxy and the proxy will turn around and send the request to the true backend after it captures valuable information. MITM is one of the popular HTTP debugging tools. I also use Fiddler for Windows which is my favorite when I'm on a Windows machine.

In this lecture we will learn all about MITM proxy, setup , configuration, certificate generation, traffic interception etc.

Using curl -x to specify the proxy and the backend as our c web server.

The server might process and serialize the response very quickly but downloading the response and deserializing it on the client might be what's taking time. Slow network or client side logic or just client might be busy might be the reason.

We can use the web server to accept the connection, immediately write content-length 10 bytes but only write 9. Then sleep for 5 seconds then write the final byte.

The cost of establishing the TCP connection can be high due to latency or simply because the server listener thread is not accepting connections fast enough. We will show how to use DevTool to identify that. Note that just the TCP might be slow but the TLS is normal.

We can simulate a slow connection establishment adding a sleep 5 to the accept and make the backlog set to 1, we can fill the 1 with a curl connection then immediately try to connect from the browser.

Most Mobile app use HTTP as a transport protocol, so while true we can't use DevTools obviously to intercept and example the requests we can still intercept HTTP traffic through MITM proxy. In this lecture we will learn how to direct mobile traffic to MITM proxy instance running on a machine to capture the traffic coming from the App. This can be applied for any HTTP application even desktop ones.

It is important to note that some apps uses certificate pinning so we can't intercept everything.

One of the tricks in debugging micro services or any two services or servers that talk to each other is intercepting the traffic. Sometimes you can set a proxy on one server and capture the traffic as it goes out, however some application simply don't obey proxy rules and override them. In this case we use another trick.

There are many ways to do this but this is just one trick. Say service A  talks to service B , we can pretend MITM proxy is Service B by having it listen on the same port and host as Service B (after stopping Service B). Then we spin up Service B with a different port. We later configure MITM to forward traffic to new Service B port.

This is effectively making MITM as a reverse proxy

Backends process request differently depending on the nature of the request, some backends might take time to process a request. Some backends might not have enough workers to process the request and might queue them, or there might be enough workers to process the request but the host is too busy and the architecture was built in a non-scalable manner where many threads running and processes are starving for CPU and ram.

We can have a C web server accept the connection but doesn't write anything to the client until 5 seconds. and close

The server might process and serialize the response very quickly but downloading the response and deserializing it on the client might be what's taking time. Slow network or client side logic or just client might be busy might be the reason.

We can use the web server to accept the connection, immediately write content-length 10 bytes but only write 9. Then sleep for 5 seconds then write the final byte.

The cost of establishing the TCP connection can be high due to latency or simply because the server listener thread is not accepting connections fast enough. We will show how to use DevTool to identify that. Note that just the TCP might be slow but the TLS is normal.

We can simulate a slow connection establishment adding a sleep 5 to the accept and make the backlog set to 1, we can fill the 1 with a curl connection then immediately try to connect from the browser.

With Wireshark we can  watch how slow the TCP handshake is.

1) When the client app is doing local process (Client-Backend-DB)

The javascript is returning all products then doing a client side filtering to populate the product types in a drop down.

/products/

the fix is to call

/productTypes/

2) When the backend application is doing local processing (Client-Backend-DB)

The client is correctly calling /producTypes but the backend is doing local processing..

Select name from product types then doing a distinct at app side.

The app displays the product types in a dropdown and populate it on start up. /productTypes

The fix is to do a select distinct type from products;

3) Add a load balancer,   sometimes page load is  slow sometimes it's fast.

We fixed one version of the backend to add the distinct, but the others still use the client side filtering.

how: doing a mitm proxy between at the load balancer.

then once we know the bad backend we go to the backend and do Wireshark at that backend to see its missing distinct.

4) Replica database missing index, product details page

we fetch a product by its id , the id index is missing on the replica but exists on the master

how: then once we know the bad backend we go to the backend and do Wireshark at it, we see the slow response at Wireshark to the database ,which tells us its the database and not the backend fault..

Source code attached.