Root Cause Analysis - Voice Quality Issue

Service Degradation: More and more customers reported month-long voice quality issues such as one-way voice and underwater quality sound. The 2nd level of technical support of the call center operation team was unable to duplicate the issue with the customer. The NOC had reached out to different platform teams such as voice, IP core, and IP edge without a clue.

Even though I was in the translation team at that time, I wanted to try to dig into this ongoing service degradation issue. Thanks to my training with the call center team, I was able to access the customer ticket database. I went through each ticket and tried to be in the customer's position to look at the problem. I read all the comments provided by the customers. That information was always missed in the NOC incident ticket when it was escalated from the call center. So I went back to the basics - focusing on the 3 main parameters (jitter, latency, and packet loss) of R-Factor (a measure of voice quality).

By going through each individual customer ticket, I was able to collect a few worse nodes where the affected customer resided. It allowed me to run the IPDR for each bad call. Analyzing hundreds of bad calls, allowed me to understand when, where, and which network components were the major offenders of the bad calls. Further digging concluded that we had severe packet loss beginning from late afternoon to evening peak hours each day at one location hosting a router that was connected to a few TDM (PSTN) gateways.

By working alongside my IP core peer, we were able to identify the root cause. The bandwidth between the router and the PSTN gateways was much reduced after a major router upgrade project about a month ago. Presenting the findings with a consolidated spreadsheet of sample bad calls to the IP Core manager, the problem was resolved with an emergency change control at midnight to increase the bandwidth. Since then, no more customer tickets were received.