Clients with CloudOne Support Contract or Managed Service only require to contact Cloud One Technical Support Team & we will do the rest
Clients without CloudOne Support Contract or Managed Service can follow the below procedure for Basic Troubleshooting of 3CX
This article will take you through the Basic Troubleshooting of 3CX, as well as explaining the SIP Protocol.
Dashboard
Let’s start with the PBX dashboard. The Dashboard will provide a quick overview of the current activity and status of the PBX. We will be able to see at a glance whether everything is running the way it should.
You can see that the Firewall Check has not passed and it is shown in red, and we also do not have automatic backups. There is nothing better than a current backup to save you when a disaster happens. Always enable automatic backups. Better to be safe than sorry.
Logs
By default, the PBX will retain minimal logs, in order to conserve CPU power and disk space.
When the services are restarted or the PBX server itself is restarted, the log files will be cleaned to allow you to create a fresh set of logs, when needed.
In order to troubleshoot possible issues with the PBX, the normal logging level of LOW will not be sufficient, and Verbose logs will be needed. By definition, verbose means having more information than what is required, and in this case, we will need a lot of information which the PBX is in a position to provide us.
The log files of the PBX are retained in a folder on the server itself.
The various log files kept by the PBX are also available for perusal from the management console in addition to being available from the file system of the server. The “Logs” button in the Activity log gives you access to the logs, directly from the browser, allowing you to download them onto your computer.
Activity Log
In the Activity Log, we are able to see basic SIP Flow messages given by the PBX in relation to phone registrations, interactions with gateways and VoIP Providers as well as information on the calls being processed by the PBX.
These can be filtered based on the date and time an event happened that we would want to troubleshoot.
If we want to filter by extension, this is also possible, and in addition to that, the relevant calls for that extension will only be visible in the Call drop-down box. The Call-ID is an incremental counter of calls since the PBX or services were last restarted.
Event Log
The event log can be found in the dashboard and it provides summarised events pertaining to the registrations of Extensions, providers, and gateways. These types of the event would be logged as Informational events
There are 3 different levels shown, warnings, errors and informational messages. These can be filtered and shown as separate, individual levels.
Requests sent to the RPS servers of IP Phone providers are also logged here giving an indication if the requests are successful or not. These are informational messages if successful, and warnings if unsuccessful.
IPs being blacklisted are also logged here. These would be logged as Errors.
An example of other Warning events would be SLA time breaches in the queues, as well as abandoned queue calls.
Emails are the main way that the PBX will inform the admin of the activity in the PBX. In the email notifications settings of the PBX it is possible for more than one admin to be configured. All that is needed is to comma separate the email addresses.
This will require a valid and tested SMTP Server to be defined on the PBX.
Terminal Troubleshooting.
The majority of configuration and troubleshooting of 3CX can be done from within a web browser, but there are some cases where access to the operating system is required.
With the 3CX Linux Edition, this would require an SSH client to establish an SSH connection to the operating system. This was proving a challenge as connecting to the Operating System in Linux is more difficult than on to a windows Operating System.
A connection to the machine is easy to achieve in a Linux environment without the need for an SSH Client. From the dashboard, a button to access the web shell terminal is available. This will give you access to a limited set of functions that may be needed for troubleshooting.
Wireshark troubleshooting.
Another tool in the administrator’s arsenal, to troubleshoot call issues is the inbuilt packet capture utility, which allows an admin to easily take a packet capture on the PBX, right from within the Management Console.
You will need to have Wireshark installed on a Windows machine to have the necessary prerequisites to run the packet capture. In a Linux environment, this is not needed as all the necessary packages are installed upon installation of 3CX.
Choose the relevant interface, or choose to capture on all the interfaces if the server has more than one interface, just click Capture to start capturing network traffic on the PBX interface or interfaces.
When the capture is stopped, a download link will be provided to download the capture directly, as well as a link to create the 3CX Support information, which will also include the capture in the log files. The support info file will be sent via email to the admin.
Management console capture troubleshooting
Using the capture taken from the management console, we can use Wireshark to decode the capture and perform a SIP flow debugging. This is very easy in Wireshark, as VoIP calls can be isolated and analyzed in a clear, easy to read format.
All we need to do is select the relevant call legs we want to analyze and click Flow Sequence.
The selected call legs will now be represented in a graphical form, making it easy to see the flow of the call from beginning to end. Selecting a particular packet on the Call Flow screen will take you to the raw packet data in Wireshark as well.
Packet Captures on Network Interface.
As the packet capture is done from the server’s network interface, any audio traffic traversing the interface will also be captured.
You will be able to see the different audio streams coming from the SIP endpoints, allowing you to determine where problematic audio is originating from, if you are facing audio issues.
Any timing issues on the packets coming in and leaving the PBX will also be shown in the audio streams.
We will need to select the call legs and click Play Streams.
Ideally, 4 audio streams will be seen in a call between 2 SIP endpoints when the audio is proxied by the PBX. One for each of the streams from the PBX to the endpoints and one each from the endpoints to the PBX.
Wireshark Information.
Let’s have a look at some of the information we can see in Wireshark.
The SIP Protocol, which 3CX uses to make and receive calls, basically consists of requests and responses. The request messages are simple words, like we can see on the slide right now.
To Register a SIP Endpoint on the PBX, the word used is REGISTER.
To begin a call, an INVITE will be sent, basically inviting another endpoint into a conversation.
A BYE request is used to terminate an active call, just like saying goodbye to someone.
CANCEL is used to terminate a call that has not been answered yet. You are basically canceling the INVITE.
ACK is used to acknowledge an action, a request and response pair.
When you configure a BLF key, a SUBSCRIBE is used to enable this button and request statistics.
When the status of the BLF changes, a NOTIFY will be used to instruct the BLF to show the new status.
REFER is used when a call is transferred from one SIP Endpoint to another.
Request SIP Protocol.
The words used by the SIP Protocol of which we saw a few examples in the previous slide, are requests. They will receive responses in the form of a numbered code, and usually followed by a word. The way the word is presented is usually up to the vendor to send.
Response codes in the 100 range are provisional and are usually followed by another provisional code or a final code. An example would be 180 Ringing, indicating that the end point is in the ringing phase.
Response codes in the 200 range are Final messages and indicate a successful response to the request.
Response codes in the 300 range are Final message and indicate a redirection. For example 302 Moved Temporarily. This indicates that the endpoint is not found at the requested location, but can be found at the defined location.
Response codes in the 400 range indicate a local failure, for example when authentication is not possible, as in the case of 401 Unauthorized. 404 Not Found is a common error, when an endpoint is not contactable.
Response codes in the 500 range indicate an internal error, for example a service being unavailable, and this would give a 503 Service Unavailable message.
Response codes in the 600 range are final as well, and indicate a global failure. This would be given when a device is totally uncontactable or the endpoint refuses to communicate, as would be the case with 603 Decline.
This is a sample call flow of a REGISTER request coming from an endpoint.
The SIP endpoint, in this case an IP Phone, will try to REGISTER on to the PBX. The endpoint sends the REGISTER request. The PBX responds with a 407 Proxy Authentication Required. It is basically telling the phone to authenticate itself.
The phone will resend the REGISTER request with authentication credentials this time.
In the REGISTER message, an Expires header is also sent, indicating the time after which this message will expire and no longer be valid.
This is a sample call flow of an INVITE request. The INVITE request is sent when a SIP Endpoint wants to engage in a conversation with another SIP Endpoint, namely making a call.
An INVITE request is sent, from the phone to the 3CX server. The 3CX server responds with a 407 Proxy Authentication Required response.
The phone ACKnowledges the request.
A new INVITE is sent, with the authentication credentials.
The PBX will then forward a 100 trying response code, after successful authentication
As it is a provisional message, it is followed by a 180 Ringing, indicating that the endpoint is ringing.
If PBX Delivers Audio, or recordings are enabled, you should also see the RTP traffic flowing to and from the server, if the capture has been performed on the server itself.
The 200 OK message indicates the call has been answered.
The Endpoint ACKnowledges that this has been answered.
When the time comes for the call to be ended, a BYE is sent from the requesting endpoint. A 200 OK message from the PBX will indicate the request is successful.
Analysis of headers.
Let's start with the FROM header. This indicates where the call is coming from. In our example here we can see the call is originating from SIP extension i.e 100 at the IP Address 192.168.9.59.
We see the TO header is showing the destination as extension 102, at the same IP Address. This indicates that they are connected to the same registrar server. Very similar to email, where you send an email to a colleague, and they are on the same domain.
In order to know where to communicate the call information, the endpoint will send its CONTACT information. Here we can see the real IP of the phone.
The VIA header is the next header we will see.
Each SIP intermediary a call goes through, whether this is the PBX or a VoIP Provider, will add its own address to the VIA header.
This will allow the return call to go through the same path on the reverse route to find the original calling party.
The CALL-ID header is a unique identifier that identifies the packets within a specific call leg. All packets which are for a particular call leg will have the same Call-ID.
The CSeq header or Command Sequence header will link packets within a call leg together.
For example, when the first INVITE is sent, it will have a command sequence number, like 1 in our example. When a packet comes in response to the INVITE, it will also have a CSeq number of 1, to link to that packet.
The User-Agent header will identify the SIP Endpoint sending the packet.
The Proxy-Authorization header is used to send the authentication credentials to the SIP Registrar. In the example we can see an INVITE without credentials, and an INVITE with credentials.
Within the INVITE packet, the media negotiation is also done via the Session Description Protocol.
From here we can see the Connection information, which is similar to the contact header, but here, the IP and PORT the AUDIO will be sent to is defined.
We can also see the codecs which are available and will be negotiated with the destination endpoint. The codecs will be shown as numbers, and each number will correspond to a codec.
Some examples:
0 is PCMU, or G711u
8 is PCMA, or G711a
9 is G722
18 is G729. Protocol 18 also has an extra attribute in this case, indicating it is not an annex B codec, so it will be negotiating a G729a codec call.
Attribute 101 is the DTMF tones.
Capture Analysis.
When we analyse a capture, we can assign certain filters, to make it easier to identify the packets we want to analyse. For example, typing “sip” into the filter field, will isolate and show all the sip packets in the capture.
Going a bit deeper now, let's go and isolate all the INVITE packets. By typing sip.Method==INVITE, only the INVITE packets are shown in the wireshark interface.
By typing sip.CSeq.method==INVITE, we will isolate and see all packets which are linked to an INVITE Packet. How are they linked? Using the CSeq header in the packets.
We can assign a variety of CSeq Method filters to link the packets of various REQUESTS, like INVITE, SUBSCRIBE, REGISTER and NOTIFY.
This makes it easy to see the packets linked to an event, for example the making of a call with the INVITE.
Sometimes we want to identify the packets which are in a particular call leg. We can use the CALL-ID as a filter in this case. Now, if we go to an INVITE packet and right-click the CALL-ID, we can choose to apply this as a filter, and it will apply the filter directly into wireshark allowing us to isolate the packets for this particular call leg.
Request for Help.
It isn't always easy to troubleshoot an issue on your own, and sometimes it is necessary to bring in the Technical support team to assist.
However, before asking for assistance, we would recommend looking at our online documentation, which contains a wealth of information concerning all aspects of the configuration of 3CX.
This ranges from the configuration guides for each model of the supported phones, gateways as well as the prerequisites of the PBX in detail.
Occasionally, when we release a new feature, or when we see that some people are having difficulty with a certain feature, we publish a technical document which will clarify things up for people.
Support
Now, after you’ve done your research and done your best, you may still be facing difficulty configuring or troubleshooting an issue.
This is where you call in Cloud One Technical Support Team.
Our Technical Support team is on hand to provide assistance to our active partners, who are our first line of defence in troubleshooting issues for our customers. Support is also provided to inactive partners and end-users, upon purchasing a support ticket.
3CX branded products have support for the latest version as well as the immediately previous version.
Support is also provided for the 3CX Apps as well as 3CX Webmeeting.
Cloud One Technical Support can be engaged in one of two ways, either by telephone for our active partners, which is convenient for receiving advice, guidance or clarification of certain issues.
The option to open a ticket, which is the preferred method when the sending of information, for example logs, is required from the customer, or sending of guides and instructions in writing from the Technical Support team.
