This section includes:

 

The goal of this page is to help with Cloud Connector Edition (CCE) deployment and troubleshooting issues.

We recommend running the SBC and SbcComms firmware 6.1.2 before deploying CCE.

Confirm CCE state

Confirm CCE Status

Once CCE deployment is complete, confirm the following:

  1. Log into the SBC Edge WebUI.

  2. Access Tasks > Application Solution Module > Operational Status.  

    Operational Status

     

    1. If there is only the AD VM, and you are using a proxy on the ASM host, exclude the Management Network and Local Network from the proxy usage. To bypass the proxy, specify the WinHTTP Proxy settings set with your Proxy Server and a Bypass list including the 192.168.213.* network used by your Cloud Connector Managements services and the Local subnet used by your VM.

      > netsh winhttp set proxy "10.10.10.175:8080" bypass-list="192.168.213.*;192.168.214.*".

> netsh winhttp show proxy
Current WinHTTP proxy settings:
    Proxy Server(s) :  10.34.207.20:8080
    Bypass List     :  192.168.213.*;192.168.214.*

    2. Ensure four VMs are displayed. If not, restart the deployment.

  3. Ensure the latest Software Installed Version (CCE) is installed.

    CCE - Sosftware Installed Version

  4. Ensure the Auto Maintenance Status is set as None.

    CCE - Auto Maintenance Status

     

    1. If the status is Update OS or Bits, you should wait until the end of the update.
    2. If the status is Auto-Recover, check: https://blogs.technet.microsoft.com/sfbhybridvoice/2017/03/10/understanding-cloud-connector-edition-auto-recovery.

  5. Ensure the proper version is displayed as the Software Running Version.

    CCE - Software Running Version

     

    1. If the Software Running version is None or Unknown, log into the ASM using Remote Desktop.

    2. Connect the SBC Edge VM, open Services and ensure the following six Skype for Business Servers are Running. See below.

      Skype for Business Services


      c. If the Edge Servers do not start, see below for identifying solutions.

      If Edge Servers Do Not Start

       Problem - What to CheckMost Likely CauseSolution

      If all of the below issues are true:

      • On HyperV Manager, there is no Virtual Switch assigned on the VM (on the ASM)
      • Some Skype for Business Server services are not running
      • Windows Lync Event Log report Errors with Event ID 42004,19005 and 14624

      		Communication IssueOn ASM:  type "Switch-CcVersion -Force"

      If all of the below issues are true:

      • Some Skype for Business Server services are not running
      • Windows Lync Event Logs report Errors with Event ID 19008 and 19005
      		Public Certificate Issue
      • Generate a new CSR on CCE Wizard.
      • Sign the Certificate on your Public CA.
      • Import the certificate returned by your Public CA on CCE Wizard.
      • Run the Step "Prepare CCE" on CCE wizard.
      • If you are on CCE 1.4.1 only: Run Install-CcInstall

      If all of the below issues are true:

      • Some "Skype for Business Server" service are not running
      • Windows Lync Event Logs reports Error with Event ID 14623 and 12303
      		Public Certificate Issue
      • Generate a new CSR on CCE Wizard.
      • Sign the Certificate on your Public CA.
      • Import the certificate returned by your Public CA on CCE Wizard.
      • Run the Step "Prepare CCE" on CCE wizard.
      • If you are on CCE 1.4.1 only: Run Install-CcInstall

Confirm Hyper-V State

  1. On HyperV manager, check the Networking for each VM and confirm that the Corpnet Adapter is always attached to the SfB CCE Corpnet Switch. If a VM has two IPv4 addresses on the Corpnet Adapter, and one is beginning with 169, there is most likely an IP address conflict with local equipment.

    HyperV Networking - Corpnet Connection Adapter

  2. On HyperV manager, check the Networking for the Edge VM and confirm that the Internet Adapter is attached to the SfB CCE Internet Switch. If the Edge VM has two IPv4 addresses on the Internet Adapter, and one is beginning with 169, there is most likely an IP address conflict with DMZ equipment.

    HyperV Networking - Internet Connection Adapter

     

Confirm Office 365 Tenant

  1. On the Office 365 Tenant, confirm that OsUpdate is not currently running:

    PS> Get-CsHybridPSTNAppliance -Identity 091287a9-1c8f-416e-b8ba-b250fba56181
Identity                 : 091287a9-1c8f-416e-b8ba-b250fba56181
.
RegistrationStatus       : RegisteredAndDeployed
.
DeploymentStatus         : {"Status":"Deployed","Version":"1.4.1","Error":null,"StartTime":"2016-12-19T21:10:50.9209Z"}
OsUpdateStatus           : {"Status":"None","Error":null,"StartTime":null}

     

    1. If OsUpdateStatus displays the status below, wait for the update to complete. To set a time window for the update, see: Managing Your Office 365 Tenant.

      {"Status":"Updating","Error":null,"StartTime":"2016-10-31T11:47:04.1526Z"},

    2. If the OsUpdateStatus displays the error below, you can ignore it since it only appears once, and will disappear the next time Windows Update is running. This can also be fixed by removing the Windows Update on the ASM host.

      "{"Status":"Error","Error":"2017-03-21T12:55:18.9300569Z: System.Management.Automation.RuntimeException: Cannot compare \"HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\WindowsUpdate\\AU\" because it is not IComparable.\r\n"

    3. While Running Publish for HA, if you receive the error "Appliance working on a task", retry again later.


At this point, the CCE is running.

Confirm SBC state

  1. On SBC Edge WebUI, access Settings > Signaling Groups. On the CCE Signaling Group, ensure the "Service Status" is Up.

  2. Click on Counters and ensure Options are flowing in both directions and that the number of 2xx answers is matching.

    Request Counters

     

    1. If there are no Options flowing, log into the ASM using Remote Desktop.

    2. Execute the following command on a shell of the ASM Host:

       ipconfig /all

    3. Confirm :

      • "Ethernet adapter vEthernet (SfB CCE Corpnet Switch)" adapter MAC start by 00:90
      • "Ethernet adapter vEthernet (SfB CCE Internet Switch)" adapter MAC start by 00:10

    4. If unable to confirm, and the node has come directly form Manufacturing, a re-initialization will solve the issue.


At this point, both the CCE and SBC are up and running.

Confirm environmental state

Confirm the public SBC Edge FQDN is properly mapped to the External IP used in Configure CCE (via SBC Edge WebUI).

  1. From a PC with open internet access, run Nslookup on the public CCE FQDN. If the result is "Error: Requested name myccesite1.mydomain.com could not be resolved", the Public DNS server needs to be corrected.

    nslookup av.contoso.com

  2. Ensure the IP address returned is the Edge Server External Public IP. If the IP is not the correct one, you need to fix your Public DNS server.

    IP Address

     

  3. Ensure the Edge VM has proper DNS resolution.
    • Log into the ASM using Remote Desktop.
    • Connect to the the Edge VM, and open the Powershell command prompt.

    • Run the command below for all the domains assigned on your O365 tenant. If the command returns as Unknown, modify the firewall to open the port from "Internet facing DMZ" to the Internet.

      nslookup -q=cname lyncdiscover.domain.com. 8.8.8.8
nslookup -q=cname sip.domain.com. 8.8.8.8
nslookup -type=SRV _sip._tls.domain.com. 8.8.8.8
nslookup -type=SRV _sipfederationtls._tcp.domain.com. 8.8.8.8

Confirm the user state

For details, see Managing Your Office 365 Tenant.

  1. Access the Office 365 portal, access Admin Center>  Skype for Business> Voice> Voice User.
  2. Confirm the user has a phone number and a license assigned.

  3. Confirm that PSTN connectivity is "On-Premises".

    Confirm User State

     

  4. Log into your Office 365 Tenant using PowerShell

  5. On Office 365 Tenant, use "Get-CsOnlineUser" to confirm that the user is activated for EnterpriseVoice, HostedVoiceMail and HybridVoice.

    PS> $user = Get-CsOnlineUser site2user1
PS> $user.EnterpriseVoiceEnabled
True
PS> $user.HostedVoiceMail
True
PS> $user.VoicePolicy
HybridVoice

  6. On Office 365 Tenant, use "Get-CsUserPstnSettings" to confirm that the user is assigned to the proper HybridPstnSiteName.

    PS> Get-CsUserPstnSettings -Identity Site2User1
RunspaceId              : aa696d9a-67b1-4580-9a60-179cb90fd854
UserPrincipalName       : site2user1@SonusMS01.com
SipAddress              : sip:site2user1@SonusMS01.com
AllowInternationalCalls : True
HybridPstnSiteName      : AEPSITE2
HybridPstnSiteFqdn      : AEPSITE2.SONUSMS01.COM

  7. Connect a Skype for Business client using the account below:

    Skype for Business Client

Confirm the call establishment

  1. On the SBC Edge WebUI, access Diagnostics> Test a Call.
  2. Enter the phone number of your Skype Client and select your Call Routing Table "From Sip Trunk". The call should reach your Skype Client.
  3. Access Settings > Signaling Groups.

  4. On the CCE SG, click on Counters.

  5. Ensure there is Incoming Invite and that the number of 1xx answer is matching .If you get a "Additional Information: Failed Reason Q.850 Cause Code: 1.", ensure that:

    • You are calling a number that exists in O365.
    • You are using the proper SG.
    • Your Calling Number Translation are sending E164 to CCE.

    Test A Call

    Modify Counters

  6. Click on Reset Counters.

  7. Make a call from the Skype client to a PSTN number that does not exist on your Office365 Tenant.

  8. From the CCE Signaling Group, click on Counters.

  9. Ensure there is an Incoming Invite and that the number of 1xx answer matches.

    1. If the PSTN provider is slow to route the call, the Skype client may cancel the call too fast. Access Tasks / Cloud Connector Edition / Setup / Confgure CCE / Raw Ini config, and add the EnableFastFailoverTimer=false parameter into the section [TrunkConfiguration].

      Request Message Counters

  10. Click on Reset Counters.
  11. Make a call from the PSTN to the Skype Client number on your Office365 Tenant.
  12. From the CCE Signaling Group, click on Counters.

  13. Ensure there is an Outgoing Invite, and that the number of 1xx answer matches.

    Outgoing Invite

  14. Click on Reset Counters.

Confirm the call status

  1. If the call is not presenting a calling number:
    • When using a SIP Trunk, you may have to Disable the "Send Assert Header" if your Sip Trunk provider does not support multiple PAI.

  2. If the call is having bad audio quality:
    • Use the Call Quality Dashboard to ensure the firewall is properly configured.

  3. If the Forward/Transfer or Conference are failing to escalate from Online call to CCE call:

    1. Verify that the Mediation server have been added to the teantusing the command:

      Get-CsOnlineUser -Filter "HostingProvider -like '*.<YourDomain.com>'" | Get-CsHybridMediationServer
    2. If the Mediation Server is missing, follow "Configure Online Hybrid Mediation Server settings" on https://technet.microsoft.com/en-us/library/mt740651.aspx.
    3. Ensure the SBC and CCE are perfectly in sync with the Internet time.
    4. If the PSTN provider is slow to route the call, the Skype client may cancel the call too fast. Access Tasks / Cloud Connector Edition / Setup / Confgure CCE / Raw Ini config, and add the EnableFastFailoverTimer=false parameter into the section [TrunkConfiguration].

Confirm Auto-Update

  1. Check https://blogs.technet.microsoft.com/sfbhybridvoice/2017/04/07/troubleshooting-cloud-connector-edition-1-4-2-updates-and-new-deployments/
    • If you receive the following message: Not enough memory in the system to start the virtual machine 1.4.2, close all the open software running on the Host.

    • If the CCE update failed to download, and you are using a proxy on the host, specify the proxy settings per machine rather than user. You can proxy settings per machine with a registry change, or with the Group Policy setting as follows:

Registry: HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CurrentVersion\Internet Settings] ProxySettingsPerUser dword:00000000.

Group Policy: Computer>Administrative Templates>Windows Components> Internet Explorer: Make Proxy settings per machine (rather than per user).

Collect Log on CCE

  1. Log into SBC EdgeWebUI.
  2. Access Diagnostics> Cloud Connector Edition Logging>  Logging Configuration.

  3. Select Enable CCE Logging.

    CCE Logging

  4. Place a call.

  5. Click Download Log and specify a time window.

    1. StartTime. The time you run Start-CCLogging. Refer to the time in the upper right corner of the WebUI.

    2. EndTime. The current time. Refer to the time in the upper right corner of the WebUI.

      Download Log

  6. Open the Log file using the latest version of Snooper.

Update and Confirm Window's Service Version

 

Cloud Link is affected by an issue on Windows Server 2012R2 and Cloud Connector Edition; this issue creates a memory leak on the CCE Virtual Machine. As a result, the Mediation Server runs out of memory after a few weeks. As a fix to this issue, ensure the Window's service version is 6.3.9600.17552 and the Windows Update KB2919355 is installed on the CCE VM, as follows:

  1. Through Windows, log into the CCE's VM.
  2. Locate C:\Windows\System32\profsvc.dll.

  3. Right-click on the filename profsvc.dll and select Properties.

    Select File

  4. Select the Details tab.

  5. If the file version is 6.3.9600.17930, no action is required.

    File Details

  6. Access the cmd prompt and execute the systeminfo.ext command.

  7. Verify prerequisite KB2919355 is already installed on the VM.

    Verify file via Command Prompt

     

  8. Access the Control Panel and select System and Security > Windows Update.

  9. Click Automatic Updates.

  10. Following Windows checking for updates, click on the “<number> important updates are available" link.

    Automatic Updates

     

  11. Navigate to the Windows Update Options and turn on Check for Updates Automatically.

  12. Select Update for Windows Server 2012 R2 (KB3078676) and click Install.


    After installiationiscomplete, the option to restart the PC to complete the update is displayed. You must turn off Windows Update before restarting.

    DO NOT RESTART THE PC PRIOR TO TURNING OFF WINDOWS UPDATE.
    Turn off windows update before you restart to prevent the VM from automatically doing other updates. See Change Settings.

    Restart Option

     

     

  13. Through System and Security > Windows Update, click Change Settings and change Important updates to “Never check for updates (not recommended) and click OK.

    Important Updates

  14. Restart the virtual machine.

    Restart the VM

  15. Check windows update history to verify KB3078676 was installed successfully as follows:
    1. Locate C:\Windows\System32\profsvc.dll.

    2. Right-click on the filename profsvc.dll and select Properties.

    3. Click View update history.

      View Update History

      Review Update History

  16. Verify File version of profsvc.dll has been updated to 6.3.9600.17930.

    1. Locate  C:\Windows\System32\profsvc.dll.
    2. Right-click on the filename profsvc.dll and select Properties.
    3. Click Details.

    4. Verify product version is 6.3.9600.17930.

      Select File

       

      Select Details