IBM Support

StoredIQ Exchange Mount/Harvest Troubleshooting Guide

Troubleshooting


Problem

I am having trouble mounting and harvesting an Exchange volume in StoredIQ. What steps can I take to to troubleshoot this issue?

Cause

When mounting an Exchange volume within StoredIQ there are many factors that must be taken into consideration to ensure success.

The basic premise of an Exchange mount is that on an exchange volume StoredIQ actually consults the Active Directory Domain controller to return a list of mailboxes to target for harvest.

Look here for initial setup of volume configuration options (under Exchange):

http://www.ibm.com/support/knowledgecenter/en/SSSHEC_7.6.0/admin/tsk/tsk_creatingprimaryvolumes.html

For initial troubleshooting and configuration of Exchange volume:

http://www.ibm.com/support/knowledgecenter/en/SSSHEC_7.6.0/admin/cpt/cpt_configuringexchangeservers.html

This guide is to address issues that delve into issues that are still persistent after consulting the pages above.

Environment

StoredIQ Dataserver

Diagnosing The Problem

When we add an Exchange volume here is a high level overview of the steps we need to be successful on:

1) Exchange uses the EWS API which lets you talk to the mailbox store via the client access server (CAS) where Outlook Web Access (OWA) is.

2) However, the above doesn’t let you locate the users that we want to scope to…this is where we leverage Active Directory to return a list of mailboxes to target.

3) In order to leverage Active Directory, we need to first generate a Kerberos Ticket for us to authenticate to perform the LDAP queries.

4) After successfully generating a Kerberos ticket we can connect to AD and ask which users are on this mailbox server, then authenticate to EWS using NTLM or Kerberos and start making queries to start harvesting

Resolving The Problem

Going off from the overview above below is troubleshooting guide:

Troubleshooting steps

1)

The Server Field should contain the Fully Qualified Domain of the CAS server where the OWA is. The OWA (Outlook Web Access) is typically where one logs on the web into the Exchange server. Consult your Exchange admin if you do not know what the address of the CAS server where the OWA is.

a. Check if SSL is enabled in the Volume config

b. Check if user is able to login via OWA with the user and verify that you are able to check other mailboxes

c. Verify user has necessary permissions - Exchange permissions (important please read): http://www.ibm.com/support/knowledgecenter/en/SSSHEC_7.6.0/admin/cpt/cpt_configuringexchangeservers.html

d. Check initial directory field -- For Exchange, this field must be left blank if you are harvesting all mailboxes. If you are harvesting a single mailbox, enter the email address for that mailbox.

2)
In order to first test whether or not we are able to reach Active Directory, try pinging the AD server:

a. If you do not know which AD server we are trying to reach, enter the command on your Dataserver console (putty session) ‘dig SRV _kerberos._tcp.domain.com +noall +answer’ replacing ‘domain.com’ with your companies domain. The output should show you the AD server you want to talk to on the output on the righthand column.

i. Ex (output we are interested in bold below):
# dig SRV _kerberos._tcp.siq-lab.local +noall +answer

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.6 <<>> SRV _kerberos._tcp.siq-lab.local +noall +answer

;; global options: +cmd

_kerberos._tcp.siq-lab.local. 600 IN SRV 0 100 88 siq-lab-ad01.siq-lab.local.

b. Run a ‘ping adserver.domain.com’ command on your Dataserver console (Putty session) to see whether or not you can ping the AD server

c. If you cannot ping the AD server, check to see that the AD server is added to the /etc/resolv.conf file. Run the command ‘cat /etc/resolv.conf’ and verify that the server exists there.

If it does not run the command ‘vim /etc/resolv.conf’ to edit the file. Press ‘insert’ to go into editing mode. Make the changes. Then press ‘Esc’ to escape editing mode. Hit ‘:wq’ and hit ‘Enter’ to write and save changes and quit.

d. Try pinging AD server again. If it still fails add the AD server hostname to the /etc/hosts on your Dataserver. Run the command ‘vim /etc/hosts’ and edit the file adding the IP address of the AD server in the first column, then pressing ‘Tab’ to add a space and then writing the host name of the AD server. This will let your Dataserver know exactly which IP corresponds to your AD server.

e. Try pinging the AD server, this should be successful at this point.

3)
As mentioned above, we need to generate a Kerberos ticket to successfully mount an Exchange volume.

a. First try to generate one (replacing DOMAIN.COM with your domain – all CAPS are important. Replace user with a domain Admin username)

Run 'kinit [email protected]' and see if successful. If successful, you should receive a prompt for the password for the domain admin username entered.

b. When the above is successful, let’s run an LDAP query to see if we are able to return a list of mailboxes – we are essentially emulating what an Exchange harvest will do here.

Enter the command below on your Dataserver (replacing domainController.domain.com with your AD server FQDN and domain and com with the appropriate values):
ldapsearch -Q -LLL -h domainController.domain.com -b "dc=domain,dc=com" "(&(&(&(mailnickname=*)(mail=*))(&(objectCategory=person)(objectClass=user))))" mail

If successful we should see a return of a list of mailboxes, for example:

ex: ldapsearch -E pr=100/noprompt -Q -LLL -h siq-lab-ad01.siq-lab.local -b "dc=siq-lab,dc=local" "(&(&(&(mailnickname=*)(mail=*))(&(objectCategory=person)(objectClass=user))))" mail

fn: [email protected]

fn: [email protected]

fn: [email protected]

fn: [email protected]

c. If our LDAP query is unsuccessful or does not show the mail objects we expect:

From our DIG command in Step 2a, try running the LDAP query on each AD server shown there and see if any return different results. The one to return the correct results, is the AD server you want to specify in the StoredIQ volume config.

i. Check that LDAP port 389 is open on the AD server Domain Controller by running the command below on your Dataserver (replacing adserver.domain.com with your AD server FQDN):

telnet adserver.domain.com 389

If successful it should say ‘Connected to adserver.domain.com’.
Press ‘Ctrl+c’to get out of this prompt

If we are unsuccessful, there is a firewall issue, please consult your Server Admin (IT Team)

d. If our LDAP query is successful then we should be able to successfully mount the Exchange volume.

4)
If you are still unable to perform an LDAP query, please add the AD server as a host entry in /etc/hosts

example:
192.168.226.2 siqAD01.ibm.com

The LDAP query should now return mailboxes

 

5)
We should be able to successfully mount the Exchange volume at this point. If we are still not able to check the last few things below:

a. Make sure SSL is enabled on the Exchange volume config

b. Check for appropriate permissions for the user being used to mount the volume

c. Make sure the server we are trying to mount is a CAS server which the OWA resides, consult IT to make sure this info is correct in addition to credentials

6)
If you are still seeing the error 'kinit Client not found in Kerberos database while getting initial credentials, when mounting the volume' on the StoredIQ Volume UI, after successfully generating a Kerberos Ticket and running an LDAP query on the Command Line Interface on your Dataserver, then it is likely the case the DOMAIN you are running the Kerberos command against is different than the Domain of the AD server you input in the vol config.

Make sure AD server and the Kerberos ticket Domains are the same in the volume configuration.

7)
If you see the case where your volume mounts or harvests sometimes are successful and sometimes just fail or stop, it is likely the case that the kinit step is failing sometimes due to it only authenticating against the correct server(s) sometimes and not every time.

It is possible to confirm this by editing your /etc/krb5.conf file and specifying a KDC for the specific realm, turning off 'dns_lookup_kdc' to False in section '[libdefaults], and then running the kinit command in Step 3 against each different KDC server listed from the Dig command in Step 2.

If you can confirm that only a few certain KDCs let you authenticate successfully, you may specify which ones to use by adding a realm entry and following the example below such as :

STOREDIQ.COM = {

kdc = adserver.storediq.com

default_domain = storediq.com

}

Additional Exchange Online Troubleshooting

> In the case volume creation for an Exchange Online is timing out, try the below:

1. Provide Exchange Administrator privilege to the required user
2. Launch Exchange Admin page https://outlook.office365.com/ecp/
3. Click on Permissions
4. Click on + under Admin role
5. Create a role group and add the required roles
Roles
– Mailbox Search
– ApplicationImpersonation
– Mail Recipients
– Mail Enabled Public Folders
– MailboxSearchApplication
– Public Folders
6. Click Save
7. Now, you can try creating the volume and harvesting

> For all other mount failures

1. Make sure you are mounting to the server: outlook.office365.com

On your Dataserver shell:
2. Make sure you can ping the address 'outlook.office365.com'

> In case you are still failing to mount/connect to the Exchange Online volume please open up a Support case and send the below - include a note to let us know you followed the steps below:


1. Try the command below and examine output - please copy paste the output to the case

get https://outlook.office365.com

2. Enabling HTTP debug logging for further debugging

> Enable HTTP logging by executing the following commands on the Dataserver:

    psql -U dfuser dfdata -c  "update adminknobs set value = 1 where name = 'log_http_request_response';"

> Restart services
service deepfiler restart

> Attempt a mount attempt - wait for the failure error
> Please send us the logs for analysis after a mount attempt - Gather logs using the below command on the Data Server: provide file named <hostname>_storediq_dataserver<date>.tar.gz  from the /tmp directory:

tar --ignore-failed-read -cvzf /tmp/"$(hostname)_storediq_dataserver$(date '+%Y-%m-%d').tar.gz" /deepfs/config/*.out /deepfs/config/*.log /deepfs/config/*.xml /deepfs/config/siqsmb* /deepfs/config/update/update.log /deepfs/postgres/data/pg_hba.conf /usr/local/iqversion/GLOBALHANDLE /deepfs/config/debug*

> Disable HTTP logging using the commands:

    psql -U dfuser dfdata -c  "update adminknobs set value = 0 where name = 'log_http_request_response';"

> Restart services
service deepfiler restart

 

6. Make sure basic authentication is enabled on your Microsoft outlook server (must ask and confirm with Microsoft)
7. If GET request seems to be failing in logs:
Get on a console session to the Dataserver and enter the commands below:
cd /usr/lib/python2.6/site-packages/deepfile/fs/exchange/
grep ‘https:\/\/%s’ exchangeonlinemgr.py  <<-- should come back w/ a single match
cp exchangeonlinemgr.py exchangeonlinemgr.py.original
sed -i ‘/https:\/\/%s/d’ exchangeonlinemgr.py
grep ‘https:\/\/%s’ exchangeonlinemgr.py <<-- should come back w/ no matches
service deepfiler restart

and then try to define the Exchange Online volume again.
_____________________
curl -v -k -u [email protected]:Gdrr@cc0unt -H "Content-Type: text/xml; charset=utf-8" -X POST https://autodiscover.titan.gr/autodiscover/autodiscover.svc
 
8. If you go to Microsoft test connectivity you can see what the correct auto discover URL for a domain
testconnectivity.microsoft.com
Outlook connectivity
Input email address - try service account first

If the above all fail, or you require assistance, please open a support ticket

[{"Product":{"code":"SSSHEC","label":"StoredIQ"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Platform","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.6","Edition":"All Editions","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
30 January 2019

UID

swg21996647