IBM Support

Analysis on documents retrieved through the custom data fetcher fails at an early stage.

Technote (troubleshooting)


If the analysis phase of a job fails when documents are retrieved through the custom data fetcher, it is most likely that the format of the documents does not match the IBM Social Media Analytics (SMA) specification. After version 1.2, an additional field not mentioned in the custom data fetcher specification needs to be added to the documents to avoid failure.


After the upgrade to IBM Social Media Analytics 1.2 or later, the analysis phase of a job fails at an early stage when processing documents retrieved through a custom data fetcher.


The field "AuthorInfo" is expected by the analysis process to be present in custom data fetcher documents. If this field has been added and analysis is still failing, please ensure that you are adhering to data fetcher development kit document format.

Resolving the problem

In addition to adhering to the proposed document format for documents retrieved through the custom data fetcher, the field "AuthorInfo" needs to be present in the documents in IBM SMA 1.2. A sample document with this field included would look like this:

{"documents": [{"SiteUrl": "", "Language": "English", "Author": "IBMSMA", "Url": "", "DocumentType": "SMA", "AuthorInfo": {}, "AuthorUrl": "", "AuthorNickName": "IBM SMA", "SiteName": "IBM", "SubjectHtml": "", "TextHtml": "IBM SMA", "Published": "2013-03-20 12:00:00"}]}

Restructuring your documents by adding "AuthorInfo": {} as a field to each record and then rerunning the job starting from the data fetch stage should resolve this issue.

Related information

The Data Fetcher Development Kit

Document information

More support for: Social Media Analytics

Software version: 1.2, 1.3

Operating system(s): Linux

Reference #: 1631551

Modified date: 13 April 2014

Translate this page: