Some months ago when installing/removing/upgrading several Documentum components, we ended up facing a strange issue (yes I know, another one!). We were able to see these specific errors during the installation or removal of a Docbase, during the installation of a patch for the Content Server, the installation of the Thumbnail Server, aso… The errors we faced change for different installers but in the end, all of these errors were linked to the same issue. The only error that wasn’t completely useless was the one faced during the installation of a new docbase: “Content is not allowed in trailing section”. Yes I know this might not be really meaningful for everybody but this kind of error usually appears when an XML file isn’t formatted properly: some content isn’t allowed at this location in the file…

The strange thing is that these installers were working fine a few days before so what changed in the meantime exactly? After some research and analysis, I finally found the guilty! One thing that has been added in these few days was D2 which has been installed a few hours before the first error. Now what can be the link between D2 and these errors when running some installers? The first thing to do when there is an issue with D2 on the Content Server is to check the Java Method Server. The first time I saw this error, it was during the installation of a new docbase. As said before, I checked the logs of the Java Method Server and I found the following WARNING which confirmed what I suspected:

2015-10-24 09:39:59,948 UTC WARNING [javax.enterprise.resource.webcontainer.jsf.config] (MSC service thread 1-3) JSF1078: Unable to process deployment descriptor for context ''{0}''.: org.xml.sax.SAXParseException; lineNumber: 40; columnNumber: 1; Content is not allowed in trailing section.
        at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:196) [xercesImpl-2.9.1-jbossas-1.jar:]
        at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:175) [xercesImpl-2.9.1-jbossas-1.jar:]
        at org.apache.xerces.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:394) [xercesImpl-2.9.1-jbossas-1.jar:]
        at org.apache.xerces.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:322) [xercesImpl-2.9.1-jbossas-1.jar:]
        at org.apache.xerces.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:281) [xercesImpl-2.9.1-jbossas-1.jar:]
        at org.apache.xerces.impl.XMLScanner.reportFatalError(XMLScanner.java:1459) [xercesImpl-2.9.1-jbossas-1.jar:]
        at org.apache.xerces.impl.XMLDocumentScannerImpl$TrailingMiscDispatcher.dispatch(XMLDocumentScannerImpl.java:1302) [xercesImpl-2.9.1-jbossas-1.jar:]
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:324) [xercesImpl-2.9.1-jbossas-1.jar:]
        at org.apache.xerces.parsers.XML11Configuration.parse(XML11Configuration.java:845) [xercesImpl-2.9.1-jbossas-1.jar:]
        at org.apache.xerces.parsers.XML11Configuration.parse(XML11Configuration.java:768) [xercesImpl-2.9.1-jbossas-1.jar:]
        at org.apache.xerces.parsers.XMLParser.parse(XMLParser.java:108) [xercesImpl-2.9.1-jbossas-1.jar:]
        at org.apache.xerces.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1196) [xercesImpl-2.9.1-jbossas-1.jar:]
        at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:555) [xercesImpl-2.9.1-jbossas-1.jar:]
        at org.apache.xerces.jaxp.SAXParserImpl.parse(SAXParserImpl.java:289) [xercesImpl-2.9.1-jbossas-1.jar:]
        at javax.xml.parsers.SAXParser.parse(SAXParser.java:195) [rt.jar:1.7.0_72]
        at com.sun.faces.config.ConfigureListener$WebXmlProcessor.scanForFacesServlet(ConfigureListener.java:815) [jsf-impl-2.1.7-jbossorg-2.jar:]
        at com.sun.faces.config.ConfigureListener$WebXmlProcessor.<init>(ConfigureListener.java:768) [jsf-impl-2.1.7-jbossorg-2.jar:]
        at com.sun.faces.config.ConfigureListener.contextInitialized(ConfigureListener.java:178) [jsf-impl-2.1.7-jbossorg-2.jar:]
        at org.apache.catalina.core.StandardContext.contextListenerStart(StandardContext.java:3392) [jbossweb-7.0.13.Final.jar:]
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:3850) [jbossweb-7.0.13.Final.jar:]
        at org.jboss.as.web.deployment.WebDeploymentService.start(WebDeploymentService.java:90) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811)
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_72]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_72]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_72]

[dmadmin@content_server_01 ~]$ cat $DOCUMENTUM_SHARED/jboss7.1.1/server/DctmServer_MethodServer/deployments/ServerApps.ear/DmMethods.war/WEB-INF/web.xml
<?xml version="1.0" encoding="UTF-8"?>
<web-app>
    <display-name>Documentum Method Invocation Servlet</display-name>
    <description>This servlet is for Java method invocation using the DO_METHOD apply call.</description>
    <servlet>
        <servlet-name>DoMethod</servlet-name>
        <description>Documentum Method Invocation Servlet</description>
        <servlet-class>com.documentum.mthdservlet.DoMethod</servlet-class>
        <init-param>
            <param-name>trace</param-name>
            <param-value>f</param-value>
        </init-param>
        <init-param>
            <param-name>docbase_install_owner_name</param-name>
            <param-value>dmadmin</param-value>
        </init-param>
        <init-param>
            <param-name>methodlocation-1</param-name>
            <param-value>$DOCUMENTUM/dba/java_methods</param-value>
        </init-param>
        <init-param>
            <param-name>docbase-GLOBAL_REGISTRY</param-name>
            <param-value>GLOBAL_REGISTRY</param-value>
        </init-param>
        <init-param>
            <param-name>docbase-DOCBASE1</param-name>
            <param-value>DOCBASE1</param-value>
        </init-param>
        <load-on-startup>1</load-on-startup>
    </servlet>
    <servlet-mapping>
        <servlet-name>DoMethod</servlet-name>
        <url-pattern>/servlet/DoMethod</url-pattern>
    </servlet-mapping>
</web-app>
ap
[dmadmin@content_server_01 ~]$

Cet article Documentum story – Documentum installers fail with various errors est apparu en premier sur Blog dbi services.

When using xPlore with Documentum, you will need to setup a DSearch which will be used to perform the searches and this DSearch uses in the background an Xhive Database. This is a native XML Database that persists XML DOMs and provide access to them using XPath and XQuery. In this blog, I will share the steps needed to change the location of the Xhive Database used by the DSearch. You usually don’t want to move this XML Database everyday but it might be useful as a one-time action. In this customer case, one of the DSearch in a Sandbox/Dev environment has been installed using a wrong path for the Xhive Database (not following our installation conventions) and therefore we had to correct that just to keep the alignment between all environments and to avoid a complete uninstall/reinstall of the IndexAgents + DSearch.

In the steps below, I will suppose that xPlore has been installed under “/app/xPlore” and that the Xhive Database has been created under “/app/xPlore/data”. This is the default value and then when installing an IndexAgent, it will create, under the data folder, a sub-folder with a name equal to the DSearch Domain’s name (usually the name of the docbase/repository). In this blog I will show you how to move this Xhive Database to “/app/xPlore/test-data” without having to reinstall everything. This means that the Xhive Database will NOT be deleted/recreated from scratch (this is also possible) and therefore you will NOT have to perform a full reindex which would have taken a looong time.

So let’s start with stopping all components first:

[xplore@xplore_server_01 ~]$ sh -c "/app/xPlore/jboss7.1.1/server/stopIndexagent.sh"
[xplore@xplore_server_01 ~]$ sh -c "/app/xPlore/jboss7.1.1/server/stopPrimaryDsearch.sh"

Once this is done, we need to backup the data and config files, just in case…

[xplore@xplore_server_01 ~]$ current_date=$(date "+%Y%m%d")
[xplore@xplore_server_01 ~]$ cp -R /app/xPlore/data/ /app/xPlore/data_bck_$current_date
[xplore@xplore_server_01 ~]$ cp -R /app/xPlore/config/ /app/xPlore/config_bck_$current_date
[xplore@xplore_server_01 ~]$ mv /app/xPlore/data/ /app/xPlore/test-data/

Ok now everything in the background is prepared and we can start the actual steps to move the Xhive Database. The first step is to change the data location in the files stored in the config folder. There is actually two files that need to be updated: indexserverconfig.xml and XhiveDatabase.bootstrap. In the first file, you need to update the “storage-location” path that defines where the data are kept and in the second file you need to update all paths pointing to the Database files. Here are some simple commands to replace the old path with the new path and check that it has been done properly:

[xplore@xplore_server_01 ~]$ sed -i "s,/app/xPlore/data,/app/xPlore/test-data," /app/xPlore/config/indexserverconfig.xml
[xplore@xplore_server_01 ~]$ sed -i "s,/app/xPlore/data,/app/xPlore/test-data," /app/xPlore/config/XhiveDatabase.bootstrap
[xplore@xplore_server_01 ~]$ 
[xplore@xplore_server_01 ~]$ grep -A2 "<storage-locations>" /app/xPlore/config/indexserverconfig.xml
    <storage-locations>
        <storage-location path="/app/xPlore/test-data" quota_in_MB="10" status="not_full" name="default"/>
    </storage-locations>
[xplore@xplore_server_01 ~]$ 
[xplore@xplore_server_01 ~]$ grep "/app/xPlore/test-data" /app/xPlore/config/XhiveDatabase.bootstrap | grep 'id="[0-4]"'
        <file path="/app/xPlore/test-data/xhivedb-default-0.XhiveDatabase.DB" id="0"/>
        <file path="/app/xPlore/test-data/SystemData/xhivedb-SystemData-0.XhiveDatabase.DB" id="2"/>
        <file path="/app/xPlore/test-data/SystemData/MetricsDB/xhivedb-SystemData#MetricsDB-0.XhiveDatabase.DB" id="3"/>
        <file path="/app/xPlore/test-data/SystemData/MetricsDB/PrimaryDsearch/xhivedb-SystemData#MetricsDB#PrimaryDsearch-0.XhiveDatabase.DB" id="4"/>
        <file path="/app/xPlore/test-data/xhivedb-temporary-0.XhiveDatabase.DB" id="1"/>

The next step is to announce the new location of the data folder to the DSearch so it can create future Xhive Databases at the right location and this is done inside the file indexserver-bootstrap.properties. After the update, this file should look like the following:

[xplore@xplore_server_01 ~]$ cat /app/xPlore/jboss7.1.1/server/DctmServer_PrimaryDsearch/deployments/dsearch.war/WEB-INF/classes/indexserver-bootstrap.properties
# (c) 1994-2009, EMC Corporation. All Rights Reserved.
#Wed May 20 10:40:49 PDT 2009
#Note: Do not change the values of the properties in this file except xhive-pagesize and force-restart-xdb.
node-name=PrimaryDsearch
configuration-service-class=com.emc.documentum.core.fulltext.indexserver.core.config.impl.xmlfile.IndexServerConfig
indexserver.config.file=/app/xPlore/config/indexserverconfig.xml
xhive-database-name=xhivedb
superuser-name=superuser
superuser-password=****************************************************
adminuser-name=Administrator
adminuser-password=****************************************************
xhive-bootstrapfile-name=/app/xPlore/config/XhiveDatabase.bootstrap
xhive-connection-string=xhive://xplore_server_01:9330
xhive-pagesize=4096
# xhive-cache-pages=40960
isPrimary = true
licensekey=**************************************************************
xhive-data-directory=/app/xPlore/test-data
xhive-log-directory=

In this file:

• indexserver.config.file => defines the location of the indexserverconfig.xml file that must be used to recreate the DSearch Xhive Database.
• xhive-bootstrapfile-name => defines the location and name of the Xhive bootstrap file that will be generated during bootstrap and will be used to create the empty DSearch Xhive Database.
• xhive-data-directory => defines the path of the data folder that will be used by the Xhive bootstrap file. This will therefore be the future location of the DSearch Xhive Database.

As you probably understood, to change the data folder, you just have to adjust the value of the parameter “xhive-data-directory” to point to the new location: /app/xPlore/test-data.

When this is done, the third step is to change the Lucene temp path:

[xplore@xplore_server_01 ~]$ cat /app/xPlore/jboss7.1.1/server/DctmServer_PrimaryDsearch/deployments/dsearch.war/WEB-INF/classes/xdb.properties
xdb.lucene.temp.path=/app/xPlore/test-data/temp

In this file, xdb.lucene.temp.path defines the path for temporary uncommitted indexes. Therefore it will just be used for temporary indexes but it is still a good practice to change this location since it’s also talking about the data of the DSearch and it helps to keep everything consistent.

Then the next step is to clean the cache and restart the DSearch. You can use your custom start/stop script if you have one or use something like this:

[xplore@xplore_server_01 ~]$ rm -rf /app/xPlore/jboss7.1.1/server/DctmServer_*/tmp/work/*
[xplore@xplore_server_01 ~]$ sh -c "cd /app/xPlore/jboss7.1.1/server;nohup ./startPrimaryDsearch.sh & sleep 5;mv nohup.out nohup-PrimaryDsearch.out"

Once done, just verify in the log file generated by the start command (for me: /app/xPlore/jboss7.1.1/server/nohup-PrimaryDsearch.out) that the DSearch has been started successfully. If that’s true, then you can also start the IndexAgent:

[xplore@xplore_server_01 ~]$ sh -c "cd /app/xPlore/jboss7.1.1/server;nohup ./startIndexagent.sh & sleep 5;mv nohup.out nohup-Indexagent.out"

And here we are, the Xhive Database is now located under the “test-data” folder!

Additional note: As said at the beginning of this blog, it is also possible to recreate an empty Xhive Database and change its location at the same time. Doing a recreation of am empty DB will result in the same thing as the steps above BUT you will have to perform a full reindexing which will take a lot of time if this isn’t a new installation (the more documents are indexed, the more time it will take)… To perform this operation, the steps are mostly the same and are summarized below:

1. Backup the data and config folders
2. Remove all files inside the config folder except the indexserverconfig.xml
3. Create a new (empty) data folder with a different name like “test-data” or “new-data” or…
4. Update the file indexserver-bootstrap.properties with the reference to the new path
5. Update the file xdb.properties with the reference to the new path
6. Clean the cache and restart the DSearch+IndexAgents

Basically, the steps are exactly the same except that you don’t need to update the files indexserverconfig.xml and XhiveDatabase.bootstrap. The first one is normally updated by the DSearch automatically and the second file will be recreated from scratch using the right data path thanks to the update of the file indexserver-bootstrap.properties.

Have fun

Cet article Documentum story – Change the location of the Xhive Database for the DSearch (xPlore) est apparu en premier sur Blog dbi services.

When you have an installation with one Content Server (CS) you do not take care where the job will be running. It’s always on your single CS.
But how should you configure the jobs in case you have several CS? Which jobs have to be executed and which one not? Let’s see that in this post.

When you have to run your jobs in a high availability installation you have to configure some files and objects.

Update the method_verb of the dm_agent_exec method:

API> retrieve,c,dm_method where object_name = 'agent_exec_method'
API> get,c,l,method_verb
API> set,c,l,method_verb
SET> ./dm_agent_exec -enable_ha_setup 1
API> get,c,l,method_verb
API> save,c,l
API> reinit,c

The java methods have been updated to be restartable:

update dm_method object set is_restartable=1 where method_type='java';

On our installation we use jms_max_wait_time_on_failures = 300 instead the default one (3000).
In server.ini (Primary Content Server) and server_HOSTNAME2_REPO01.ini (Remote Content Server), we have:

incremental_jms_wait_time_on_failure=30
jms_max_wait_time_on_failures=300

Based on some issues we faced, for instance with the dce_clean job that ran twice when we had both JMS projected to each CS, EMC advised us to each JMS with its local CS only. With this configuration, in case the JMS is down on the primary CS, the job (using a java method) is started on the remote JMS via the remote CS.

Regarding which jobs have to be executed – I am describing only the one which are used for the housekeeping.
So the question to answer is which job does what and what is “touched”, either metadata or/and content.

To verify that, check how many CS are used and where they are installed:

select object_name, r_host_name from dm_server_config
REPO1               HOSTNAME1.DOMAIN
HOSTNAME2_REPO1     HOSTNAME2.DOMAIN

Verify on which CS the jobs will run and “classify” them.
Check the job settings:

select object_name, target_server, is_inactive from dm_job

Metadata

The following jobs work only on metadata, they can run anywhere so the target_server has to be empty

Content

The following jobs work only on content.

As we are using a NAS for the Data directory which is shared for both servers, only one of the two jobs has to run. Per default the target_server is defined. So for the one which has to run, target_server has to be empty.

Metadata and Content

These following jobs are working on metadata and content.

Filescan scans the NAS content storage. As said above, it is shared and therefore the job only need to be execute once: the target_server has to be empty to be run everywhere.

LogPurge is also cleaning files under $DOCUMENTUM/dba/log and subfolders which are obviously not shared and therefore both dm_LogPurge jobs have to run. You just have to use another start time to avoid an overlap when objects are removed from the repository.

Normally with this configuration your housekeeping jobs should be well configured.

One point you have to take care is when you use DA to configure your jobs. Once you open the job properties, the “Designated Server” is set to one of your server and not to “Any Running Server” which means target_server = ‘ ‘. If you click the OK button, you will set the target server and in case this CS is down, the job will fail because it cannot use the second CS.

Cet article Documentum story – Jobs in a high availability installation est apparu en premier sur Blog dbi services.

I’ve been patching around some xPlore servers for a while and recently I went into an issue regarding the ftintegrity tool. Maybe you figured it out as well, for xPlore 1.5 Patch 15 the ftintegrity tool available in $DSEARCH_HOME/setup/indexagent/tools was corrupted by the patch.
I think for some reason the libs were changed and the tool wasn’t able to load anymore. I asked EMC and they said it was a known bug which will be fixed in next release.

So I came to patch it again when the Patch 17 went out (our customer processes doesn’t allow us to patch every month, so I skipped the Patch 16). After patching, I directly started the ftintegrity tool in order to check that everything was fixed, and…. no.

In fact yes, but you have something to do before making it work. The error you have is like ‘Could not load because config is NULL’, or ‘dfc.properties not found’. I found these error kinda strange, so I wondered if the ftintegrity tool script was patched as well. And the answer is no, the script is still the same but the jar libraries have been changed, that means that the script is pointing to the wrong libraries and it can’t load properly.

Thus the solution is simple, I uninstalled the Index Agent and installed it again right after. The ftintegrity tool script was re-created with the good pointers to the new libraries. Little tip: If you have several Index Agents and don’t want to re-install them all, you may want to copy the content of the updated ftintegrity tool script and paste it into the other instances (do not forget to adapt it because it may point to different docbases).

To summary, if you have issues regarding the execution of the ftintegrity tool, check the libraries call in the script and try to re-install it in order to have the latest one.

Cet article Documentum xPlore – ftintegrity tool not working est apparu en premier sur Blog dbi services.

Some time ago, I was working on the preparation of an upgrade of a Content Server and everything was working fine so I was about to begin but just before that I checked our monitoring interface for this environment to crosscheck and I saw the following alerts coming from the log file of the docbases installed on this CS:

2016-04-05T10:47:01.411651      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"

As you might know, an object with an ID starting with “3c” represents the dm_docbase_config and therefore I was a little bit afraid when I saw this alert. So first thing, I tried to open an idql session to see if the docbase was responding:

[dmadmin@content_server_01 log]$ iapi DOCBASE1 -Udmadmin -Pxxx


        EMC Documentum iapi - Interactive API interface
        (c) Copyright EMC Corp., 1992 - 2015
        All rights reserved.
        Client Library Release 7.2.0050.0084


Connecting to Server using docbase DOCBASE1
[DM_SESSION_I_SESSION_START]info:  "Session 013f245a800d7441 started for user dmadmin."


Connected to Documentum Server running Release 7.2.0050.0214  Linux64.Oracle
Session id is s0
API> retrieve,c,dm_docbase_config
...
3c3f245a60000210
API> exit
Bye

Ok so apparently the docbase is working properly, we can access D2, DA, the idql is also working and even the dm_docbase_config object can be retrieved and seen but there is still the exact same error in the log file coming every 5 minutes exactly. So I took a look at the error message in detail because I didn’t want to scroll hundreds of lines for maybe nothing. The string description containing the object ID wouldn’t really be useful inside the log file to find what might be the root cause and the same thing apply for the DM_OBJ_MGR_E_FETCH_FAIL error, it would just print the exact same error again and again with just different timestamps. When thinking about that in my mind, I actually realised that the 2 or 3 error lines I was able to see on my screen were completely exact – except the timestamps – and that include the process ID that is throwing this error (second column on the log file).

With this new information, I tried to find all log entries related to this process ID:

[dmadmin@content_server_01 log]$ grep "21425\[21425\]" $DOCUMENTUM/dba/log/DOCBASE1.log | more
2016-04-04T06:58:09.315163      21425[21425]    0000000000000000        [DM_SERVER_I_START_SERVER]info:  "Docbase DOCBASE1 attempting to open"
2016-04-04T06:58:09.315282      21425[21425]    0000000000000000        [DM_SERVER_I_START_KEY_STORAGE_MODE]info:  "Docbase DOCBASE1 is using database for cryptographic key storage"
2016-04-04T06:58:09.315316      21425[21425]    0000000000000000        [DM_SERVER_I_START_SERVER]info:  "Docbase DOCBASE1 process identity: user(dmadmin)"
2016-04-04T06:58:11.400017      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Post Upgrade Processing."
2016-04-04T06:58:11.401753      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Base Types."
2016-04-04T06:58:11.404252      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmRecovery."
2016-04-04T06:58:11.412344      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmACL."
2016-04-04T06:58:11.438249      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmDocbaseIdMap."
2016-04-04T06:58:11.447435      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Error log streams."
2016-04-04T06:58:11.447915      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmUser."
2016-04-04T06:58:11.464912      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmGroup."
2016-04-04T06:58:11.480200      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmSysObject."
2016-04-04T06:58:11.515201      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmExprCode."
2016-04-04T06:58:11.524604      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmKey."
2016-04-04T06:58:11.533883      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmValueAssist."
2016-04-04T06:58:11.541708      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmValueList."
2016-04-04T06:58:11.551492      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmValueQuery."
2016-04-04T06:58:11.559569      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmValueFunc."
2016-04-04T06:58:11.565830      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmExpression."
2016-04-04T06:58:11.594764      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmLiteralExpr."
2016-04-04T06:58:11.603279      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmBuiltinExpr."
2016-04-04T06:58:11.625736      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmFuncExpr."
2016-04-04T06:58:11.636930      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmCondExpr."
2016-04-04T06:58:11.663622      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmCondIDExpr."
2016-04-04T06:58:11.707363      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmDDInfo."
2016-04-04T06:58:11.766883      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmScopeConfig."
2016-04-04T06:58:11.843335      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmDisplayConfig."
2016-04-04T06:58:11.854414      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmNLSDDInfo."
2016-04-04T06:58:11.878566      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmDomain."
2016-04-04T06:58:11.903844      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmAggrDomain."
2016-04-04T06:58:11.929480      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmMountPoint."
2016-04-04T06:58:11.957705      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmLocation."
2016-04-04T06:58:12.020403      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Server Configuration."
2016-04-04T06:58:12.135418      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmPolicy."
2016-04-04T06:58:12.166923      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmDDCommonInfo."
2016-04-04T06:58:12.196057      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmDDAttrInfo."
2016-04-04T06:58:12.238040      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmDDTypeInfo."
2016-04-04T06:58:12.269202      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmRelation."
2016-04-04T06:58:12.354573      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmForeignKey."
2016-04-04T06:58:12.387309      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmEvent."
2016-04-04T06:58:12.403895      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize DQL."
2016-04-04T06:58:12.405622      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmFolder."
2016-04-04T06:58:12.433583      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmDocument."
2016-04-04T06:58:12.480234      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Plugin Type."
2016-04-04T06:58:12.490196      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmNote."
2016-04-04T06:58:12.518305      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmComposite."
2016-04-04T06:58:12.529351      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmStorage."
2016-04-04T06:58:12.539944      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Common area."
2016-04-04T06:58:12.612097      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize File Store."
2016-04-04T06:58:12.675604      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Optical Store."
2016-04-04T06:58:12.717573      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Linked Stores."
2016-04-04T06:58:12.803227      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Distributed Stores."
2016-04-04T06:58:13.102632      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Blob Stores."
2016-04-04T06:58:13.170074      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize External Store."
2016-04-04T06:58:13.242012      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize External File Store."
2016-04-04T06:58:13.305767      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize External URL."
2016-04-04T06:58:13.363407      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize External Free."
2016-04-04T06:58:13.429547      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmContent."
2016-04-04T06:58:13.461400      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmiSubContent."
2016-04-04T06:58:13.508588      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmOutputDevice."
2016-04-04T06:58:13.630872      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmMethod."
2016-04-04T06:58:13.689265      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmRouter."
2016-04-04T06:58:13.733289      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmFederation."
2016-04-04T06:58:13.807554      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmAliasSet."
2016-04-04T06:58:13.871634      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Content Addressable Storage."
2016-04-04T06:58:13.924874      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Formats."
2016-04-04T06:58:13.995154      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Convert."
2016-04-04T06:58:13.998050      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Indices."
2016-04-04T06:58:14.025587      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Dump Files."
2016-04-04T06:58:14.107689      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Load Files."
2016-04-04T06:58:14.176232      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize In Box."
2016-04-04T06:58:14.225954      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Distributed References."
2016-04-04T06:58:14.292782      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Client Registrations."
2016-04-04T06:58:14.319699      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Client Rights."
2016-04-04T06:58:14.330420      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Client Rights Domain."
2016-04-04T06:58:14.356866      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Business Activities."
2016-04-04T06:58:14.411545      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Business Processes."
2016-04-04T06:58:14.444264      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Packages."
2016-04-04T06:58:14.493478      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Workitems."
2016-04-04T06:58:14.521836      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Workflows."
2016-04-04T06:58:14.559605      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Audit Trail."
2016-04-04T06:58:14.639303      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Clean Old Links."
2016-04-04T06:58:14.640511      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Compute Internal Type Tag Cache."
2016-04-04T06:58:14.696040      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize LastActionProcs."
2016-04-04T06:58:14.696473      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize User Types."
2016-04-04T06:58:14.696828      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Start Up - Phase 1."
2016-04-04T06:58:15.186812      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Start Up - Phase 2."
2016-04-04T06:58:15.666149      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Crypto Objects."
2016-04-04T06:58:15.677119      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Queue Views."
2016-04-04T06:58:15.678343      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Port Info Views."
2016-04-04T06:58:15.679532      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Global Cache Finalization."
2016-04-04T06:58:16.939100      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize National Language Character Translation."
2016-04-04T06:58:16.941815      21425[21425]    0000000000000000        [DM_CHARTRANS_I_TRANSLATOR_OPENED]info:  "Translator in directory ($DOCUMENTUM/product/7.2/install/external_apps/nls_chartrans) was added succesfully initialized.  Translator specifics: (Chararacter Translator: , Client Locale: (Windows :(4099), Version: 4.0), CharSet: ISO_8859-1, Language: English_US, UTC Offset: 0, Date Format:%2.2d/%2.2d/%2.2d %2.2d:%2.2d:%2.2d, Java Locale:en, Server Locale: (Linux :(8201), Version: 2.4), CharSet: UTF-8, Language: English_US, UTC Offset: 0, Date Format:%2.2d/%2.2d/%2.2d %2.2d:%2.2d:%2.2d, Java Locale:en, Shared Library: $DOCUMENTUM/product/7.2/install/external_apps/nls_chartrans/unitrans.so)"
2016-04-04T06:58:16.942377      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize LDAP setup."
2016-04-04T06:58:16.961767      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Distributed change-checking."
2016-04-04T06:58:17.022448      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Fulltext Plugin and Configuration."
2016-04-04T06:58:17.113147      21425[21425]    0000000000000000        [DM_FULLTEXT_T_QUERY_PLUGIN_VERSION]info:  "Loaded FT Query Plugin: $DOCUMENTUM/product/7.2/bin/libDsearchQueryPlugin.so, API Interface version: 1.0, Build number: HEAD; Sep 14 2015 07:48:06, FT Engine version: xPlore version 1.5.0020.0048"
2016-04-04T06:58:17.122313      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Distributed Content."
2016-04-04T06:58:17.125027      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Distributed Content Map."
2016-04-04T06:58:17.125467      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Distributed Content Digital Signatures."
2016-04-04T06:58:17.621156      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Acs Config List."
2016-04-04T06:58:17.621570      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmLiteSysObject."
2016-04-04T06:58:17.623010      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize dmBatchManager."
2016-04-04T06:58:17.624369      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Partition Scheme."
2016-04-04T06:58:17.627552      21425[21425]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Authentication Plugins."
2016-04-04T06:58:17.631207      21425[21425]    0000000000000000        [DM_SESSION_I_AUTH_PLUGIN_LOADED]info:  "Loaded Authentication Plugin with code 'dm_krb' ($DOCUMENTUM/dba/auth/libkerberos.so)."
2016-04-04T06:58:17.631480      21425[21425]    0000000000000000        [DM_SESSION_I_AUTH_PLUGIN_LOAD_INIT]info:  "Authentication plugin ( 'dm_krb' ) was disabled. This is expected if no keytab file(s) at location ($DOCUMENTUM/dba/auth/kerberos).Please refer the content server installation guide."
2016-04-04T06:58:17.638885      21425[21425]    0000000000000000        [DM_SERVER_I_START_SERVER]info:  "Docbase DOCBASE1 opened"
2016-04-04T06:58:17.639005      21425[21425]    0000000000000000        [DM_SERVER_I_SERVER]info:  "Setting exception handlers to catch all interrupts"
2016-04-04T06:58:17.639043      21425[21425]    0000000000000000        [DM_SERVER_I_START]info:  "Starting server using service name:  DOCBASE1"
2016-04-04T06:58:17.810637      21425[21425]    0000000000000000        [DM_SERVER_I_LAUNCH_MTHDSVR]info:  "Launching Method Server succeeded."
2016-04-04T06:58:17.818319      21425[21425]    0000000000000000        [DM_SERVER_I_LISTENING]info:  "The server is listening on network address (Service Name: DOCBASE1_s, Host Name: content_server_01 :V4 IP)"
2016-04-04T06:58:17.821615      21425[21425]    0000000000000000        [DM_SERVER_I_IPV6_DISABLED]info:  "The server can not listen on IPv6 address because the operating system does not support IPv6"
2016-04-04T06:58:19.301490      21425[21425]    0000000000000000        [DM_WORKFLOW_I_AGENT_START]info:  "Workflow agent master (pid : 21612, session 013f245a80000007) is started sucessfully."
2016-04-04T06:58:19.302601      21425[21425]    0000000000000000        [DM_WORKFLOW_I_AGENT_START]info:  "Workflow agent worker (pid : 21613, session 013f245a8000000a) is started sucessfully."
2016-04-04T06:58:20.304937      21425[21425]    0000000000000000        [DM_WORKFLOW_I_AGENT_START]info:  "Workflow agent worker (pid : 21626, session 013f245a8000000b) is started sucessfully."
2016-04-04T06:58:21.307256      21425[21425]    0000000000000000        [DM_WORKFLOW_I_AGENT_START]info:  "Workflow agent worker (pid : 21639, session 013f245a8000000c) is started sucessfully."
2016-04-04T06:58:22.307448      21425[21425]    0000000000000000        [DM_SERVER_I_START]info:  "Sending Initial Docbroker check-point "
2016-04-04T06:58:22.325337      21425[21425]    0000000000000000        [DM_MQ_I_DAEMON_START]info:  "Message queue daemon (pid : 21655, session 013f245a80000456) is started sucessfully."

Ok so this is the first and second pages I got as a result and that’s actually when I realised that the process I was talking about above was in fact the docbase process… So I displayed the third page of the more command and I got the following:

2016-04-05T10:04:28.305238      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_CURSOR_FAIL]error:  "In operation Exec an attempt to create cursor failed; query was: 'SELECT * FROM DM_DOCBASE_CONFIG_RV dm_dbalias_B , DM_DOCBASE_CONFIG_SV dm_dbalias_C  WHERE (dm_dbalias_C.R_OBJECT_ID=:dmb_handle AND dm_dbalias_C.R_OBJECT_ID=dm_dbalias_B.R_OBJECT_ID) ORDER BY dm_dbalias_B.R_OBJECT_ID,dm_dbalias_B.I_POSITION'; error from database system was: ORA-03114: not connected to ORACLE"
2016-04-05T10:04:28.305385      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:04:28.317505      21425[21425]    0000000000000000        [DM_SESSION_I_RETRYING_DATABASE_CONNECTION]info:  "The following error was encountered trying to get a database connection:  ORA-12541: TNS:no listener
2016-04-05T10:04:28.317591      21425[21425]    013f245a80000002        [DM_SESSION_I_RETRYING_DATABASE_CONNECTION]info:  "The following error was encountered trying to get a database connection:  ORA-12541: TNS:no listener
2016-04-05T10:04:58.329725      21425[21425]    0000000000000000        [DM_SESSION_I_RETRYING_DATABASE_CONNECTION]info:  "The following error was encountered trying to get a database connection:  ORA-12541: TNS:no listener
2016-04-05T10:04:58.329884      21425[21425]    013f245a80000002        [DM_SESSION_I_RETRYING_DATABASE_CONNECTION]info:  "The following error was encountered trying to get a database connection:  ORA-12541: TNS:no listener
2016-04-05T10:05:28.339052      21425[21425]    0000000000000000        [DM_SESSION_I_RETRYING_DATABASE_CONNECTION]info:  "The following error was encountered trying to get a database connection:  ORA-12541: TNS:no listener
2016-04-05T10:05:28.339143      21425[21425]    013f245a80000002        [DM_SESSION_I_RETRYING_DATABASE_CONNECTION]info:  "The following error was encountered trying to get a database connection:  ORA-12541: TNS:no listener
2016-04-05T10:05:49.077076      21425[21425]    0000000000000000        [DM_SESSION_I_RETRYING_DATABASE_CONNECTION]info:  "The following error was encountered trying to get a database connection:  ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
2016-04-05T10:05:49.077163      21425[21425]    013f245a80000002        [DM_SESSION_I_RETRYING_DATABASE_CONNECTION]info:  "The following error was encountered trying to get a database connection:  ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
2016-04-05T10:06:23.461495      21425[21425]    013f245a80000002        [DM_SESSION_W_RESTART_AGENT_EXEC]warning:  "The agent exec program has stopped running.  It will be restarted."
2016-04-05T10:06:48.830854      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:11:52.533340      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:16:52.574766      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:21:52.546389      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:26:52.499108      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:31:52.232095      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:36:57.700202      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:42:01.198050      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:47:01.411651      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:52:02.242612      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T10:57:11.886518      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T11:02:13.133405      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"
2016-04-05T11:07:15.364236      21425[21425]    0000000000000000        [DM_OBJ_MGR_E_FETCH_FAIL]error:  "attempt to fetch object with handle 3c3f245a60000210 failed"

As you can see above, the first error occurred at “2016-04-05T10:04:28.305385″ and that’s actually 0,000147s (=0,1ms) after an error while trying to execute an SQL query on the database for the operation Exec… So this must be linked… The database errors stopped one minute after the first one so the DB was available again. I quickly verified that using sqlplus.

We opened a SR on the EMC website to work on it with them but as far as I know, no solution were found. The only workaround that we found is that a simple restart of the docbase will cleanup these errors from the docbase log file and they will not appear afterwards but that’s pretty annoying to restart the docbase while it is actually working properly (jobs are running, dfc clients are OK, aso…) just because there is one error message printed every 5 minutes in the log file that is flooding our monitoring tool.

The problem with this error is that it can happen frequently if the Network isn’t really reliable, if the Database Listener isn’t always responsive or if anything else prevent Documentum from reaching the Database while it is doing something with the dm_docbase_config objects… Something that we didn’t try yet is to re-initialize the Content Server, to see if it can help to restore a proper log file. I think I’m gonna try that next time this issue occurs!

Edit: Re-initialize the Content Server isn’t helping :(.

Cet article Documentum story – Attempt to fetch object with handle 3c failed est apparu en premier sur Blog dbi services.

In this blog, I will talk about the Thumbnail Server. It’s a component of Documentum that you can install on the Content Server to generate… thumbnails! Basically what it does is that it will work in correlation with the ADTS/CTS in order to generate different kinds of previews of your document. For example jpeg_lres (low resolution) or jpeg_story (StoryBoard). You can define how many previews you want per document (1 preview per page, only the first page, aso…) and of which type. Then D2 can use these previews in a widget for the end-users to see what the document looks like. Apparently with D2 4.6, the Thumbnail Server and ADTS/CTS aren’t needed anymore to generate thumbnails but it was the case for D2 4.5 and previous versions.

That’s my first blog related to the Thumbnail Server because that’s actually a (the?) component of Documentum that is working pretty well without much issue so I absolutely wanted to explain the issue I faced and what has been done to solve that.

So let’s start with some background: I was working on a project where the TEST and PROD environments contain only one docbase (DOCBASE1) in addition to the Global Registry. In the DEV environment, there were three docbases for development purposes but the additional two (DOCBASE2 and DOCBASE3) were stopped for a few days because we were running a lot of Performance Tests with EMC and we wanted the results to reflect the TEST/PROD environments. At this point, the Thumbnail Server was working properly, previews were generated successfully, aso… Then to apply a change related to the Performance Tests, we had to restart the whole Content Server, including the Thumbnail Server since it has been installed on all Content Servers (HA environment). The change wasn’t related to the Thumbnail Server at all but we discovered a small bug because of this: after the restart, the Thumbnail Server wasn’t working anymore. Just like the JMS/ACS, there is a way to very quickly know if the Thumbnail Server is up & running or not and that can be checked by entering the following URL in a web browser: http(s)://content_server_01:port/thumbsrv/getThumbnail?

In our case, this URL wasn’t working anymore after the restart while it was working properly before and therefore I had to look at the Thumbnail log file. One important thing to note here is that the Thumbnail Server is bundled with Tomcat. For those of you who are used to work with Tomcat, your first reaction might be to open the file $TOMCAT_HOME/logs/catalina.out. For the Thumbnail, this would be the file: $DM_HOME/thumbsrv/container/logs/catalina.out. That’s what I did… But there were absolutely no useful information because only the Tomcat initialization is displayed in this file by default. The actual useful information for the Thumbnail Server are stored in the localhost log file… That’s the file I’m usually completely ignoring because there is less inside it than in the catalina.out file but apparently EMC took a difference approach!

So let’s take a look at what the log file is providing:

May 04, 2016 2:07:08 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_RESOURCES] Initialized resource strings
May 04, 2016 2:07:08 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Loaded Manifest entries from $DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/WEB-INF/lib/thumbsrv.jar
May 04, 2016 2:07:08 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail:
        Initializing Documentum Thumbnail Server 7.2.0000 , build: 0154
        Created on
        debug             = false
        ticket_timeout    = 300
        application path  = $DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/
May 04, 2016 2:07:08 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Read path to configuration file: $DOCUMENTUM/product/7.2/thumbsrv/conf/user.dat
May 04, 2016 2:07:08 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing Storage Area Manager...
May 04, 2016 2:07:17 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing Connection Manager...
May 04, 2016 2:07:17 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_STORAGE_AREA_MGR] Initialized storage area manager
May 04, 2016 2:07:17 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing crypto classes, key file at $DOCUMENTUM/dba/secure/aek.key
May 04, 2016 2:07:17 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE2
May 04, 2016 2:07:23 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE2
May 04, 2016 2:07:30 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE2
  ...
May 04, 2016 2:10:05 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE2
May 04, 2016 2:10:11 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE2
May 04, 2016 2:10:17 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initDefaultThumbnailsFromRepository: DOCBASE2 - 0.0
May 04, 2016 2:10:18 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE3
May 04, 2016 2:10:24 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE3
May 04, 2016 2:10:30 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE3
  ...
May 04, 2016 2:13:06 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE3
May 04, 2016 2:13:12 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [WARN] getRepositoryVersion cannot find the repository - DOCBASE3
May 04, 2016 2:13:18 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initDefaultThumbnailsFromRepository: DOCBASE3 - 0.0
May 04, 2016 2:13:19 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: getRepositoryVersion: DOCBASE1 - 1666666 - 7.2.0050.0214  Linux64.Oracle
May 04, 2016 2:13:19 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initDefaultThumbnailsFromRepository: DOCBASE1 - 7.2
May 04, 2016 2:13:20 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails
May 04, 2016 2:13:22 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails/formats, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails/formats
May 04, 2016 2:13:22 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails/types, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails/types
May 04, 2016 2:13:26 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: checkRuleVersion: the default thumbnail rule version - 6.0.0.101
May 04, 2016 2:13:26 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initRepositoryRules: repoitory=DOCBASE1, id=1666666
May 04, 2016 2:13:34 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_DEF_THUMB_MGR] Initialized default thumbnails manager
May 04, 2016 2:13:34 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Failed to get a session for DOCBASE2: DfNoServersException:: THREAD: pool-2-thread-1; MSG: [DM_DOCBROKER_E_NO_SERVERS_FOR_DOCBASE]error:  "The DocBroker running on host (content_server_01:1489) does not know of a server for the specified docbase (DOCBASE2)"; ERRORCODE: 100; NEXT: null
May 04, 2016 2:13:34 PM org.apache.catalina.core.ApplicationContext log
SEVERE: getThumbnail: [DM_TS_E_INIT_FORMATS_MGR] Falied to initialize formats.
DfNoServersException:: THREAD: pool-2-thread-1; MSG: [DM_DOCBROKER_E_NO_SERVERS_FOR_DOCBASE]error:  "The DocBroker running on host (content_server_01:1489) does not know of a server for the specified docbase (DOCBASE2)"; ERRORCODE: 100; NEXT: null
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.__AW_getDataFromDocbroker(ServerMapBuilder.java:171)
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.getDataFromDocbroker(ServerMapBuilder.java)
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.__AW_getMap(ServerMapBuilder.java:60)
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.getMap(ServerMapBuilder.java)
        at com.documentum.fc.client.impl.docbroker.DocbrokerClient.getServerMap(DocbrokerClient.java:152)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.__AW_updateServerChoices(ServerChoiceManager.java:159)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.updateServerChoices(ServerChoiceManager.java)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.updateServerChoicesIfNecessary(ServerChoiceManager.java:148)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.getServerChoices(ServerChoiceManager.java:47)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.getServerChoices(DocbaseConnection.java:273)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.__AW_establishNewRpcClient(DocbaseConnection.java:227)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.establishNewRpcClient(DocbaseConnection.java)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.__AW_open(DocbaseConnection.java:126)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.open(DocbaseConnection.java)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.<init>(DocbaseConnection.java:100)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.<init>(DocbaseConnection.java:60)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionFactory.newDocbaseConnection(DocbaseConnectionFactory.java:26)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.createNewConnection(DocbaseConnectionManager.java:180)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.getDocbaseConnection(DocbaseConnectionManager.java:110)
        at com.documentum.fc.client.impl.session.SessionFactory.newSession(SessionFactory.java:23)
        at com.documentum.fc.client.impl.session.PrincipalAwareSessionFactory.newSession(PrincipalAwareSessionFactory.java:44)
        at com.documentum.fc.client.impl.session.PooledSessionFactory.__AW_newSession(PooledSessionFactory.java:49)
        at com.documentum.fc.client.impl.session.PooledSessionFactory.newSession(PooledSessionFactory.java)
        at com.documentum.fc.client.impl.session.SessionManager.getSessionFromFactory(SessionManager.java:134)
        at com.documentum.fc.client.impl.session.SessionManager.newSession(SessionManager.java:72)
        at com.documentum.fc.client.impl.session.SessionManager.getSession(SessionManager.java:191)
        at com.documentum.thumbsrv.docbase.DocbaseConnectionMgr.getSessionForSection(DocbaseConnectionMgr.java:197)
        at com.documentum.thumbsrv.docbase.FormatMapperMgr.__AW_loadFormatsFromRepositories(FormatMapperMgr.java:95)
        at com.documentum.thumbsrv.docbase.FormatMapperMgr.loadFormatsFromRepositories(FormatMapperMgr.java)
        at com.documentum.thumbsrv.docbase.FormatMapperMgr.<init>(FormatMapperMgr.java:62)
        at com.documentum.thumbsrv.getThumbnail.__AW_init(getThumbnail.java:214)
        at com.documentum.thumbsrv.getThumbnail.init(getThumbnail.java)
        at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1266)
        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1185)
        at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1080)
        at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5015)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5302)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1095)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1617)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.__AW_run(FutureTask.java:262)
        at java.util.concurrent.FutureTask.run(FutureTask.java)
        at java.util.concurrent.ThreadPoolExecutor.__AW_runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
 
May 04, 2016 2:13:34 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_E_INIT_FAILED] Failed to initialize Documentum Thumbnail Server
May 04, 2016 2:13:34 PM org.apache.catalina.core.ApplicationContext log
INFO: Marking servlet getThumbnail as unavailable
May 04, 2016 2:13:34 PM org.apache.catalina.core.StandardContext loadOnStartup
SEVERE: Servlet /thumbsrv threw load() exception
javax.servlet.UnavailableException: [DM_TS_E_INIT_FAILED] Failed to initialize Documentum Thumbnail Server
        at com.documentum.thumbsrv.getThumbnail.__AW_init(getThumbnail.java:221)
        at com.documentum.thumbsrv.getThumbnail.init(getThumbnail.java)
        at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1266)
        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1185)
        at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1080)
        at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5015)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5302)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1095)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1617)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.__AW_run(FutureTask.java:262)
        at java.util.concurrent.FutureTask.run(FutureTask.java)
        at java.util.concurrent.ThreadPoolExecutor.__AW_runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
 

May 04, 2016 2:17:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:22:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:27:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:32:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:37:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:42:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable
May 04, 2016 2:47:16 PM org.apache.catalina.core.StandardWrapperValve invoke
INFO: Servlet getThumbnail is currently unavailable

Ok that’s quite a long trace and I already cut some parts because it was really too long. As you can see above, the very beginning of the startup is going on properly and then the Thumbnail Server is trying to contact the DOCBASE2. Hum why exactly is it trying to contact the DOCBASE2 while this docbase isn’t running? That’s a first strange thing. Then I cut a lot of lines but you can see that it is actually trying to do that for 3 minutes and it is not doing anything else during that time. Once the three minutes are over, it is trying to contact the DOCBASE3 for 3 minutes too (and failed again) and finally it contacted the DOCBASE1 which was the only docbase running at that time. This last one succeeded so the Thumbnail Server should have started properly… But actually it couldn’t because the first docbase that it tried fail (DOCBASE2) and that’s the docbase that will be used by the Thumbnail Server to open a session to retrieve some information.

So I did more tests to try to understand where the issue was and what could be done to solve that. First of all, the important thing to understand here is that the Thumbnail Server isn’t trying to contact all docbases ever installed on this Content Server. It will only try to contact the docbases that have been configured for it. To be more precise, the configuration of a docbase for the Thumbnail Server will update the file user.dat ($DM_HOME/thumbsrv/conf/user.dat) and add inside it the configuration for this specific docbase. When the Thumbnail Server will start, it will parse this file and see which docbases should be contacted during the startup.

Therefore my first test was to simply comment all lines related to DOCBASE2 and DOCBASE3 and then restart the Thumbnail Server. This is what I got:

May 04, 2016 3:23:32 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_RESOURCES] Initialized resource strings
May 04, 2016 3:23:32 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Loaded Manifest entries from $DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/WEB-INF/lib/thumbsrv.jar
May 04, 2016 3:23:32 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail:
        Initializing Documentum Thumbnail Server 7.2.0000 , build: 0154
        Created on
        debug             = false
        ticket_timeout    = 300
        application path  = $DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/
May 04, 2016 3:23:32 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Read path to configuration file: $DOCUMENTUM/product/7.2/thumbsrv/conf/user.dat
May 04, 2016 3:23:32 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing Storage Area Manager...
May 04, 2016 3:23:40 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing Connection Manager...
May 04, 2016 3:23:40 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_STORAGE_AREA_MGR] Initialized storage area manager
May 04, 2016 3:23:40 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: Initializing crypto classes, key file at $DOCUMENTUM/dba/secure/aek.key
May 04, 2016 3:23:40 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: getRepositoryVersion: DOCBASE1 - 1666666 - 7.2.0050.0214  Linux64.Oracle
May 04, 2016 3:23:40 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initDefaultThumbnailsFromRepository: DOCBASE1 - 7.2
May 04, 2016 3:23:41 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails
May 04, 2016 3:23:43 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails/formats, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails/formats
May 04, 2016 3:23:43 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: createDefaultThumnailsInRespoitory: repo=/System/ThumbnailServer/thumbnails/types, local=$DOCUMENTUM/product/7.2/thumbsrv/container/webapps/thumbsrv/thumbnails/types
May 04, 2016 3:23:47 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: checkRuleVersion: the default thumbnail rule version - 6.0.0.101
May 04, 2016 3:23:47 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: initRepositoryRules: repoitory=DOCBASE1, id=1666666
May 04, 2016 3:23:54 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_DEF_THUMB_MGR] Initialized default thumbnails manager
May 04, 2016 3:23:54 PM org.apache.catalina.core.ApplicationContext log
INFO: getThumbnail: [DM_TS_T_INIT_COMPLETE] Documentum Thumbnail Server initialization complete!

As you can see, the content is exactly the same except that the Thumbnail Server is now only contacting the DOCBASE1 (successfully again) but this time, the Thumbnail Server is up & running properly. Therefore commenting the lines in the file user.dat solved this issue… But that’s not enough. For me, this was clearly a bug and therefore I did more tests.

For the next test, I restored the lines related to DOCBASE2 and DOCBASE3 in the file user.dat and I tried to change the order of the lines inside this file… Because DOCBASE1 was the first docbase in this file, DOCBASE2 the second and DOCBASE3 the last one. So I thought that maybe the second docbase was the first to be initialized? And if the first docbase to be initialized (2nd in the file?) is responding, will the Thumbnail Server work?

Therefore I changed the order:

• Switching DOCBASE1 in second position, DOCBASE2 in first position, DOCBASE3 still in last => Not working, same issue, DOCBASE2 is the first to be initialised
• Switching DOCBASE1 in second position, DOCBASE2 in last position, DOCBASE3 in first position => Not working, same issue, DOCBASE3 is the first to be initialised
• …

This test wasn’t successful because there is absolutely no logic behind which docbase will be initialized first… Therefore I restored the initial values (1 in first, 2 in second, 3 in last) and I performed a last test: stopping the DOCBASE1 and starting the DOCBASE2 so that there is still one docbase to be running which would be the first to be initialized. After doing that, I restarted a last time the Thumbnail Server and the first docbase to be initialized was indeed the DOCBASE2, which was running. Then it tried to initialise the DOCBASE1 and DOCBASE3 which weren’t running and therefore it failed. BUT in the end, the Thumbnail Server was able to start properly, even if it took 6 minutes + ~20 seconds instead of 20 seconds to start.

With all these results, I opened a Service Request on the EMC website and they were able to find the root cause for both issues (why it is looping for 3 minutes on nothing and why it is sometimes not able to start when one docbase is properly responding). In the end, they provided us a hotfix that has been incorporated in newer versions of the software normally and this hotfix fixed both issues properly.

Cet article Documentum story – Thumbnail Server not starting if docbases are down est apparu en premier sur Blog dbi services.

In this blog, I will show you how to setup the Primary DSearch and IndexAgent in HTTPS for xPlore 1.5. The documentation about that is available on the EMC WebSite as always and the pdf name is: “xPlore 1.5 Installation Guide.pdf”. The reason why I wanted to wrote this blog is because the documentation is not too bad but there are still some missing parts and without these parts, your configuration will just not work properly. Moreover, I think it is better to have a concrete an complete example rather than just reading a PDF with some information spread on 40 different pages.

So let’s begin the configuration. The first thing to do is obviously to login to the Full Text Server where you xPlore 1.5 is installed. For this blog, I will use /app/xPlore as the installation folder of xPlore. I will also use a Self-Signed SSL Certificate with a Certificate Chain composed of a Root and Gold CA. So let’s import the Certificate Chain in xPlore (following commands assume all certificates are available under “/app/xPlore/jboss7.1.1/certs”):

[xplore@xplore_server_01 ~]$ /app/xPlore/java64/1.7.0_72/bin/keytool -import -trustcacerts -alias root_ca -keystore /app/xPlore/java64/1.7.0_72/jre/lib/security/cacerts -file /app/xPlore/jboss7.1.1/certs/Internal_Root_CA.cer
Enter keystore password:
[xplore@xplore_server_01 ~]$ /app/xPlore/java64/1.7.0_72/bin/keytool -import -trustcacerts -alias gold_ca -keystore /app/xPlore/java64/1.7.0_72/jre/lib/security/cacerts -file /app/xPlore/jboss7.1.1/certs/Internal_Gold_CA1.cer
Enter keystore password:

All java processes using /app/xPlore/java64/1.7.0_72/bin/java will now trust the Self-Signed SSL Certificate because the Certificate Chain is trusted. When this is done, shutdown all xPlore processes (Primary DSearch and IndexAgent(s)) and let’s configure the Primary DSearch in HTTPS:

[xplore@xplore_server_01 ~]$ /app/xPlore/scripts/startstop stop
  ** Indexagent_DOCBASE1 has been stopped successfully
  ** PrimaryDsearch has been stopped successfully
[xplore@xplore_server_01 ~]$ cd /app/xPlore/dsearch/admin
[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IS \
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" \
    -storepass K3ySt0r3P4ssw0rd -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" \
    -isname PrimaryDsearch -ianame Indexagent_DOCBASE1

Some remarks:

• “-enable” means that HTTPS will be enabled and HTTP will be disabled. If you want both to be enabled, use the “-dual” option instead
• “-component” defines which component should be configured with this command. It can be “IS” (IndexServer), “IA” (IndexAgent) or “ALL” (IndexServer and IndexAgent)
• “-isname” defines the name of the IndexServer/Primary DSearch that you installed
• “-ianame” defines the name of the IndexAgent that you installed

Now what happen if you have more than one IndexAgent on the same server? Well the script isn’t smart enough for that and that’s the reason why I didn’t put “ALL” above but just “IS”. You might also noticed that I defined the “-ianame” parameter with “Indexagent_DOCBASE1”. This is because even if we are configuring the Primary DSearch in HTTPS, all IndexAgents have a reference in a configuration file that defines which port and protocol the IA should use to connect to the DSearch and if this isn’t setup properly, the IA will not be able to start.

Now the IndexServer is configured in HTTPS so let’s do the same thing for the IndexAgent:

[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IA \
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" \
    -storepass K3ySt0r3P4ssw0rd -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" \
    -ianame Indexagent_DOCBASE1 -iaport 9200

As you can see above, this time no need to add the “-isname” parameter, it is not needed for the IndexAgent(s). Let’s say that you have a second IndexAgent for the docbase named DOCBASE2 , then you also have to execute the above command for this second indexAgent:

[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IA \
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" \
    -storepass K3ySt0r3P4ssw0rd -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" \
    -ianame Indexagent_DOCBASE2 -iaport 9220

In case you didn’t know, yes each IndexAgent need at least 20 consecutive ports (so 9200 to 9219 for Indexagent_DOCBASE1 // 9220 to 9239 for Indexagent_DOCBASE2).

When configuring the IndexServer in HTTPS, I specified the “-ianame”. This is, like I said before, because there is a reference somewhere to the Protocol/Port used. This reference has been updated properly for Indexagent_DOCBASE1 normally but not for Indexagent_DOCBASE2. Therefore you need to do that manually:

[xplore@xplore_server_01 admin]$ grep -B1 -A10 dsearch_qrserver_protocol /app/xPlore/jboss7.1.1/server/DctmServer_Indexagent_DOCBASE2/deployments/IndexAgent.war/WEB-INF/classes/indexagent.xml
        <parameter>
            <parameter_name>dsearch_qrserver_protocol</parameter_name>
            <parameter_value>HTTP</parameter_value>
        </parameter>
        <parameter>
            <parameter_name>dsearch_config_host</parameter_name>
            <parameter_value>xplore_server_01</parameter_value>
        </parameter>
        <parameter>
            <parameter_name>dsearch_config_port</parameter_name>
            <parameter_value>9300</parameter_value>
        </parameter>

Just open this file and update the few lines that I printed above by replacing “HTTP” with “HTTPS” and “9300” with “9302” and that’s it. If you have several IndexAgents, then you need to do that for all of them.

The next step is to login to the Content Server (e.g.: ssh dmadmin@content_server_01) and update some properties in the docbase:

[dmadmin@content_server_01 ~]$ iapi DOCBASE1 -Udmadmin -Pxxx
 

        EMC Documentum iapi - Interactive API interface
        (c) Copyright EMC Corp., 1992 - 2015
        All rights reserved.
        Client Library Release 7.2.0050.0084


Connecting to Server using docbase DOCBASE1
[DM_SESSION_I_SESSION_START]info:  "Session 013f245a8014087a started for user dmadmin."


Connected to Documentum Server running Release 7.2.0050.0214  Linux64.Oracle
Session id is s0
API> ?,c,select r_object_id from dm_ftengine_config where any lower(param_value) = lower('xplore_server_01');
r_object_id
----------------
083f245a800052ae
(1 row affected)

API> fetch,c,083f245a800052ae
...
OK
API> dump,c,l
...
USER ATTRIBUTES

  object_name                     : DSearch Fulltext Engine Configuration
  ...
  param_name                   [0]: dsearch_qrserver_protocol
                               [1]: dsearch_qrygen_mode
                               [2]: dsearch_qrserver_target
                               [3]: dsearch_qrserver_port
                               [4]: dsearch_config_port
                               [5]: dsearch_qrserver_host
                               [6]: dsearch_domain
                               [7]: dsearch_config_host
                               [8]: query_plugin_mapping_file
                               [9]: load_balancer_enabled
                              [10]: ft_wildcards_mode
  param_value                  [0]: HTTP
                               [1]: both
                               [2]: /dsearch/IndexServerServlet
                               [3]: 9300
                               [4]: 9300
                               [5]: xplore_server_01
                               [6]: DOCBASE1
                               [7]: xplore_server_01
                               [8]: /app/dctm/server/fulltext/dsearch/dm_AttributeMapping.xml
                               [9]: true
                              [10]: trailing_implicit
  ...

You might have noticed that I specified a WHERE clause on the select to find the r_object_id of the xPlore Server. That’s basically because in this case, there are two xPlore Servers in a HA setup (the parameter load_balancer_enabled is setup to true) but I just want to update the right object. So let’s update the parameters highlighted above (“dsearch_qrserver_protocol”, “dsearch_qrserver_port” and “dsearch_config_port”):

API> set,c,l,param_value[0]
SET> HTTPS
...
OK
API> set,c,l,param_value[3]
SET> 9302
...
OK
API> set,c,l,param_value[4]
SET> 9302
...
OK
API> save,c,l
...
OK

With these steps, we updated the dm_ftengine_config object. The next step is to also update the URL of the IndexAgent:

API> ?,c,select r_object_id from dm_server_config
r_object_id
----------------
3d3f245a80000102
3d3f245a80003796
(2 rows affected)

API> fetch,c,3d3f245a80000102
...
OK
API> dump,c,l
...
USER ATTRIBUTES

  object_name                     : DOCBASE1
  ...
  app_server_name              [0]: do_method
                               [1]: do_mail
                               [2]: do_bpm
                               [3]: xplore_server_01_9200_IndexAgent
                               [4]: xplore_server_02_9200_IndexAgent
  app_server_uri               [0]: https://content_server_01:9082/DmMethods/servlet/DoMethod
                               [1]: https://content_server_01:9082/DmMail/servlet/DoMail
                               [2]: https://content_server_01:9082/bpm/servlet/DoMethod
                               [3]: http://xplore_server_01:9200/IndexAgent/servlet/IndexAgent
                               [4]: http://xplore_server_02:9200/IndexAgent/servlet/IndexAgent
  ...

So we setup the IndexAgent installed on xplore_server_01 in HTTPS and therefore we also need to update the URL referenced in the docbase. That’s actually one of the things that aren’t in the official documentation at the moment. This is done as before:

API> set,c,l,app_server_uri[3]
SET> https://xplore_server_01:9202/IndexAgent/servlet/IndexAgent
...
OK
API> save,c,l
...
OK

As you saw above, this is an environment with two dm_server_config objects (two Content Servers) and two IndexAgents. Above, we setup the Primary DSearch and IndexAgent installed on xplore_server_01 in HTTPS. The dm_ftengine_config for this Primary DSearch has been updated and the URLs defined in one dm_server_config object have been updated too. But the same references are present in the second dm_server_config and therefore you also need to do that for the second one (3d3f245a80003796 in this case). Same steps so just repeat with the other r_object_id!

Ok so now all objects in the docbase have been updated successfully. Therefore return on the xPlore Server, clean the caches and start everything again:

[xplore@xplore_server_01 ~]$ rm -rf /app/xPlore/jboss7.1.1/server/DctmServer_*/tmp/work/*
[xplore@xplore_server_01 ~]$ /app/xPlore/scripts/startstop start
  ** PrimaryDsearch has been started successfully
  ** Indexagent_DOCBASE1 has been started successfully

As said before, some of these steps aren’t described/explained in the official documentation and that will lead you to a non-working situation… In addition to that, there are some bugs impacting the proper behavior of the Primary DSearch and/or the Index Agents when configured in HTTPS. We reported these bugs to EMC which was able to provide a fix for some of them and include that in a later patch but as you know it is not always possible to upgrade or patch your environment. For example with the CS 7.2 P02 or P05, the searches will NOT work against a DSearch in HTTPS (corrected in the P08 or P09 if I remember correctly) but I will not describe that in this blog. If you are facing an issue with the IndexAgents not responding in HTTPS, please check this blog.

Cet article Documentum story – Setup the DSearch & IndexAgent in HTTPS (xPlore) est apparu en premier sur Blog dbi services.

In one of our project, we were using CS 7.2 P02 (upgraded later with P05) in correlation with xPlore 1.5 P02. With these versions, we wanted to setup the IndexAgents in HTTPS to have a completely secured environment. We choose to follow the xPlore documentation and use the Groovy script that EMC is now providing to facilitate the setup in HTTPS of the IndexAgents & Primary DSearch.

The first thing to note is that the documentation is describing some parts of how to setup the SSL in xPlore but it is not the complete setup and some other parts are missing! You can use one of my previous blog to know how to setup an IndexAgent in HTTPS using the Groovy Script: click here. The second thing to note is that after setting our IndexAgents in HTTPS, we faced an issue where the status of the IndexAgents weren’t available anymore (it was working in HTTP)! After following the steps described in the other blogs, please take a look at the explications below in case you are having the same issue.

So after doing what is described in the link above, the status of the IndexAgents in Documentum Administrator was “Not responding” and we weren’t able to start/stop them. Same behavior via iapi, the status command returned a 200 code. Unlike the HTTP response codes, the IndexAgents codes are:

• 0 => Everything is running & working
• 100 => The JBoss instance hosting the IndexAgent is running BUT the IndexAgent isn’t started (E.g.: not in the status “Start in Normal Mode”)
• 200 => The JBoss instance hosting the IndexAgent isn’t running OR not responding

Now to check the status of the IndexAgent, you can do the following using iapi:

API> ?,c,select object_name, index_name from dm_ftindex_agent_config;
object_name                                      index_name
-----------------------------------------------  ------------------------------------------------
xplore_server_01_9200_IndexAgent                  DOCBASE_ftindex_01
(1 row affected)

API> apply,c,,FTINDEX_AGENT_ADMIN,NAME,S,DOCBASE_ftindex_01,AGENT_INSTANCE_NAME,S,xplore_server_01_9200_IndexAgent,ACTION,S,status
...
q0
API> next,c,q0
...
OK
API> dump,c,q0
...
USER ATTRIBUTES
 
  name                         [0]: xplore_server_01_9200_IndexAgent
  status                       [0]: 200

SYSTEM ATTRIBUTES


APPLICATION ATTRIBUTES
 

INTERNAL ATTRIBUTES
 

API> exit
Bye

You can see above a status 200… I can assure you that the JBoss instance and IndexAgent are up, running & working properly (we had to start it using the GUI since it was the only way to check the status)! Then why is a status code 200 returned? Well that’s the purpose of this blog!

Like I said before, we choose to setup the SSL for the IndexAgents using the Groovy script and that’s the reason why we faced this issue in the first place… The Groovy script is setup to automatically update the file “standalone.xml” to add some configuration for the SSL. This is what the script will add:

[xplore@xplore_server_01 ~]$ grep -A4 "connector name=\"https\"" /app/xPlore/jboss7.1.1/server/DctmServer_Indexagent_DOCBASE/configuration/standalone.xml
    <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">
        <ssl name="https" password="***********"
            certificate-key-file="/app/xPlore/dsearch/admin/./../../jboss7.1.1/server/DctmServer_Indexagent_DOCBASE/configuration/my.keystore"
            cipher-suite="TLS_RSA_WITH_AES_128_CBC_SHA"/>
    </connector>

Now what is wrong with this configuration? It actually looks quite good… We were working on this specific issue for a few months with EMC without any special findings (no errors in logs, nothing in the DFC/RPC/HTTP_POST traces, aso…) when we noted that the Groovy script only added a “normal” Cipher to the standalone.xml file but our Content Servers were setup at that time using the default SSL mode of Documentum which is commonly named “Anonymous SSL”. As shown above, the SSL Cipher that the Groovy script added is “TLS_RSA_WITH_AES_128_CBC_SHA”. The problem is that the JBoss instance hosting the IndexAgent is restricting the Ciphers available but the docbase on the other side isn’t able to use this Cipher because it is using the “Anonymous SSL”. Therefore there is mismatch in the Ciphers available and the communication can’t be completed.

So what is the solution to that? Well we simply need to extend the Cipher list to find a Cipher that is available for both the docbase and the IndexAgent. To solve this issue, you just have to replace the “cipher-suite” as follow:

[xplore@xplore_server_01 ~]$ grep -A4 "connector name=\"https\"" /app/xPlore/jboss7.1.1/server/DctmServer_Indexagent_DOCBASE/configuration/standalone.xml
    <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">
        <ssl name="https" password="***********"
            certificate-key-file="/app/xPlore/dsearch/admin/./../../jboss7.1.1/server/DctmServer_Indexagent_DOCBASE/configuration/my.keystore"
            cipher-suite="TLS_DH_anon_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA"/>
    </connector>

With this setup, you just have to restart the JBoss instance hosting the IndexAgent, reinitialize the Content Server and after doing that, the status will be available again:

API> apply,c,,FTINDEX_AGENT_ADMIN,NAME,S,DOCBASE_ftindex_01,AGENT_INSTANCE_NAME,S,xplore_server_01_9200_IndexAgent,ACTION,S,status
...
q0
API> next,c,q0
...
OK
API> dump,c,q0
...
USER ATTRIBUTES

  name                         [0]: xplore_server_01_9200_IndexAgent
  status                       [0]: 100

SYSTEM ATTRIBUTES
 

APPLICATION ATTRIBUTES
 

INTERNAL ATTRIBUTES


API> apply,c,,FTINDEX_AGENT_ADMIN,NAME,S,DOCBASE_ftindex_01,AGENT_INSTANCE_NAME,S,xplore_server_01_9200_IndexAgent,ACTION,S,start
...
q1
API> next,c,q1
...
OK
API> dump,c,q1
...
USER ATTRIBUTES

  name                         [0]: xplore_server_01_9200_IndexAgent
  status                       [0]: 0

SYSTEM ATTRIBUTES


APPLICATION ATTRIBUTES
 

INTERNAL ATTRIBUTES
 

API> exit
Bye

As you might know, the Cipher that we added contains “_anon_”. This means Anonymous (no authentication) and this also means that it is not the most secure of all Ciphers but this is necessary if you are using the default SSL mode in Documentum (without specifically setting up your own SSL Certificates). That’s why EMC recommends you to setup a non-Anonymous SSL mode but that’s a little bit more difficult to setup than just using the default one. Might be the subject of another blog

Cet article Documentum story – Status of IndexAgent not visible in HTTPS (xPlore) est apparu en premier sur Blog dbi services.

In a previous blog (click here), I described how to setup the DSearch and IndexAgent(s) in HTTPS using the Groovy script provided by EMC in newer versions of Documentum. This script is quite cool because it will allow you to automatically do some stuff like updating the xml configuration files, put the java keystore in the right location and configure JBoss to use it, aso… It also allows you to quickly apply/revert changes to do some tests for example. Several months ago when I first used this script, I faced some issues. I will present in this blog some of the errors I’ve seen and what was the reason behind that.

Actually I have a “funny” story related to these errors: several months after seeing these errors for the first time (and telling EMC about it with complete description of the issue and according solution), I worked on another new issue related to the Full Text Server with the help of EMC. Our EMC contact was trying to replicate our new issue and for that purpose, he had to setup the IndexAgent in SSL and he faced an error with the SSL setup… So he opened a ticket with the EMC Engineering Team to ask them to check why this was happening. In the meantime, I also asked him to share the error he was facing with me because I was curious… Of course, it turned out to be one of the errors I saw and shared with EMC several months before that. Therefore I provided him the solution to correct this bug and he was able to install the IndexAgent in SSL. I found that funny how sometimes you can actually be the one helping EMC with their own products ^^.

So let’s start with some of the errors I saw:

1. While configuring a DSearch in HTTPS

[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IS \
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" \
    -storepass $JKS_Password -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" \
    -isname PrimaryDsearch

Component: IS
====== Configure Index Server: PrimaryDsearch ======
Copy: /app/xPlore/jboss7.1.1/certs/xplore_server_01.jks to: ./../../jboss7.1.1/server/DctmServer_PrimaryDsearch/configuration/my.keystore
Copy: /app/xPlore/config/indexserverconfig.xml to: /app/xPlore/config/indexserverconfig.xml.bakHttp
Primary instance: PrimaryDsearch
Host name: xplore_server_01
Http port: 9300, https port: 9302
Updated: /app/xPlore/config/indexserverconfig.xml
Copy: ./../../jboss7.1.1/server/DctmServer_PrimaryDsearch/configuration/standalone.xml to: ./../../jboss7.1.1/server/DctmServer_PrimaryDsearch/configuration/standalone.xml.bakHttp
Exception in thread "main" java.lang.IndexOutOfBoundsException: No group 6
        at java.util.regex.Matcher.start(Matcher.java:374)
        at java.util.regex.Matcher.appendReplacement(Matcher.java:831)
        at java.util.regex.Matcher.replaceFirst(Matcher.java:955)
        at java.lang.String.replaceFirst(String.java:2119)
        at java_lang_String$replaceFirst.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
        at ConfigSSL.configIS(ConfigSSL.groovy:624)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
        at ConfigSSL$_run_closure1.doCall(ConfigSSL.groovy:333)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
        at groovy.lang.Closure.call(Closure.java:412)
        at groovy.lang.Closure.call(Closure.java:425)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1376)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1348)
        at org.codehaus.groovy.runtime.dgm$162.invoke(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
        at ConfigSSL.run(ConfigSSL.groovy:331)
        at groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:266)
        at groovy.lang.GroovyShell.run(GroovyShell.java:229)
        at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.callEntrance(DSearchAdminCLI.java:83)
        at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.main(DSearchAdminCLI.java:22)

2. While configuring an IndexAgent in HTTPS

[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IA \
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" \
    -storepass $JKS_Password -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" \
    -ianame Indexagent -iaport 9200

Component: IA
====== Configure Index Agent: Indexagent ======
Copy: /app/xPlore/jboss7.1.1/certs/xplore_server_01.jks to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/my.keystore
Copy: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml.bakHttp
Exception in thread "main" java.lang.IndexOutOfBoundsException: No group 6
        at java.util.regex.Matcher.start(Matcher.java:374)
        at java.util.regex.Matcher.appendReplacement(Matcher.java:831)
        at java.util.regex.Matcher.replaceFirst(Matcher.java:955)
        at java.lang.String.replaceFirst(String.java:2119)
        at java_lang_String$replaceFirst.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
        at ConfigSSL.configIA(ConfigSSL.groovy:888)
        at ConfigSSL$configIA.callCurrent(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
        at ConfigSSL.run(ConfigSSL.groovy:338)
        at groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:266)
        at groovy.lang.GroovyShell.run(GroovyShell.java:229)
        at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.callEntrance(DSearchAdminCLI.java:83)
        at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.main(DSearchAdminCLI.java:22)

3. While configuring an IndexAgent in HTTPS: another one

[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IA \
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" \
    -storepass Test4Pass+word$ -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" \
    -ianame Indexagent -iaport 9200
    
Component: IA
====== Configure Index Agent: Indexagent ======
Copy: /app/xPlore/jboss7.1.1/certs/xplore_server_01.jks to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/my.keystore
Copy: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml.bakHttp
Exception in thread "main" java.lang.IllegalArgumentException: Illegal group reference
      at java.util.regex.Matcher.appendReplacement(Matcher.java:808)
      at java.util.regex.Matcher.replaceFirst(Matcher.java:955)
      at java.lang.String.replaceFirst(String.java:2119)
      at java_lang_String$replaceFirst.call(Unknown Source)
      at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
      at ConfigSSL.configIA(ConfigSSL.groovy:888)
      at ConfigSSL$configIA.callCurrent(Unknown Source)
      at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
      at ConfigSSL.run(ConfigSSL.groovy:338)
      at groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:266)
      at groovy.lang.GroovyShell.run(GroovyShell.java:229)
      at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.callEntrance(DSearchAdminCLI.java:83)
      at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.main(DSearchAdminCLI.java:22)

4. The solution?

You might have noticed above that each time the exception is thrown just after the backup of the file standalone.xml. So what is the Groovy script doing in this file exactly which can cause these errors? Well when configuring JBoss in HTTPS, there is only one thing to do in this file: add a new connector to enable the HTTPS and to point to the correct keystore/password. Based on this information, it is really easy to find the reason for all these exceptions. In addition to that, you might also have noticed that in the two first points above, the password of the keystore was “$JKS_Password” while in the last command, the password is in clear text (Test4Pass+word$). So what is that environment variable $JKS_Password? Well we had to use IQs to install all these environments with logging tools to review and validate the installation and since these logging tools are storing everything displayed to the console, we were using environment variable to store our password so it is not displayed on the console or on the history of the “xplore” user. This is done as follow:

[xplore@xplore_server_01 ~]$ stty -echo; read JKS_Password; stty echo
                                                         => Enter the password here, it will be invisible
[xplore@xplore_server_01 ~]$
[xplore@xplore_server_01 ~]$ echo $JKS_Password
Test4Pass+word$
[xplore@xplore_server_01 ~]$

Basically the first command will ask you to type the password and when you press Enter, the prompt is returned and the environment variable $JKS_Password contains your password.

Now that this is clear, what is the solution to correct the errors mentioned above? Well the solution is the same for all errors above and this solution is quite simple: don’t use any annoying special characters in your passwords (like “$” or “\”). The problem with these characters is that the Groovy script isn’t able to manage them at the moment and therefore creating a password that contains one of these characters might result is errors and/or strange results as shown above.

Even if annoying special characters should be avoided to prevent these errors, you can continue to use some other special characters if you want to use a strong password like “+”, “-“, “?” and you can also continue to use the environment variable if you want. This is an example of the command that is working:

[xplore@xplore_server_01 ~]$ echo $JKS_Password
Test4Pass+word
[xplore@xplore_server_01 ~]$ 
[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IA \
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" \
    -storepass $JKS_Password -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" \
    -ianame Indexagent -iaport 9200

Component: IA
====== Configure Index Agent: Indexagent ======
Copy: /app/xPlore/jboss7.1.1/certs/xplore_server_01.jks to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/my.keystore
Copy: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml.bakHttp
Updated: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml
Copy: ./../../watchdog/config/dsearch-watchdog-config.xml to: ./../../watchdog/config/dsearch-watchdog-config.xml.bakHttp
Updated: ./../../watchdog/config/dsearch-watchdog-config.xml
Copy: ./../../jboss7.1.1/server/DctmServer_Indexagent/deployments/IndexAgent.war/WEB-INF/classes/indexagent.xml to: ./../../jboss7.1.1/server/DctmServer_Indexagent/deployments/IndexAgent.war/WEB-INF/classes/indexagent.xml.bakHttp
Updated: ./../../jboss7.1.1/server/DctmServer_Indexagent/deployments/IndexAgent.war/WEB-INF/classes/indexagent.xml
SSL is enabled, clear jboss cache and start server to verify result.

Once this execution completed successfully, you can just continue the SSL setup as described in this blog linked at the top and you shouldn’t face any other issue.

As soon as you put a “$” in your password and use this password in the command line (or use the environment variable), you will in the end face an error that will depend on whether or not you used the environment variable to hide it. As an additional note, when I was working on these errors, I did a lot of tests and I already saw the script working (or so it seemed) even with a password containing a “$” (depending on how it is written, with single quotes, double quotes, escaped, aso…) when put directly in the command line. But when I tried to start the IndexAgent, it wasn’t able to start… Therefore I took a look at the file standalone.xml and it was corrupted! The Groovy script screwed up my standalone.xml file and I had to revert a backup to be able to restart it. Hopefully the script is smart enough to backup all files before doing anything…

In case you absolutely want a “$” in your password, then it is possible but you will have to do one step manually. First you need to execute the script with a DUMMY password without any special character:

[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IA \
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" \
    -storepass DUMMY_PASSWORD -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" \
    -ianame Indexagent -iaport 9200
...

Once done, you can simply open the file standalone.xml and replace “DUMMY_PASSWORD” with the real password that contains a “$”. Restart the IndexAgent and this time, it should work.

Cet article Documentum story – SSL Password for the JKS of the DSearch & IndexAgent (xPlore) est apparu en premier sur Blog dbi services.

This first day at Momentum 2016

Normally I should write the second one as we started yesterday with a partner session where we got some information. One of these news was that EMC had more than 400 partners a few years ago and today this has been reduced to less than 80 and dbi services is still one of them. For us it is a good news, I hope this is also one for our current and future customers.

Today the different sessions, a part of the key notes hold by Rohit Ghai, were more related to customer experience, solutions ECD partners can propose, business presentations, description of particular challenges that companies had to face and how they dealt with it without presenting technical details.
As I am more on the technical side, this was more for my culture, I would say.

In the keynote we learned that with Documentum 7.3 cost saving will increase. For instance, PostgreSQL can be used with Document 7.3, the upgrade will be faster, aso… Since time is money…
PostgreSQL can be an interesting subject as dbi services is also active in this database and I will have to work with our DB experts to see what we have to test, how and find out the pro and cons using PostgreSQL on a technical point of view, as for sure the license cost will decrease. I planned, no I have, to go to the technical session tomorrow about “What’s new in Documentum 7.3″.

I also took the opportunity to discuss with some Dell EMC partners to learn more about the solutions they propose. For instance I was able to talk with Neotys people to understand what their product can bring us compared to JMeter or LoadRunner which we or our customers are using to do the load tests. Having a better view of possible solutions in this area can help me in case some customers have specific requirements to help him choose the best tool.
I also had a chat with Aerow and they showed me how ARender4Documentum is working and how fast “big” documents can be displayed in their html5 viewer. So even if the fist day cannot be viewed as a technical day, I actually learned a lot.
In this kind of event, what I find cool too, is that you can meet people, for instance at lunch time around a table and start talking about your/their experiences, your/their concerns, solutions, aso… So today we had a talk about cloud (private, public) and what does this means in case you have a validated system.

So let’s see what will happen tomorrow, the day where more technical information will be shared.

Note: Read Morgan’s blog where you can find technical stuff. You know I felt Morgan frustrated today as he could not “eat” technical food

Cet article Momentum16 – Day 1 – Feelings est apparu en premier sur Blog dbi services.

As Gérard explained in its first blog today was the first day not specific to the partners. I had the opportunity to attend some business centric (not really) interesting sessions in the morning. Then the morning ended and the afternoon begun with two keynotes: “Dell EMC Opening Keynote” and “Digital Transformation Keynote”. Finally I was able to attend a hands on session on InfoArchive and that’s what I will talk about in this blog since that’s the only piece of technical information I was able to get today.

Like every other events, there are exhibitions and exhibitors that are showing what they are doing around EMC in their booths. Of course there is also a booth regarding the InfoArchive solution if you want to talk to some EMC experts and I think that’s a pretty good way to see and understand what this solution is doing.

EMC InfoArchive is a unified enterprise archiving platform that stores related structured data and unstructured content in a single consolidated repository. This product enables corporations to preserve the value of enterprise information in a single, compliant, and easily accessible unified archive. Basically, that’s a place where you can store your content to be archived on a low price storage because this kind of information is usually kept only for legal constraints (read only) and don’t need to be accessed very often.

InfoArchive is composed of three components: an included Web Server, a Server (core of the application) and finally a Database (it is using an Xhive Database (XML), just like xPlore). Therefore you can very easily provide an XML file that will be used as an import file and that contains content to be archived by InfoArchive. Basically everything that can be transformed to an XML format (metadata/content) can be put inside InfoArchive. This solution provides some default connectors like:

• Documentum
• SharePoint (can archive documents and/or complete sites)
• SAP
• …

These default connectors are great but if that’s not enough, then you can just define your own with the information that you want to store and how you want to index them, transform them, aso… And of course this is defined in XML files. At the moment, this configuration can be a little bit scary since it is all done manually but I heard that a GUI configuration might be coming soon if it’s not in the version 4.2 already? InfoArchive is apparently fully web-based and  therefore based on a discussion I had with an EMC colleague, it should technically be possible to archive all the content of an SharePoint Site for example and then accessing this content from Documentum or any other location as long as it is using web-based requests to query the InfoArchive.

During the hands on session (first time working with InfoArchive for me), I had to create a new application/holding that can be used to archive Tweets. At the end of the one and a half hour, I had successfully created my application and I was able to search for Tweets based on their creationDate, userName, hashTags, retweetCount, aso… That was done actually pretty easily by following the help guide provided by EMC (specific to this use case) but if you don’t have this help guide, you better be an InfoArchive expert because you need to know each and every one of the XML tags that need to be added and where to add them to get something working properly.

See you tomorrow for the next blog with hopefully more technical stuff to share.

Cet article Momentum16 – Day 1 – InfoArchive first approach est apparu en premier sur Blog dbi services.

The aim of this blog is to provide you a way to configure the JMS Logs in order to align all applications logging with date information, log rotation and retention. Some changes have to be done on the jboss container as well as on the log4j utility for each deployed JMS applications (acs.ear, serverapps.ear and bpm.ear).

General configuration

First, go to the JMS configuration at $DOCUMENTUM_SHARED/jboss7.1.1/server/DctmServer_MethodServer/configuration/
The jboss version may vary depending on your Content Server version.
Do a backup of the standalone file like follow:

cp standalone.xml standalone.xml_$(date +%Y%m%d).log

Then edit the file standalone.xml by replacing each pattern-formatter with the following configuration:

<pattern-formatter pattern="%d{yyyy-MM-dd HH:mm:ss,SSS z} %-5p [%c] (%t) %s%E%n"/>

Note that you can change this setting which will change the way the log file will look like, but try to be consistent with other environments and components.
Now go to application deployments: $DOCUMENTUM_SHARED/jboss7.1.1/server/DctmServer_MethodServer/deployments
Once again, depending your Content Server version you could have to go into deploy instead of deployments.

For ServerApps.ear

Backup the current log4j.properties file:

cp ./ServerApps.ear/APP-INF/classes/log4j.properties ./ServerApps.ear/APP-INF/classes/log4j.properties_$(date +%Y%m%d).log

Then edit ./ServerApps.ear/APP-INF/classes/log4j.properties and set it like this:

log4j.rootCategory=WARN, A1, F1
 log4j.category.MUTE=OFF
 log4j.additivity.tracing=false
 log4j.category.tracing=DEBUG, FILE_TRACE
#------------------- CONSOLE --------------------------
 log4j.appender.A1=org.apache.log4j.ConsoleAppender
 log4j.appender.A1.threshold=ERROR
 log4j.appender.A1.layout=org.apache.log4j.PatternLayout
 log4j.appender.A1.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS z} %-5p [%c] (%t) %m%n
#------------------- FILE --------------------------
 log4j.appender.F1=org.apache.log4j.RollingFileAppender
 log4j.appender.F1.File=$DOCUMENTUM_SHARED/jboss7.1.1/server/DctmServer_MethodServer/logs/ServerApps.log
 log4j.appender.F1.MaxFileSize=10MB
 log4j.appender.F1.layout=org.apache.log4j.PatternLayout
 log4j.appender.F1.MaxBackupIndex=10
 log4j.appender.F1.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS z} %-5p [%c] (%t) %m%n
#------------------- ACS --------------------------
 log4j.category.acs=WARN, ACS_LOG
 log4j.appender.ACS_LOG=org.apache.log4j.RollingFileAppender
 log4j.appender.ACS_LOG.File=$DOCUMENTUM_SHARED/jboss7.1.1/server/DctmServer_MethodServer/logs/AcsServer.log
 log4j.appender.ACS_LOG.MaxFileSize=100KB
 log4j.appender.ACS_LOG.layout=org.apache.log4j.PatternLayout
 log4j.appender.ACS_LOG.MaxBackupIndex=10
 log4j.appender.ACS_LOG.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS z} %-5p [%c] (%t) %m%n
#------------------- FILE_TRACE --------------------------
 log4j.appender.FILE_TRACE=org.apache.log4j.RollingFileAppender
 log4j.appender.FILE_TRACE.File=$DOCUMENTUM_SHARED/jboss7.1.1/server/DctmServer_MethodServer/logs/ServerApps_trace.log
 log4j.appender.FILE_TRACE.MaxFileSize=100MB
 log4j.appender.FILE_TRACE.layout=org.apache.log4j.PatternLayout
 log4j.appender.FILE_TRACE.MaxBackupIndex=10
 log4j.appender.FILE_TRACE.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS z} %-5p [%c] (%t) %m%n

We changed the conversion pattern in order to add more info when logging. For example, here we added “z” in order to show the TimeZone. With such configuration it is easier to compare several logs which can be generated from different servers in different time zones.

We also added the MaxFileSize and MaxBackupIndex in order to manage the retention. In the code above the logs will be generated in maximum 10 files of 100MB, hence it will never exceed 1GB on the file system. The drawback is that if you have a lot of logs generated it will grow fast and the older files will be replaced by the new ones.

For acs.log

You can do the same as before, first backup the configuration file:

cp ./acs.ear/lib/configs.jar/log4j.properties ./acs.ear/lib/configs.jar/log4j.properties_$(date +%Y%m%d).log

Then edit ./acs.ear/lib/configs.jar/log4j.properties and set it like this:

log4j.rootCategory=WARN, A1, F1
 log4j.category.MUTE=OFF
 log4j.additivity.tracing=false
 log4j.category.tracing=DEBUG, FILE_TRACE
#------------------- CONSOLE --------------------------
 log4j.appender.A1=org.apache.log4j.ConsoleAppender
 log4j.appender.A1.threshold=ERROR
 log4j.appender.A1.layout=org.apache.log4j.PatternLayout
 log4j.appender.A1.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS z} %-5p [%c] (%t) %m%n
#------------------- FILE --------------------------
 log4j.appender.F1=org.apache.log4j.RollingFileAppender
 log4j.appender.F1.File=$DOCUMENTUM_SHARED/jboss7.1.1/server/DctmServer_MethodServer/logs/acs.log
 log4j.appender.F1.MaxFileSize=10MB
 log4j.appender.F1.layout=org.apache.log4j.PatternLayout
 log4j.appender.F1.MaxBackupIndex=10
 log4j.appender.F1.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS z} %-5p [%c] (%t) %m%n
#------------------- ACS --------------------------
 log4j.category.acs=WARN, ACS_LOG
 log4j.appender.ACS_LOG=org.apache.log4j.RollingFileAppender
 log4j.appender.ACS_LOG.File=$DOCUMENTUM_SHARED/jboss7.1.1/server/DctmServer_MethodServer/logs/AcsServer.log
 log4j.appender.ACS_LOG.MaxFileSize=100KB
 log4j.appender.ACS_LOG.layout=org.apache.log4j.PatternLayout
 log4j.appender.ACS_LOG.MaxBackupIndex=10
 log4j.appender.ACS_LOG.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS z} %-5p [%c] (%t) %m%n
#------------------- FILE_TRACE --------------------------
 log4j.appender.FILE_TRACE=org.apache.log4j.RollingFileAppender
 log4j.appender.FILE_TRACE.File=$DOCUMENTUM_SHARED/jboss7.1.1/server/DctmServer_MethodServer/logs/acs_trace.log
 log4j.appender.FILE_TRACE.MaxFileSize=100MB
 log4j.appender.FILE_TRACE.layout=org.apache.log4j.PatternLayout
 log4j.appender.FILE_TRACE.MaxBackupIndex=10
 log4j.appender.FILE_TRACE.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS z} %-5p [%c] (%t) %m%n
#-------------------- ATMOS LOGGING ---------------------
 log4j.logger.com.documentum.content.store.plugin.atmos=DEBUG,ACS_LOG
 log4j.logger.com.emc.esu=WARN,ACS_LOG

For bpm.ear

You can do the same as before, first backup the configuration file:

cp ./bpm.ear/APP-INF/classes/log4j.properties ./bpm.ear/APP-INF/classes/log4j.properties_$(date +%Y%m%d).log

Then edit ./bpm.ear/APP-INF/classes/log4j.properties and set it like this:

log4j.rootCategory=WARN, A1, F1
 log4j.category.MUTE=OFF
 log4j.additivity.tracing=false
 log4j.category.tracing=DEBUG, FILE_TRACE
#------------------- CONSOLE --------------------------
 log4j.appender.A1=org.apache.log4j.ConsoleAppender
 log4j.appender.A1.threshold=ERROR
 log4j.appender.A1.layout=org.apache.log4j.PatternLayout
 log4j.appender.A1.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS z} %-5p [%c] (%t) %m%n
#------------------- FILE --------------------------
 log4j.appender.F1=org.apache.log4j.RollingFileAppender
 log4j.appender.F1.File=$DOCUMENTUM_SHARED/jboss7.1.1/server/DctmServer_MethodServer/logs/bpm.log
 log4j.appender.F1.MaxFileSize=10MB
 log4j.appender.F1.layout=org.apache.log4j.PatternLayout
 log4j.appender.F1.MaxBackupIndex=10
 log4j.appender.F1.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS z} %-5p [%c] (%t) %m%n
#------------------- BPM --------------------------
 log4j.logger.com.documentum.bpm=WARN, bpmappender
 log4j.logger.com.documentum.bps=WARN, bpmappender
 log4j.additivity.com.documentum.bpm=false
 log4j.additivity.com.documentum.bps=false
 log4j.appender.bpmappender=org.apache.log4j.RollingFileAppender
 log4j.appender.bpmappender.File=$DOCUMENTUM_SHARED/jboss7.1.1/server/DctmServer_MethodServer/logs/bpm-runtime.log
 log4j.appender.bpmappender.MaxFileSize=1MB
 log4j.appender.bpmappender.layout=org.apache.log4j.PatternLayout
 log4j.appender.bpmappender.MaxBackupIndex=10
 log4j.appender.bpmappender.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS z} %-5p [%c] (%t) %m%n
#------------------- FILE_TRACE --------------------------
 log4j.appender.FILE_TRACE=org.apache.log4j.RollingFileAppender
 log4j.appender.FILE_TRACE.File=$DOCUMENTUM_SHARED/jboss7.1.1/server/DctmServer_MethodServer/logs/bpm_trace.log
 log4j.appender.FILE_TRACE.MaxFileSize=100MB
 log4j.appender.FILE_TRACE.layout=org.apache.log4j.PatternLayout
 log4j.appender.FILE_TRACE.MaxBackupIndex=10
 log4j.appender.FILE_TRACE.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS z} %-5p [%c] (%t) %m%n

When everything is setup you can restart the JMS and verify that all logs are properly written in $DOCUMENTUM_SHARED/jboss7.1.1/server/DctmServer_MethodServer/logs

Now you have your JMS log system setup consistently.

Cet article Documentum story – Documentum JMS Log Configuration est apparu en premier sur Blog dbi services.

As described in my last post related to Momentum 16, I was excited to see the session “What’s new in Documentum 7.3″. Do not worry, I could slept last night. But are my expectations fulfilled?

So yesterday I wrote about PostgreSQL which will decrease the cost of the DB licensing and then your TCO. It seems that based on benchmarks, there is still some place for small improvements to have the same performance as with traditional DBs. Dell EMC is working on it with the PostgreSQL experts. Let’s see with real case and based on the applications usage.

So a lot of news have been described. In this post, I will list some of them which will help me and my team to deliver even better service. Of course this has to be tested and see if those news can really be applied on our customer’s platforms.

• Docker and Documentum: starting with Documentum 7.3, docker is now supported. Not only the Content Server but also D2, xCP, aso… So I guess this could help us when we have to build a platform. Using docker images as templates for the Content Server, D2 and their web servers for instance.
• Renaming repository name: this can be helpful in case you build a docker image which can then be used as a template for your Content Server
• Renaming Docbase ID: same as for the previous point. Building a docker image, using it for a container, changing the docbase name and ID to have the Content Server and repositories installed as you need.
• Changing the installation owner: can this be used in case you have to move one repository to another Content Server installation where the installation owner is not the same? This has to be studied and tested…
• Dormant State for non-Admins: With this new feature you can work on your repository but the normal users cannot access it. So for maintenance tasks, I think this can be helpful too.
• Upgrade process without touching the database: this should decrease the upgrade time.
• JDK update tool: with this tool, the JDK patches and security update can be applied. This can not be used to upgrade to a new major version (like from Java 7 to Java 8).
• Xplore 1.6: fast search within folders, non admin reports are some enhancements in the fulltext area.
• UCF will no more be used, this is replaced by an Applet-free transfer where there is no more dependencies with the JRE plugin or applet.

In addition to that, SAML 2.0 Single Sign-On, Cross Forest Kerberos, more REST Services, CMIS 1.1 are also some new features coming with Document 7.3! So I hope that we can download this new version in few weeks so we can play with, do tests and see how this can fit with our daily work and future visions.

Now two questions:

•  Are my expectations fulfilled?
  I would say yes at 90%. Regarding the remaining 10%, I talked to the Content Server and D2 product managers and they were able to give me some answers which comfort me for the future.
• Am I more exited now compared to yesterday, waiting for the session? I guess yes

New tasks and challenges are coming…

Cet article Momentum16 – Day 2 – What’s new in Documentum 7.3 ? est apparu en premier sur Blog dbi services.

So today is the second day of the Momentum 16 in Barcelona and as expected, this day contained many more technical presentations and discussions than the first day. I attended several interesting presentations and hackathon/hands on sessions as well so I haven’t really had time to rest today.

So first thing in the morning, I started with a two hours hackathon session related to how it is possible to augment Documentum reliability and performance. This hackathon was lead by Henrik Rexed from Neotys. This company is actually providing Neoload, which is a load and performance testing tool that realistically simulates user activity and monitors infrastructure behavior so you can eliminate bottlenecks in all your web and mobile applications. This Load Test solution isn’t related to Documentum so you can actually Load Test everything but they do have some experience with Documentum which makes it even better. In this session, Henrik presented us the methodology that should be used to load test and monitor each and every application starting during the project phase with Unit Testing and Unit Load to ends up with complete SLA verifications by Load Testing the whole application. One thing that it is important to note and remember is that Load Testing each and every component one by one is good but that’s not enough. Sometimes, your system will break only when all components are solicited all together and that’s what you should test in the end!

After that, I attended a session presented by Mirko Weller (Volkswagen) and Christian Meier (fme) related to how 60k users are accessing the Documentum Platform at Volkswagen. This wasn’t actually a technical presentation but I wanted to follow it because we, at dbi services, are also managing a similar Documentum Platform at one of our customers with so many users and environments/instances that I thought it would be good to compare a little bit with what others are doing and to exchange feelings and ideas about how to improve our Platform.

Finally at the end of the morning, I attended a session about how to provide the right interface to your users at the right time. Boris Carbonneill (C6) and Hugo Simon-Chautemps (Sanofi) presented us the “C6 Mobile” application, the “C6 Office” interface as well as their “any” solution. The purpose of these three products is to help your users accessing their documents whenever they want with the best possible user’s experience. I actually found these applications/UIs quite good. There is a good balance in the sizing, colors, features displayed, aso…

• The C6 Mobile application can be used to access documents while traveling directly using your phone and it can be used to quickly review/approve your tasks.
• The C6 Office is actually an add-in for Microsoft Office, PowerPoint and Excel that lets you create/update documents based on templates defined in D2 and rules/permissions/acls defined in D2-Config without having to launch an internet browser. What you have to do is just to start Office, connect to the repository and work directly inside Microsoft’s application. For the moment it is apparently not working for Outlook. Maybe a future version?
• The any solution can be used in the Cloud to merge all kind of repositories coming from Documentum, SharePoint, OpenText (and more) and store all that in one place. This solution actually makes me thing to InfoArchive which is providing similar thing and some other/different features… You can synchronize documents from D2 to the Cloud to share them with external people for example or just do that so that you will have a duplicate copy in case it is needed (E.g.: copying IQs/OQs/DR/IT stuff to the Cloud might be a good idea in case there is a Disaster and your DMS isn’t there anymore, then you would still have enough to actually execute and document your Disaster Recovery!)

Beginning of the afternoon, I followed another two hours hackathon on the following subject: Developing Custom Applications with Documentum APIs. I’m not a Documentum developer but I wanted to do this hackathon so I can learn some stuff related to the Documentum REST API and it was a good session. There were three different levels for this hackathon: beginner, intermediate and advanced. I choose the intermediate one and I had to build an application using Angular and the Documentum REST API. That was actually pretty funny and I was able to play with the APIs calls, aso… Really cool.

This concludes the summary of the second day. See you tomorrow for the third and last one!

Cet article Momentum16 – Day 2 – C6, Load Testing and REST API est apparu en premier sur Blog dbi services.

The last day of the Momentum 16 in Barcelona has been a short one… Indeed there were presentations only in the morning and the event ended at 13:00 with the farewell lunch. So this will be a short blog too!

So first thing in the morning, I had the opportunity to attend a very good presentation about containerizing your On-Premise environment with Docker. This was actually the main purpose of this third day for me: learn more about Documentum with Docker. Dennis Buis (ING) and Mike Mohen (Dell EMC) presented us what has been done at ING and what is coming soon with the Documentum 7.3 release around Docker. Dell EMC will provide soon Docker images containing one of the numerous components that will be available: Content Server + DocBroker, JMS, D2 Client, D2 Config, DA, BPM, aso… The purpose of Documentum with Docker is to accelerate your deployment/upgrade processes by separating all components in simple containers. This is really a huge challenge because there is a lot of open questions BUT once it will be mature enough and once the processes and documentations will be complete enough, it will be really great. We will be able to build complete environments within a few days instead of weeks with the insurance that everything will always be setup in the same way because images are images! You can use them as many times as you want and you will always have the same result.

Mike Mohen presenting Docker @ Momentum 16

After this first session, I wanted to practice/play a little bit with Docker and therefore I participated in a hackathon simply named Using Docker with Documentum. In this hackathon, I was indeed able to play with deployment and upgrade processes of Documentum with Docker as well as creating a custom DA image from a Dockerfile. In addition to that, I also had to manage the deployment models, the security and data management/storage. For the latter, it is simply because the data should not be stored in containers… Indeed, if a container is upgraded, then all data will be lost and that’s not what you want ;).

The morning ended with a closing keynote with Beau Lotto (Neuro-Scientist and Founder of Lotto Lab) which was actually very funny. He explained and showed us, among other things, why the brain is an extraordinary thing and why the information is meaningless! I will let you think about that ;).

So in conclusion, the first day of the Momentum 16 was a little bit boring, the second one was a little bit better and finally the third one was too short but really interesting.

Cet article Momentum16 – Day 3 – Documentum with Docker est apparu en premier sur Blog dbi services.

In this blog post I will explain the migration of the crypto key (aek.key) to the crypto keystore (lockbox) for an upgraded repository. In fact, during customer activities, I has been involved in the migration of a repository hosted on a content server 6.7 SP2 to a new infrastructure. A content server 7.2 P05 was on the new infrastructure where the lockbox was already used by other repositories. The pre-requisite to achieve this migration is to have an already configured repository using the lockbox which will help to re-encrypt the password of the database owner stored later in the dbpasswd.txt file. The repository for which we wanted to migrate the crypto key was properly upgraded to the content server 7.2 P05 using the Content Server Configuration Program keeping the existing aek.key from the old system. Our goal was to align the cryptography mechanism to use the lockbox.

Please note that the above instructions cannot be used if the file store is encrypted with the Trusted Content Services.

We will also recommend removing LDAP objects and re-create them later so the password can be properly re-encrypted and DER et PEM file re-created under $DOCUMENTUM/dba/secure/ldapdb.

First of all we need to check which encryption algorithm is currently used by retrieving the r_crypto_mode from the docbase config

1> select r_crypto_mode from dm_docbase_config
2> GO
r_crypto_mode
----------------------------------------------------------------
3DES_RSA1024_SHA256
(1 row affected)

Before shutting down the repository we will retrieve a list of all users that are configured with an inline password. We will have to reset their password as the encryption mechanism will be changed.

Query to retrieve inline password users:

select user_login_name from dm_user where user_source = 'inline password';

We can now shutdown the repository:

$DOCUMENTUM/dba/dm_shutdown_repo1

It’s a best practice to backup the aek.key and the lockbox files located under the $DOCUMENTUM/dba/secure folder:

cd $DOCUMENTUM/dba
tar -czvf secure-bck-MigAek-$(date +%Y%m%d).tgz secure

It’s also a good practice to backup all files that will be updated by this procedure:

cd $DOCUMENTUM/dba/config/repo1
cp server.ini server.ini.bck-MigAek-$(date +%Y%m%d)
cp dbpasswd.txt dbpasswd.txt.bck-MigAek-$(date +%Y%m%d)

We will now connect to the Database schema to remove all encryption values. During the next startup of the repository, encryption values will be properly regenerated automatically.

sqlplus repo1/passwd@db1

Once connected you can issue the following SQL instructions :

update dm_docbase_config_s set i_crypto_key = ' ', i_ticket_crypto_key = ' ';

delete from dmi_vstamp_s where i_application = 'dm_docbase_config_crypto_key_init';
delete from dmi_vstamp_s where i_application = 'dm_docbase_config_ticket_crypto_key_init';

delete dm_sysobject_s where r_object_id = (select r_object_id from dm_public_key_certificate_s where key_type = 1);
delete dm_sysobject_r where r_object_id = (select r_object_id from dm_public_key_certificate_s where key_type = 1);
delete dm_public_key_certificate_s where key_type = 1;

delete dm_sysobject_s where r_object_id = (select r_object_id from dm_cryptographic_key_s where key_type = 1);
delete dm_sysobject_r where r_object_id = (select r_object_id from dm_cryptographic_key_s where key_type = 1);
delete dm_cryptographic_key_s where key_type = 1;
commit;

The repository initialization file can now be updated to move from the crypto key (aek.key) to the the crypto keystore (lockbox):

vi $DOCUMENTUM/dba/config/repo1/server.ini

Inside this file, you should comment the following lines:

crypto_keystore=Local
crypto_mode=3DES_RSA1024_SHA256
crypto_keyname=aek.key

Then uncomment/add the following lines:

crypto_mode = AES256_RSA1024_SHA256
crypto_keystore = Local
crypto_lockbox = lockbox.lb
crypto_keyname = CSaek

Please make sure that the crypto_mode in the first section is the same as the one retrieved before stopping the repository. The crypto_mode of the second section (new one) can contain the value that you want. The strongest mode available is “AES256_RSA1024_SHA256″ so that’s probably the value that you will want to set. In our case, we faced an issue because of the old repository and we had to keep the same crypto_mode as the original which was “3DES_RSA1024_SHA256″. Therefore at the end, our server.ini file contained the following:

crypto_mode = 3DES_RSA1024_SHA256
crypto_keystore = Local
crypto_lockbox = lockbox.lb
crypto_keyname = CSaek

Once this is done, we now need to encrypt the database owner of the repository repo1. But how can we do this since the repository isn’t started? Simple: we need an already available repository on this content server (it’s a prerequisite actually) which is already configured to use the lockbox. We used the global registry for that purpose:

> iapi gr_dbi

Please enter a user (dmadmin):
Please enter password for dmadmin:


EMC Documentum iapi - Interactive API interface
(c) Copyright EMC Corp., 1992 - 2015
All rights reserved.
Client Library Release 7.2.0050.0084


Connecting to Server using docbase gr_dbi
[DM_SESSION_I_SESSION_START]info: "Session 010f583d80095f8a started for user dmadmin."


Connected to Documentum Server running Release 7.2.0050.0214 Linux64.Oracle
Session id is s0
API> initcrypto,c
...
OK
API> encrypttext,c,xxxx
...
DM_ENCR_TEXT_V2=AAAAEJpJA5bVkJGghYFqrik3kxJ0gaWIRNvhVmZA586sBuFx7NqKnbKSpdUYf/BvJgn10OQpoZXL1T7Y2L+RmwZRUPkWqsv139zbU7u2vw7UYKX3

We can now set the encrypted password to the dbpasswd.txt as follow:

vi /app/dctm/server/dba/config/repo1/dbpasswd.txt

Add the following content inside this file:

DM_ENCR_TEXT_V2=AAAAEPAcQcFYDlYJ26cqrQ17oC0yXek7E+1g83e069TNkYXjwoRd5zXpZmCm4NdrNYAJDLJ4iEKEzrzFlAuGUWw25KXs/98L0AI7VJx2DLUZNkjX

When this has been done, the repository is now ready to start:

$DOCUMENTUM/dba/dm_start_repo1

Check the log files and validate that the repository has been properly started. Something similar should be displayed:

2016-09-05T09:11:28.907641 21003[21003] 0000000000000000 [DM_SERVER_I_START_SERVER]info: "Docbase repo1 opened"

2016-09-05T09:11:28.907790 21003[21003] 0000000000000000 [DM_SERVER_I_SERVER]info: "Setting exception handlers to catch all interrupts"

2016-09-05T09:11:28.907820 21003[21003] 0000000000000000 [DM_SERVER_I_START]info: "Starting server using service name: repo1"

2016-09-05T09:11:29.402450 21003[21003] 0000000000000000 [DM_SERVER_I_LAUNCH_MTHDSVR]info: "Launching Method Server succeeded."

2016-09-05T09:11:29.420505 21003[21003] 0000000000000000 [DM_SERVER_I_LISTENING]info: "The server is listening on network address (Service Name: repo1, Host Name: content_server_01 :V4 IP)"

2016-09-05T09:11:29.420579 21003[21003] 0000000000000000 [DM_SERVER_I_IPV6_DISABLED]info: "The server can not listen on IPv6 address because the operating system does not support IPv6"

2016-09-05T09:11:29.426518 21003[21003] 0000000000000000 [DM_SERVER_I_LISTENING]info: "The server is listening on network address (Service Name: repo1_s, Host Name: content_server_01:V4 IP)"

2016-09-05T09:11:29.429969 21003[21003] 0000000000000000 [DM_SERVER_I_IPV6_DISABLED]info: "The server can not listen on IPv6 address because the operating system does not support IPv6"

2016-09-05T09:11:31.162134 21003[21003] 0000000000000000 [DM_WORKFLOW_I_AGENT_START]info: "Workflow agent master (pid : 21060, session 010095b980000007) is started sucessfully."
IsProcessAlive: Process ID 0 is not > 0
2016-09-05T09:11:31.163095 21003[21003] 0000000000000000 [DM_WORKFLOW_I_AGENT_START]info: "Workflow agent worker (pid : 21061, session 010095b98000000a) is started sucessfully."
IsProcessAlive: Process ID 0 is not > 0
2016-09-05T09:11:32.165584 21003[21003] 0000000000000000 [DM_WORKFLOW_I_AGENT_START]info: "Workflow agent worker (pid : 21074, session 010095b98000000b) is started sucessfully."
IsProcessAlive: Process ID 0 is not > 0
2016-09-05T09:11:33.167476 21003[21003] 0000000000000000 [DM_WORKFLOW_I_AGENT_START]info: "Workflow agent worker (pid : 21087, session 010095b98000000c) is started sucessfully."
2016-09-05T09:11:34.167673 21003[21003] 0000000000000000 [DM_SERVER_I_START]info: "Sending Initial Docbroker check-point "

2016-09-05T09:11:34.188560 21003[21003] 0000000000000000 [DM_MQ_I_DAEMON_START]info: "Message queue daemon (pid : 21105, session 010095b980000456) is started sucessfully."
2016-09-05T09:11:34.488373 21102[21102] 010095b980000003 [DM_DOCBROKER_I_PROJECTING]info: "Sending information to Docbroker located on host (content_server_01) with port (1490). Information: (Config(repo1), Proximity(1), Status(Open), Dormancy Status(Active))."
Mon Sep 05 09:12:28 2016 [INFORMATION] [AGENTEXEC 21546] Detected during program initialization: Version: 7.2.0050.0214 Linux64
Mon Sep 05 09:12:32 2016 [INFORMATION] [AGENTEXEC 21546] Detected during program initialization: Agent Exec connected to server repo1: [DM_SESSION_I_SESSION_START]info: "Session 010095b980502d02 started for user dmadmin."

The migration of the encryption key to the encryption keystore is nearly completed. It remains 2 tasks which are the password reset of the internal Documentum users such as the dm_bof_registry, dmc_wdk_preferences_owner and the dmc_wdk_presets_owner users and the password reset of the application specific users (the list created just before stopping the repository) but this will not be described in this blog.

As said before, if the LDAP configuration is required, it is better to remove the LDAP objects before the migration and recreate them now that the repository is running properly using the new encryption mechanisms.

All the steps and information to migrate the encryption mechanisms to the lockbox have been provided and I hope that it can be useful for you.

Cet article Documentum story – Migrate repository cryptography from crypto key to the crypto keystore est apparu en premier sur Blog dbi services.

We had a request by a customer where he wanted the pdf automatic rendition to be disabled but keep allowing the jpeg renditions to be used as thumbnails for D2. The pdf is generated by an eternal tool so it wasn’t needed here. Here is what we did:

Login to the ADTS server and edit the following file:
<CTS_HOME>\config\CTSServerService.xml

For the repository named “DOCBASE1″, comment the QueueProcessorContext tag as follow and only if it include the dm_autorender_win31 value:

<!-- Start of comment to be added
<QueueProcessorContext DocbaseName="DOCBASE1">
    <CTSServer AttributeName="queueItemName" AttributeValue="dm_autorender_win31"/>
    ...
</QueueProcessorContext>
       End of comment to be added -->

Then restart the services

After the restart of the Content Transformation Services, wait for a few minutes and open the CTS log file <CTS_HOME>\logs\CTS_log.txt.

On the CTS log file:
– Ensure that there are no important errors and/or exceptions
– Search for the message “Product ADTS version <CTS_VERSION> is installed on repository: DOCBASE1″

Open the file <CTS_HOME>\logs\Polling_log.txt in order to verify that the changes have been applied successfully.

In this file, find all “DOCBASE1″ related CTSLegacyQueueProcessor log entries by searching for the following text:

DOCBASE1 CTSLegacyQueueProcessor

There should not be any matching text in the log file.

Note that you can use the jpeg renditions in the D2 4.5 Preview widget in order to display a kind of preview directly in the D2 interface.

Cet article Documentum story – Disable PDF automatic generation but allow jpeg est apparu en premier sur Blog dbi services.

I went into an error that you might already have once, so here is a quick guide on how to fix it.

When deleting a document from D2 I had a popup with an error like: Cannot delete file because it is still referenced in a virtual document or an assembly. Well indeed the document that I wanted to delete was previously in a VD, but for some reasons I couldn’t find the VD by the search engine as it was deleted, but not its old versions.

To summarize, my document was still referenced by the old versions of a deleted VD. So here’s the way I fixed it:

First, I needed to know which VD referenced it, for this simply use the table dmr_containment:

select * from dmr_containment where component_id='<DOCUMENT_ID>';

You should see a list of ids like:

r_object_id: ID of the current row (we will need it)

parent_id: ID of the VD that references our document

component_id: ID of our document

order_no: Index order of all documents in this VD

The problem with old versions VD is that it is set as immutable so you can’t remove the document from the VD through DQL and API. You first have to set it mutable.

So here is the condensed version of API to remove a component in an immutable VD:

fetch,c,<PARRENT_ID>
set,c,l,r_immutable_flag
F
save,c,l
removepart,c,l,<R_OBJECT_ID>
save,c,l
fetch,c,l
set,c,l,r_immutable_flag
T
save,c,l

If you have a lot of entries to remove, you may want a scripted version in DQMan:

First select all entries:

select * from dmr_containment where component_id='<DOCUMENT_ID>';

Select all rows, right click and generate the script:

fetch,c,{parent_id}
set,c,l,r_immutable_flag
F
save,c,l
removepart,c,l,{r_object_id}
save,c,l
set,c,l,r_immutable_flag
T
save,c,l

Verify that each entries have been deleted and now you should be able to remove your document without errors.

Cet article Documentum – Cannot delete document due to VD/assembly references est apparu en premier sur Blog dbi services.

As already explained in this blog, we had to remove the PDF renditions for a customer to only keep the jpeg renditions on the ADTS side because the PDF was generated by another third-party tool. If you take a look at the dmr_content items attached to a specific document (parent_id), you will understand that there is by default only one PDF content while there might be a lot more jpeg renditions. Why? The answer to that is pretty simple, the content of the PDF rendition contains all the pages of the document while the jpeg renditions are only for a single page… Therefore if an ADTS is processing a document of 25 pages, then it will create 25 jpeg renditions. Actually that’s even more than that… Indeed by default the ADTS generates two types/formats of jpeg renditions for each page which double the total number of renditions. Here are the two formats available by default:

• jpeg_lres: (Low Resolutions) that’s actually a real size preview in low resolution of the page
• jpeg_story: (StoryBoards) that’s a reduced size preview of the page, quite hard to read what’s written…

jpeg_lres is the format used by the Thumbnail Server and also by D2 for the preview widget. On the other hand, jpeg_story isn’t used at all in D2 4.5. According to EMC, it *might* has been used for D2 4.1 and previous versions but they aren’t sure about it…

Now that this has been said, let’s go back to the title of this blog. Because of this behavior of the ADTS, it might happen that one day you will see hundreds or even thousands of dmr_content items deleted. You might think that there is something wrong, that the cleanup jobs deleted too many objects or something like that… So if this happens to you, please take a look at the format of these dmr_content! Several months ago, the cleanup jobs were inactive for a few weeks because of a bug and when we reactivated them, this happened to us and we finally found out that 95% of these items where only jpeg renditions and that this was actually the expected behavior!

After that, we started thinking about how we should handle these jpeg renditions for really big documents? Because having the preview of the documents available in D2 is great but then is it really needed? Generating a preview of the first page or of the 10 first pages might makes sense but would it makes sense to generate a preview for each page of a document bigger than 10 pages? 100 pages? 1 000 pages? These previews are used in D2 and you need to move from one page to the other one starting with the page 1. If you absolutely want to see the preview of the page *364* in D2, then you will need 20 minutes to reach that page in the first place… I think that downloading the document is a little bit faster ;).

So is it possible to only generate previews for a few pages and setup a maximum number of jpeg renditions per document? The short answer to that is: yes! And that’s the purpose of this blog, wonderful, isn’t it?!

First check out the configuration file that will need to be updated:

[dmadmin@content_server_01 workspace]$ iapi DOCBASE -Udmadmin -Pxxx


        EMC Documentum iapi - Interactive API interface
        (c) Copyright EMC Corp., 1992 - 2015
        All rights reserved.
        Client Library Release 7.2.0050.0084


Connecting to Server using docbase DOCBASE
[DM_SESSION_I_SESSION_START]info:  "Session 013f245a802173d6 started for user dmadmin."


Connected to Documentum Server running Release 7.2.0050.0214  Linux64.Oracle
Session id is s0
API> retrieve,c,dm_document where folder('/System/Media Server/Command Line Files') and object_name = 'storyboard_pdfstoryboard.xml'
...
093f245a801c9075
API> checkout,c,l
...
093f245a801c9075
API> getfile,c,l,/tmp/workspace/storyboard_pdfstoryboard.xml
...
/tmp/workspace/storyboard_pdfstoryboard.xml
API> flushcache,c
...
OK

Then retrieve a test document that will be used to see how it is working:

API> retrieve,c,dm_document where r_object_id='093f245a801a8f56'
...
093f245a801a8f56
API> getfile,c,l,/tmp/workspace/TestRenditions.docx
...
/tmp/workspace/TestRenditions.docx

Ok so now the two files are stored locally. I retrieved the content of the file TestRenditions.docx in order to be able to regenerate the renditions, you will see how it works later. So let’s check how many renditions this document currently has:

API> ?,c,select r_object_id, full_format, parent_id, content_size, full_content_size, set_time, set_file from dmr_content where any parent_id='093f245a801a8f56'
r_object_id       full_format         parent_id         content_size  full_content_size       set_time                   set_file                                                                              
----------------  ------------------  ----------------  ------------  ----------------------  -------------------------  ---------------------------------------------------------------------------------------
063f245a801d95cc  jpeg_lres           093f245a801a8f56         60467                   60467  7/4/2016 13:22:39          C:\Users\SYS_AD~1\AppData\Local\Temp\batchFile353776780929575961.tar
063f245a801d95cd  jpeg_lres           093f245a801a8f56        138862                  138862  7/4/2016 13:22:39          C:\Users\SYS_AD~1\AppData\Local\Temp\batchFile353776780929575961.tar
063f245a801d95ce  jpeg_lres           093f245a801a8f56         29596                   29596  7/4/2016 13:22:39          C:\Users\SYS_AD~1\AppData\Local\Temp\batchFile353776780929575961.tar
.....
063f245a8024b99e  jpeg_story          093f245a801a8f56          3392                    3392  7/4/2016 13:22:39          C:\Users\SYS_AD~1\AppData\Local\Temp\batchFile7193259325098763580.tar
063f245a8024b99f  jpeg_story          093f245a801a8f56          4718                    4718  7/4/2016 13:22:39          C:\Users\SYS_AD~1\AppData\Local\Temp\batchFile7193259325098763580.tar
063f245a8024b9a0  jpeg_story          093f245a801a8f56          1567                    1567  7/4/2016 13:22:39          C:\Users\SYS_AD~1\AppData\Local\Temp\batchFile7193259325098763580.tar
.....
063f245a8024b622  msw12               093f245a801a8f56         90535                   90535  7/4/2016 13:22:30          /app/weblogic/tmp/DOCBASE/msD2-01/DefaultFileRenamePolicy.rename8572010254156028259.docx
(52 rows affected)

API> exit
Bye

As explained previously in this blog, the document “TestRenditions.docx” (093f245a801a8f56) has 25 pages and therefore there are 25 (jpeg_lres) + 25 (jpeg_story) + 1 (pdf) + 1 (real document) = 52 dmr_content items attached to it and therefore 51 renditions. Now let’s see the content of the configuration file and what to do to change the number of renditions we need:

[dmadmin@content_server_01 workspace]$ pwd
/tmp/workspace
[dmadmin@content_server_01 workspace]$ cat storyboard_pdfstoryboard.xml
<PDFSTORYBOARD_MP_PROPERTIES>
    <FORMAT name="JPEG">
        <PROP name="Format" type="string">JPEG</PROP>
        <PROP name="Width" type="unsigned long" token="doc_token_width">200</PROP>
        <PROP name="Height" type="unsigned long" token="doc_token_height">200</PROP>
        <PROP name="Dpi" type="unsigned long" token="doc_token_dpi">72</PROP>
        <PROP name="KeepRatio" type="boolean">true</PROP>
        <PROP name="Password" type="string">your_password_be_here</PROP>
        <PROP name="Max Pages" type="unsigned long" token="doc_token_maxPages">-1</PROP>
        <PROP name="Frames Requested" type="unsigned long" token="doc_token_frames_requested">-1</PROP>
    </FORMAT>
</PDFSTORYBOARD_MP_PROPERTIES>
[dmadmin@content_server_01 workspace]$ 
[dmadmin@content_server_01 workspace]$ sed -i 's/doc_token_maxPages">-1</doc_token_maxPages">1</' storyboard_pdfstoryboard.xml
[dmadmin@content_server_01 workspace]$ 
[dmadmin@content_server_01 workspace]$ cat storyboard_pdfstoryboard.xml
<PDFSTORYBOARD_MP_PROPERTIES>
    <FORMAT name="JPEG">
        <PROP name="Format" type="string">JPEG</PROP>
        <PROP name="Width" type="unsigned long" token="doc_token_width">200</PROP>
        <PROP name="Height" type="unsigned long" token="doc_token_height">200</PROP>
        <PROP name="Dpi" type="unsigned long" token="doc_token_dpi">72</PROP>
        <PROP name="KeepRatio" type="boolean">true</PROP>
        <PROP name="Password" type="string">your_password_be_here</PROP>
        <PROP name="Max Pages" type="unsigned long" token="doc_token_maxPages">1</PROP>
        <PROP name="Frames Requested" type="unsigned long" token="doc_token_frames_requested">-1</PROP>
    </FORMAT>
</PDFSTORYBOARD_MP_PROPERTIES>

As you can see above, I just changed the value assigned to the “doc_token_maxPages” from -1 (unlimited) to 1 (1 page) and that should be it! To apply this change, we need to check in the storyboard_pdfstoryboard.xml file:

[dmadmin@content_server_01 workspace]$ iapi DOCBASE -Udmadmin -Pxxx


        EMC Documentum iapi - Interactive API interface
        (c) Copyright EMC Corp., 1992 - 2015
        All rights reserved.
        Client Library Release 7.2.0050.0084


Connecting to Server using docbase DOCBASE
[DM_SESSION_I_SESSION_START]info:  "Session 013f245a80217406 started for user dmadmin."


Connected to Documentum Server running Release 7.2.0050.0214  Linux64.Oracle
Session id is s0
API> retrieve,c,dm_document where folder('/System/Media Server/Command Line Files') and object_name = 'storyboard_pdfstoryboard.xml'
...
093f245a801c9075
API> setfile,c,l,/tmp/workspace/storyboard_pdfstoryboard.xml
...
OK
API> checkin,c,l
...
093f245a8027254e
API> flushcache,c
...
OK

Once this is done, we can remove all current renditions of this document (we saw above pdf, jpeg_lres and jpeg_story renditions) to only let the docx/msw12 file:

API> retrieve,c,dm_document where r_object_id='093f245a801a8f56'
...
093f245a801a8f56
API> removerendition,c,l,pdf
...
OK
API> save,c,l
...
OK
API> flushcache,c
...
OK
API> removerendition,c,l,jpeg_lres
...
OK
API> save,c,l
...
OK
API> flushcache,c
...
OK
API> removerendition,c,l,jpeg_story
...
OK
API> save,c,l
...
OK
API> flushcache,c
...
OK
API> ?,c,select r_object_id, full_format, parent_id, content_size, full_content_size, set_time, set_file from dmr_content where any parent_id='093f245a801a8f56'
r_object_id       full_format         parent_id         content_size  full_content_size       set_time                   set_file                                                                              
----------------  ------------------  ----------------  ------------  ----------------------  -------------------------  ---------------------------------------------------------------------------------------
063f245a8024b622  msw12               093f245a801a8f56         90535                   90535  7/4/2016 13:22:30          /app/weblogic/tmp/DOMAIN/msD2-01/DefaultFileRenamePolicy.rename8572010254156028259.docx
(1 row affected)

The last step is to request the recreation of these renditions (using a setfile), wait 20 seconds or so and then check how much renditions have been recreated:

API> retrieve,c,dm_document where r_object_id='093f245a801a8f56'
...
093f245a801a8f56
API> setfile,c,l,/tmp/workspace/TestRenditions.docx
...
OK
API> save,c,l
...
OK
API> ?,c,select r_object_id, full_format, parent_id, content_size, full_content_size, set_time, set_file from dmr_content where any parent_id='093f245a801a8f56';
r_object_id       full_format         parent_id         content_size  full_content_size       set_time                   set_file                                                                              
----------------  ------------------  ----------------  ------------  ----------------------  -------------------------  ---------------------------------------------------------------------------------------
063f245a8024b622  msw12               093f245a801a8f56         90535                   90535  11/13/2016 12:49:48        /tmp/workspace/TestRenditions.docx
063f245a8024b957  jpeg_lres           093f245a801a8f56         60467                   60467  11/13/2016 12:49:56        C:\Users\SYS_AD~1\AppData\Local\Temp\batchFile1116397023537525059.tar
063f245a8024b958  jpeg_story          093f245a801a8f56          3392                    3392  11/13/2016 12:49:57        C:\Users\SYS_AD~1\AppData\Local\Temp\batchFile6054850753334521340.tar
(3 rows affected)

As you can see, there is now only one jpeg rendition per format and that’s for the first page only so that’s a success! If you want to keep only X jpeg renditions per document, now you know how to do it

Cet article Documentum story – Restrict the number of jpeg renditions est apparu en premier sur Blog dbi services.

A few weeks ago at a customer, I was installing a completely new sandbox environment for one of our Application Teams. On this environment, we used a new version stack with the most recent patch set (at that time): CS 7.2 P16, D2 4.6 P05, aso… A new patch level means a lot of “fun” trying to install and execute the updated installers of Documentum/BPM/D2… And of course a lot of new bugs ;). In this blog I will talk about a new issue we never faced before that is related to the Lockbox libraries.

With the older version stack, when we installed D2, we always kept the D2 Lockbox libraries under $DOCUMENTUM/d2-lib/lockbox on the CS and then we referenced these libraries in the environment variable ($PATH, $LD_LIBRARY_PATH and $CLASSPATH) of our Installation Owner. In the documentation, it is not asked to set these variables as “permanent” by adding them in the bash_profile for example or something similar but this is what we used to do to avoid adding them again and again each time we needed to update the D2 Lockbox.

For this new version stack and since the Official Documentation didn’t change, we kept the same approach. After installing D2, we had to restart the Content Server for a maintenance at the OS level. As you probably already know if you are familiar with the CS 7.2, there is also a Lockbox specific to the CS that is normally stored under $DOCUMENTUM/dba/secure/. Whenever you restart the host, you will need to store the lockbox inside the Shared Memory and that’s how it is done:

[dmadmin@content_server_01 ~]$ pwlockbox="L0ckb0xP4ssphr4se"
[dmadmin@content_server_01 ~]$ pwaek="A3kP4ssphr4se"
[dmadmin@content_server_01 ~]$ 
[dmadmin@content_server_01 ~]$ dm_crypto_boot -lockbox lockbox.lb -lockboxpassphrase $pwlockbox -passphrase $pwaek -all

Please wait. This will take a few seconds ...

Please wait, this will take a few seconds..
Setting up the (single) passphrase for all keys in the shared memory region..
Operation succeeded
[dmadmin@content_server_01 ~]$ 
[dmadmin@content_server_01 ~]$ 
[dmadmin@content_server_01 ~]$ dm_crypto_manage_lockbox -lockbox lockbox.lb -lockboxpassphrase $pwlockbox -resetfingerprint
Lockbox lockbox.lb
Lockbox Path $DOCUMENTUM/dba/secure/lockbox.lb
Reset host done

These two commands are working properly so the Lockbox has been put in the Shared Memory – normally – and the docbases should now be able to start. As explained before, we are referencing the D2 Lockbox libraries inside the environment variable by default and not the CS specific versions. So I tried to start the docbase as always and checked the logs to ensure that it was running properly:

[dmadmin@content_server_01 ~]$ $DOCUMENTUM/dba/dm_start_GR_DOCBASE
starting Documentum server for repository: [GR_DOCBASE]
with server log: [$DOCUMENTUM/dba/log/GR_DOCBASE.log]
server pid: 7056
[dmadmin@content_server_01 ~]$
[dmadmin@content_server_01 ~]$ cat $DOCUMENTUM/dba/log/GR_DOCBASE.log
The Lockbox file is newer than the active Lockbox library. Retry with a newer version of the Lockbox library.
The Lockbox file is newer than the active Lockbox library. Retry with a newer version of the Lockbox library.
2016-09-29T12:02:38.314624      7056[7056]      0000000000000000        [DM_CRYPTO_F_KEYSTORE_INIT]fatal:  "Failed to initialize keystore at $DOCUMENTUM/dba/secure/CSaek. Internal error - 1057226514"

[dmadmin@content_server_01 ~]$
[dmadmin@content_server_01 ~]$

So that’s the error I wanted to share in this blog. As you can see, the docbase wasn’t able to start because the lockbox file is “newer” than the active Lockbox libraries. So I started to do some tests to try to understand what was happening exactly and how to solve it.

Ok so the first test I did is removing the D2 Lockbox libraries from the environment variables (.bash_profile + reload of shell session) and then try to start the docbase again:

[dmadmin@content_server_01 ~]$
[dmadmin@content_server_01 ~]$ exit
logout
[morganpatou@content_server_01 ~]$ su - dmadmin
[dmadmin@content_server_01 ~]$
[dmadmin@content_server_01 ~]$
[dmadmin@content_server_01 ~]$ pwlockbox="L0ckb0xP4ssphr4se"
[dmadmin@content_server_01 ~]$ pwaek="A3kP4ssphr4se"
[dmadmin@content_server_01 ~]$ dm_crypto_boot -lockbox lockbox.lb -lockboxpassphrase $pwlockbox -passphrase $pwaek -all

Please wait. This will take a few seconds ...

Please wait, this will take a few seconds..
Setting up the (single) passphrase for all keys in the shared memory region..
Operation succeeded
[dmadmin@content_server_01 ~]$ 
[dmadmin@content_server_01 ~]$ 
[dmadmin@content_server_01 ~]$ dm_crypto_manage_lockbox -lockbox lockbox.lb -lockboxpassphrase $pwlockbox -resetfingerprint
Lockbox lockbox.lb
Lockbox Path $DOCUMENTUM/dba/secure/lockbox.lb
Lockbox open failed The Lockbox file is newer than the active Lockbox library. Retry with a newer version of the Lockbox library.
** Operation failed **
[dmadmin@content_server_01 ~]$

As you can see, after removing the D2 Lockbox libraries from the environment, the lockbox can’t be stored in the Shared Memory anymore… So based on this information, it is clear that the Content Server Lockbox libraries (stored under $DM_HOME/bin) are different than the D2 Lockbox libraries (in our case stored under $DOCUMENTUM/d2-lib/lockbox)… It’s “fun” that two patches that have been released exactly at the same time are using different versions of the Lockbox libraries… The annoying point here is that the CS Lockbox isn’t readable anymore using the CS Lockbox libraries and that can be checked using the following command:

[dmadmin@content_server_01 ~]$ dm_crypto_create -lockbox lockbox.lb -lockboxpassphrase $pwlockbox -keyname CSaek -passphrase $pwaek -algorithm AES_256_CBC -check

The Lockbox file is newer than the active Lockbox library. Retry with a newer version of the Lockbox library.
** No AEK store exists in lockbox lockbox.lb and got status code returned as '1057226514'.
[dmadmin@content_server_01 ~]$ 
[dmadmin@content_server_01 ~]$

The problem I can see here is that when installing D2, the CS Lockbox has apparently somehow been corrupted by the D2 Lockbox libraries. The fact is that when a docbase is trying to start, the start script (dm_start_GR_DOCBASE) is setting the environment using the script $DM_HOME/bin/dm_set_server_env.sh which overwrites our environment variables because this file is using the CS Lockbox libraries. That’s the reason why the docbase wasn’t able to start earlier: the docbase is trying to start using the CS libraries (older) while the lockbox has been tempered with the D2 libraries (newer).

There are several ways to quickly workaround this specific issue and I will try to describe them below.

First solution: Create a copy of the dm_crypto_create and dm_crypto_create.bin files to force it to use the environment variable defined in the .bash_profile and not reload them based on the dm_set_server_env.sh (removing the 3 lines related to the LD_LIBRARY_PATH):

[dmadmin@content_server_01 ~]$ cd $DM_HOME/bin
[dmadmin@content_server_01 bin]$ cp dm_crypto_create my_dm_crypto_create
[dmadmin@content_server_01 bin]$ cp dm_crypto_create.bin my_dm_crypto_create.bin
[dmadmin@content_server_01 bin]$ vi my_dm_crypto_create
[dmadmin@content_server_01 bin]$ diff dm_crypto_create my_dm_crypto_create
21,23d20
< else
<   LD_LIBRARY_PATH=${location}
<   export LD_LIBRARY_PATH

Once this is done, you can compare the result of the check command:

[dmadmin@content_server_01 bin]$ ./dm_crypto_create -lockbox lockbox.lb -lockboxpassphrase $pwlockbox -keyname CSaek -passphrase $pwaek -algorithm AES_256_CBC -check

The Lockbox file is newer than the active Lockbox library. Retry with a newer version of the Lockbox library.
** No AEK store exists in lockbox lockbox.lb and got status code returned as '1057226514'.
[dmadmin@content_server_01 bin]$
[dmadmin@content_server_01 bin]$
[dmadmin@content_server_01 bin]$ ./my_dm_crypto_create -lockbox lockbox.lb -lockboxpassphrase $pwlockbox -keyname CSaek -passphrase $pwaek -algorithm AES_256_CBC -check

Key - CSaek uses algorithm AES_256_CBC.

** An AEK store with the given passphrase exists in lockbox lockbox.lb and got status code returned as '0'.
[dmadmin@content_server_01 bin]$
[dmadmin@content_server_01 bin]$

=> As you can see, this is now working with the updated script but that’s not really a sexy solution…

Second solution: Update the file dm_set_server_env.sh to force it to use the D2 Lockbox libraries instead of the CS ones. You can check the Official Documentation about how to setup environment variables for D2 and then just put that at the end of this file. In our case, this was something like:

[dmadmin@content_server_01 bin]$ tail -6 $DM_HOME/bin/dm_set_server_env.sh
export LD_LIBRARY_PATH=$DOCUMENTUM/d2-lib/lockbox/lib/native/linux_gcc34_x64:$LD_LIBRARY_PATH
export PATH=$DOCUMENTUM/d2-lib/lockbox/lib/native/linux_gcc34_x64:$PATH
export CLASSPATH=$DOCUMENTUM/d2-lib/D2.jar:$DOCUMENTUM/d2-lib/LB.jar:$DOCUMENTUM/d2-lib/LBJNI.jar:$CLASSPATH
# The environment that exists after this script executes is output to
# the dm_set_server_env.log file in $DOCUMENTUM_SHARED/logs
env >> $DOCUMENTUM_SHARED/logs/dm_set_server_env.log

This solution is a little bit better but as soon as you will upgrade/migrate your Content Server, this file might be overwritten by the Patch and therefore you will face this issue again…

The solution N°2 is what EMC recommended us to do so that’s what we did in the first place and we were able to start the docbase, aso… But one week later, we tried to install a new docbase and then the Server Configuration Program wasn’t able to read the lockbox anymore and therefore we weren’t able to create any new docbase! This is an extract from the file Server Configuration Program log file:

13:58:10,266  INFO [AWT-EventQueue-0] com.documentum.install.server.ui.panels.DiWPServerUseExistingLockBox - UserSelection: "lockbox.lb"
13:58:10,266  INFO [AWT-EventQueue-0] com.documentum.install.server.ui.panels.DiWPServerUseExistingLockBox - UserSelection: "AES_256_CBC"
13:58:10,266  INFO [AWT-EventQueue-0] com.documentum.install.server.ui.panels.DiWPServerUseExistingLockBox - UserInput: "CSaek"
13:58:10,266  INFO [AWT-EventQueue-0] com.documentum.install.server.ui.panels.DiWPServerUseExistingLockBox - UserInputPassword: "**********"
13:58:10,266  INFO [AWT-EventQueue-0] com.documentum.install.server.ui.panels.DiWPServerUseExistingLockBox - UserSelection: "true"
13:58:10,266  INFO [AWT-EventQueue-0] com.documentum.install.server.ui.panels.DiWPServerUseExistingLockBox - UserInput: "lockbox.lb"
13:58:10,266  INFO [AWT-EventQueue-0] com.documentum.install.server.ui.panels.DiWPServerUseExistingLockBox - UserInputPassword: "**********"
13:58:10,507 ERROR [Thread-97] com.documentum.install.server.installanywhere.actions.DiWAServerValidateLockboxPassphrase - Check AEK key passphrase failed

Therefore the solutions 1 and 2 are actually not good because we are able to start the docbase but not to install a new one… So we had to find another, better solution and it all started with this question: which component(s) is/are actually using the D2 Lockbox on the Content Server? Of course the answer to this question is: only the Java Method Server.

Based on this information, we therefore decided to restrict the access to the D2 Lockbox libraries only to the JMS. This has been done by removing the environment variables mentioned above from the bash_profile and/or from the dm_set_server_env.sh and then by updating the JMS startup script with the same:

[dmadmin@content_server_01 ~]$ head -6 $DOCUMENTUM_SHARED/jboss7.1.1/server/startMethodServer.sh
#!/bin/sh

#for D2.lockbox
export LD_LIBRARY_PATH=$DOCUMENTUM/d2-lib/lockbox/lib/native/linux_gcc34_x64:$LD_LIBRARY_PATH
export PATH=$DOCUMENTUM/d2-lib/lockbox/lib/native/linux_gcc34_x64:$PATH
export CLASSPATH=$DOCUMENTUM/d2-lib/D2.jar:$DOCUMENTUM/d2-lib/LB.jar:$DOCUMENTUM/d2-lib/LBJNI.jar:$CLASSPATH

After that, we “simply” regenerated the Content Server Lockbox with the CS libraries (basically you need to remove completely the current encryption inside the docbase and force the CS to recreate it from scratch… Arnaud Berbier will most probably write a blog on this subject in the next few days/weeks). Then we restarted the whole Content Server and everything was working properly with the right Lockbox libraries and we were able to create new docbases.

Cet article Documentum story – Lockbox file is newer than the active Lockbox library est apparu en premier sur Blog dbi services.