Around four years ago, I did a few presentations, here, in Switzerland about “Security & Documentum”. In there, I talked about a lot of different subjects related to both security and Documentum (you guessed it…) like: ciphers, SHA, FIPS & JCE, Documentum & DFC connect mode (characteristics, ciphers, protocols, encryptions, limitations), Documentum & DFC encryptions in transit and at rest (AEK/DBK/LTK/FSK/FEK, TCS, CS Lockbox, D2 Lockbox vs D2 Keystore, Passwords encryption and decryption), and some other topics (HTTPS on WebLogic, JBoss/WildFly, Tomcat, best practices for security, LDAPS support, FIPS 140-2 support and compliance).

“Why the hell are you talking about presentations you gave 4 years ago?”. Good question, thank you! This presentation was really dense so all I could do was just put an example of the configuration files needed for real SSL Certificate based secure communication but not how exactly to reach this point. I talked about this configuration in several blogs already but never took the time to explain/show it from A to Z. So, that’s what I’m going to do here because without being able to create the SSL Certificate and trust stores, you will probably have some trouble to really configure Documentum to use the real-secure mode (in opposition to the default-secure, which is using anonymous and therefore not fully secure).

In this blog, I will use self-signed SSL Certificate only. It is possible to use CA signed SSL Certificate, the only thing it would change is that you would need to set the trust chain into the different trust stores instead of the self-signed SSL Certificate. This has pros and cons however… This means it is easier to automate because a CA trust chain is a public SSL Certificate and therefore in case you are in a CI/CD infrastructure, you can easily create the needed Documentum trust stores from anywhere (any pods, any containers, any VMs, aso…). However, that also means that anybody with access to this trust chain can potentially create the needed files used by a DFC Client to talk to your Docbroker and Repositories. That might or might not be a problem for you so I will let you decide on that. On the other hand, using a self-signed SSL Certificate makes it more difficult to gain access to the certificates (unless you are storing it in a public and open location of course) but at the same time, this complicates a little bit the setup for remote DFC Clients since you will need to share, somehow, the Docbroker and Repositories certificates in order to create a trust store for the DFC Clients.

I split the steps into different sections: one global definition of parameters & passwords and then one section per component. Please note that for the DFC Client section, I used the JMS. The same steps can be applied for any DFC Client, you will just need to have access to the needed input files. Please make sure that all components are shutdown when you start the configuration, to avoid expected errors: it will be easier to spot errors if something you expect to be working isn’t, if you don’t have hundreds of expected errors in the middle because all clients are still trying to use non-secure (or default-secure) modes. Alright, enough blabbering, let’s start with the setup.

I. Global setup/parameters

All the files needed for the Docbroker and Repositories setup needs to be put into the $DOCUMENTUM/dba/secure/ folder so all the commands will be executed in there directly. I defined here some environment variables that will be used by all the commands. The read commands will simply ask you to enter the needed password and press enter. Doing that will store the password into the environment variable (lb_pp, b_pwd, s_pwd and d_pwd). If you aren’t using any Lockbox (since deprecated since Documentum 16.7), just ignore the Lockbox part.

cd $DOCUMENTUM/dba/secure/
lb_name="lockbox.lb"
aek_name="CSaek"
b_name="docbroker"
s_name="contentserver"
d_name="dfc"

read -s -p "  ----> Please enter the ${lb_name} passphrase: " lb_pp

read -s -p "  ----> Please enter the ${b_name} related password: " b_pwd

read -s -p "  ----> Please enter the ${s_name} related password: " s_pwd

read -s -p "  ----> Please enter the ${d_name} related password: " d_pwd

echo "
Lockbox passphrase entered: ${lb_pp}
Broker password entered: ${b_pwd}
Server password entered: ${s_pwd}
DFC password entered: ${d_pwd}"

II. Docbroker setup – SSL Server only

In this section, we will create the certificate for the Docbroker, create the needed keystore (it needs to be PKCS12) and encrypt the keystore password. If you aren’t using any Lockbox, in the “dm_encrypt_password” command, just remove the two parameters related to it (and its associated value/password) and remove the “crypto_lockbox” from the Docbroker.ini file (or whatever the name of your file is).

openssl req -x509 -days 1096 -newkey rsa:2048 -keyout ${b_name}.key -out ${b_name}.crt -subj "/C=CH/ST=Jura/L=Delemont/O=dbi services/OU=IT/CN=${b_name}" -passout pass:"${b_pwd}"

openssl pkcs12 -export -out ${b_name}.p12 -inkey ${b_name}.key -in ${b_name}.crt -name ${b_name} -descert -passin pass:"${b_pwd}" -passout pass:"${b_pwd}"

dm_encrypt_password -lockbox "${lb_name}" -lockboxpassphrase "${lb_pp}" -keyname "${aek_name}" -encrypt "${b_pwd}" -file ${b_name}.pwd

cp $DOCUMENTUM/dba/Docbroker.ini $DOCUMENTUM/dba/Docbroker.ini.orig

echo "[DOCBROKER_CONFIGURATION]
secure_connect_mode=secure
crypto_keyname=${aek_name}
crypto_lockbox=${lb_name}
keystore_file=${b_name}.p12
keystore_pwd_file=${b_name}.pwd" > $DOCUMENTUM/dba/Docbroker.ini

At this point, you can start the Docbroker and it should start only on the secure port, without errors. If there are still clients up&running, you will probably face a lot of handshake failure errors… It is possible to define the list of ciphers to use in the Docbroker.ini file (cipherlist=xxx:yyy:zzz) but if you do so, please make sure that all the SSL Clients (Repository and DFC Clients alike) that will talk to it does support this cipher as well.

III. Repository setup – SSL Server and SSL Client

In this section, we will create the certificate for the Repositories (each repo can have its own if you prefer), create the needed keystore (it needs to be PKCS12), create the needed trust store (it needs to be PKCS7) and encrypt the keystore password. If you aren’t using any Lockbox, in the “dm_encrypt_password” command, just remove the two parameters related to it (and its associated value/password). In case you have several Lockbox and AEK Key, you might want to retrieve their names from the server.ini directly (inside the loop) and then use these to encrypt the password, for each Repository, independently.

openssl req -x509 -days 1096 -newkey rsa:2048 -keyout ${s_name}.key -out ${s_name}.crt -subj "/C=CH/ST=Jura/L=Jura/O=dbi services/OU=IT/CN=${s_name}" -passout pass:"${s_pwd}"

openssl pkcs12 -export -out ${s_name}.p12 -inkey ${s_name}.key -in ${s_name}.crt -name ${s_name} -descert -passin pass:"${s_pwd}" -passout pass:"${s_pwd}"

dm_encrypt_password -lockbox "${lb_name}" -lockboxpassphrase "${lb_pp}" -keyname "${aek_name}" -encrypt "${s_pwd}" -file ${s_name}.pwd

openssl crl2pkcs7 -nocrl -certfile ${b_name}.crt -outform der -out ${s_name}-trust.p7b

for s_ini in $(ls $DOCUMENTUM/dba/config/*/server.ini); do
  cp ${s_ini} ${s_ini}.orig
  sed -i --follow-symlinks "/keystore_file/d" ${s_ini}
  sed -i --follow-symlinks "/keystore_pwd_file/d" ${s_ini}
  sed -i --follow-symlinks "/truststore_file/d" ${s_ini}
  sed -i --follow-symlinks "/cipherlist/d" ${s_ini}
  sed -i --follow-symlinks "/^crypto_keyname/a truststore_file = ${s_name}-trust.p7b" ${s_ini}
  sed -i --follow-symlinks "/^crypto_keyname/a keystore_pwd_file = ${s_name}.pwd" ${s_ini}
  sed -i --follow-symlinks "/^crypto_keyname/a keystore_file = ${s_name}.p12" ${s_ini}
done

At this point, you can start the different Repositories and it should start and project itself to the Docbroker. However, The AgentExec should still fail to start properly because it should use the global dfc.properties of the Documentum Server, which wasn’t updated yet. So you might want to configure the global dfc.properties before starting the Repositories. It is possible to define the list of ciphers to use in the server.ini file (cipherlist=xxx:yyy:zzz) but if you do so, please make sure that all the SSL Clients (DFC Clients) that will talk to it and SSL Servers (Docbroker) it talks to does support this cipher as well.

IV. DFC Clients setup (JMS, IndexAgent, DA, D2, …) – SSL Client only

In this section, we will create the needed trust store (it needs to be JKS) and encrypt the trust store password. Regarding the password encryption, this command will work on any DFC Client, you will just need to add the dfc.jar in the classpath (for example on xPlore: -cp “$XPLORE_HOME/dfc/dfc.jar”) if you aren’t executing it on a Documentum Server.

openssl x509 -outform der -in ${b_name}.crt -out ${b_name}.der

openssl x509 -outform der -in ${s_name}.crt -out ${s_name}.der

$JAVA_HOME/bin/keytool -importcert -keystore ${d_name}-trust.jks -file ${b_name}.der -alias ${b_name} -noprompt -storepass ${d_pwd}

$JAVA_HOME/bin/keytool -importcert -keystore ${d_name}-trust.jks -file ${s_name}.der -alias ${s_name} -noprompt -storepass ${d_pwd}

d_pwd_enc=$($JAVA_HOME/bin/java com.documentum.fc.tools.RegistryPasswordUtils ${d_pwd})

cp $DOCUMENTUM/config/dfc.properties $DOCUMENTUM/config/dfc.properties.orig
sed -i '/dfc.session.secure_connect_default/d' $DOCUMENTUM/config/dfc.properties
sed -i '/dfc.security.ssl.use_existing_truststore/d' $DOCUMENTUM/config/dfc.properties
sed -i '/dfc.security.ssl.truststore/d' $DOCUMENTUM/config/dfc.properties
sed -i '/dfc.security.ssl.truststore_password/d' $DOCUMENTUM/config/dfc.properties

echo "dfc.session.secure_connect_default=secure
dfc.security.ssl.use_existing_truststore=false
dfc.security.ssl.truststore=$DOCUMENTUM/dba/secure/${d_name}-trust.jks
dfc.security.ssl.truststore_password=${d_pwd_enc}" >> $DOCUMENTUM/config/dfc.properties

This is technically the global dfc.properties of a Documentum Server and not really the JMS one but I assume almost everybody in the world is just including this one (using #include) for the dfc.properties of the JMS (ServerApps, acs, bpm, …), to avoid duplication of generic parameters/configurations at multiple locations and just manage them globally.

At this point, you can start the DFC Client and it should be able to communicate with the Docbroker and with the Repositories. As said before, if you already started the Repository, you might want to make sure that the AgentExec is running and if not, maybe restart the Repositories quickly.

Some final remarks on the SSL Certificate based secure configuration of Documentum:

• Other Content Servers & Docbrokers (HA part) must re-use the exact same keystores (and therefore trust store as well in the end). Files must be sent to all other hosts and re-used exactly in the same way
• Other DFC clients can use newly created files but in the end, it will contain the exact same content (either the self-signed Docbroker and Repositories certificates or the CA-signed trust chain)… Therefore, files can be sent to all DFC clients and re-used exactly in the same way as well
• After the initial generation, you don’t need any of the key, crt or der files anymore so you can remove them for security reasons:
  • rm ${b_name}.key ${b_name}.crt ${b_name}.der ${s_name}.key ${s_name}.crt ${s_name}.der
• I didn’t describe everything in full-length here, there are a bunch of other things and limitations to know before going into that direction so you will probably want to read the documentation carefully

L’article Documentum – SSL Certificate based secure communication setup est apparu en premier sur dbi Blog.

When working with containers, the usual trend is to make them as compact as possible by removing any file or executable that is not used by what is deployed in it. Typically, interactive commands are good candidates not to be installed. Sometimes, this trend is so extreme that even simple utilities such as ping or even the less pager are missing from the containers. This is generally fine once a container reaches the production stage but not so much while what’s deployed in it is still in development and has not reached its full maturity yet. Without such essentials utilities, one might as well be blind and deaf.
Recently, I wanted to check if a containerized D2 installation could reach its target repository but could not find any of the usual command-line tools that are included with the content server, such as iapi or dctmbroker. Bummer, I thought, but less politely. Admittedly, those tools belong to the content server binaries and not to DFCs clients such as D2 and DA, so that makes sense. Maybe, but that does not solve my problem.
In article A Small Footprint Docker Container with Documentum command-line Tools, I showed how to have a minimalist installation of the iapi, idql and dmdocbroker command-line tools that could be containerized but for my current need, I don’t need so much power. A bare repository ping, a dctmping if you will, would be enough. Clearly, as D2 is a DFCs application, a simple DFCs-based java utility would be just what the doctor ordered. There are probably hundreds variants of such a basic utility floating on the Web but here is my take on it anyway.

The code

The following listing generates the dctmping.java program.

$ cat - << dctmping.java
// a healthcheck program to check whether all the repositories whose docbroker's hosts are defined in dfc.properties are reachable;
// cec at dbi-services.com, June 201;
// to compile: export CLASSPATH=/path/to/the/dfc.jar javac dctmping.java
// to execute: java -classpath .:/path/to/dfc/config:$CLASSPATH dctmping [target_docbase user_name password]
// if target_docbase is given, the programm will attempt to connect to it using the command-line parameters user_name and password and run a SELECT query;
 
import com.documentum.fc.client.IDfClient;
import com.documentum.fc.client.DfClient;
import com.documentum.fc.client.DfQuery;
import com.documentum.fc.client.IDfCollection;
import com.documentum.fc.client.IDfDocbaseMap;
import com.documentum.fc.client.IDfQuery;
import com.documentum.fc.client.IDfSession;
import com.documentum.fc.client.IDfSessionManager;
import com.documentum.fc.common.DfLoginInfo;
import com.documentum.fc.common.IDfLoginInfo;
import com.documentum.fc.client.IDfTypedObject;
import java.io.RandomAccessFile;
 
public class dctmping {
   IDfSessionManager sessMgr = null;
   IDfSession idfSession;
 
   public void showDFC_properties_file() throws Exception {
      System.out.println("in showDFC_properties_file");
      IDfClient client = DfClient.getLocalClient();
      IDfTypedObject apiConfig = client.getClientConfig();
      String[] values = apiConfig.getString("dfc.config.file").split(":");
      System.out.printf("dfc.properties file found in %snn", values[1]);
      RandomAccessFile f_in = new RandomAccessFile(values[1], "r");
      String s;
      while ((s = f_in.readLine()) != null) {
         System.out.println(s);
      }
      f_in.close();
   }
 
   public void getAllDocbases() throws Exception {
      System.out.printf("%nin getAllDocbases%n");
      IDfClient client = DfClient.getLocalClient();
      IDfDocbaseMap docbaseMap = client.getDocbaseMap();
      for (int i = 0; i < docbaseMap.getDocbaseCount(); i++) {
          System.out.println("Docbase Name : " + docbaseMap.getDocbaseName(i));
          System.out.println("Docbase Desc : " + docbaseMap.getDocbaseDescription(i));
      }
   }
 
   public void getDfSession(String repo, String user, String passwd) throws Exception {
      System.out.printf("%nin getDfSession%n");
      IDfLoginInfo login = new DfLoginInfo();
      login.setUser(user);
      login.setPassword(passwd);
      IDfClient client = DfClient.getLocalClient();
      sessMgr = client.newSessionManager();
      sessMgr.setIdentity(repo, login);
      idfSession = sessMgr.getSession(repo);
      if (idfSession != null)
          System.out.printf("Session created successfully in repository %s as user %sn", repo, user);
      else
         throw new Exception();
   }
 
   public void releaseSession() throws Exception {
      sessMgr.release(idfSession);
   }
 
   public void API_select(String repo, String dql) throws Exception {
      System.out.printf("%nin API_select%s%n", repo);
      System.out.printf("SELECT-ing in repository %sn", repo);
      System.out.printf("Query is: %sn", dql);
      IDfQuery query = new DfQuery();
      query.setDQL(dql);
      IDfCollection collection = null;
      String r_object_id = null;
      String object_name = null;
      final int max_listed = 20;
      int count = 0;
      System.out.printf("%-16s  %sn", "r_object_id", "object_name");
      System.out.printf("%-16s  %sn", "----------------", "-----------");
      try {
         collection = query.execute(idfSession, IDfQuery.DF_READ_QUERY);
         while (collection.next()) {
            count++;
            if (max_listed == count) {
               System.out.printf("... max %d reached, skipping until end of result set ...n", max_listed);
               continue;
            }
            else if (count > max_listed)
               continue;
            r_object_id = collection.getString("r_object_id");
            object_name = collection.getString("object_name");
            System.out.printf("%-16s  %sn", r_object_id, object_name);
         }
         System.out.printf("%d documents foundsn", count);
      } finally {
         if (collection != null) {
            collection.close();
         }
      }
   }
  
   public static void main(String[] args) throws Exception {
      dctmping dmtest = new dctmping();
      dmtest.showDFC_properties_file();
      dmtest.getAllDocbases();
      String docbase;
      String user;
      String passwd;
      if (0 == args.length || args.length > 3) {
         System.out.println("nUsage: dctmping [target_docbase [user_name [password]]]");
         System.exit(1);
      }
      if (1 == args.length) {
         docbase = args[0];
         user    = "dmadmin";
         passwd  = "trusted:no_password_needed";
      } 
      else if (2 == args.length) {
         docbase = args[0];
         user    = args[1];
         passwd  = "trusted:no_password_needed";
      } 
      else {
         docbase = args[0];
         user    = args[1];
         passwd  = args[2];
      } 

      String[] queries = {"SELECT r_object_id, object_name from dm_document where folder('/System/Sysadmin/Reports', descend);",
                          "SELECT r_object_id, object_name from dm_cabinet;"};
      for (String dql_stmt: queries) {
         try {
            dmtest.getDfSession(docbase, user, passwd);
            dmtest.API_select(docbase, dql_stmt);
         }
         catch(Exception exception) {
            System.out.printf("Error while attempting to run DQl query", docbase, user);
            exception.printStackTrace();
         }
         finally {
            try {
               dmtest.releaseSession();
            }
            catch(Exception exception) {}
         }
      }
   }
}
eoj

Compile it as instructed in the header, e.g.:

$ export DOCUMENTUM=/home/dmadmin/documentum
$ export CLASSPATH=$DOCUMENTUM/shared/dfc/dfc.jar
$ javac dctmping.java

Use the command below to execute it:

$ java -classpath .:$DOCUMENTUM/shared/config:$CLASSPATH dctmping [target_docbase [user_name [password]]]

When no parameter is given on the command-line (line 110), dctmping attempts to reach the dfc.properties file and displays its content if it succeeds (function showDFC_properties_file starting on line 25). This proves that the file is accessible through one of the paths in $CLASSPATH.
It then displays the help message (line 16 below):

$ java -classpath .:$DOCUMENTUM/shared/config:$CLASSPATH dctmping
in showDFC_properties_file
dfc.properties file found in /home/dmadmin/documentum/shared/config/dfc.properties

dfc.data.dir=/home/dmadmin/documentum/shared
dfc.tokenstorage.dir=/home/dmadmin/documentum/shared/apptoken
dfc.tokenstorage.enable=false

dfc.docbroker.host[0]=dmtest.cec
dfc.docbroker.port[0]=1489

in getAllDocbases
Docbase Name : dmtest73
Docbase Desc : a v7.3 test repository

Usage: dctmping [target_docbase [user_name [password]]]

Starting on line 12 above (see function getAllDocbases starting on line 39), a list of all the reachable docbases is given, here only one, dmtest73. In some installations, this list be more populated, e.g.:

Docbase Name : global_repository
Docbase Desc : Global Repository
Docbase Name : SERAC
Docbase Desc : SERAC CTLQ Repository
Docbase Name : CARAC
Docbase Desc : CARAC CTLQ Repository

At least one parameter is needed: the repository name. The user default to “dmadmin”. A password must be given when connecting remotely from the content server. If connecting locally, it can be anything as the user is trusted (if trusted authentication is allowed), or even left empty. The session is opened by function getDfSession() starting on line 49.

A Real Case Application

To check a remote repository’s access, all 3 parameters are required, e.g.:

$ java -classpath .:$CLASSPATH dctmping dmtest73 dmadmin my_password
in showDFC_properties_file
dfc.properties file found in /home/dmadmin/documentum/shared/config/dfc.properties

dfc.data.dir=/home/dmadmin/documentum/shared
dfc.tokenstorage.dir=/home/dmadmin/documentum/shared/apptoken
dfc.tokenstorage.enable=false

dfc.docbroker.host[0]=dmtest.cec
dfc.docbroker.port[0]=1489

in getAllDocbases
Docbase Name : dmtest73
Docbase Desc : a v7.3 test repository

in getDfSession
Session created successfully in repository dmtest73 as user dmadmin

in API_selectdmtest73
SELECT-ing in repository dmtest73
Query is: SELECT r_object_id, object_name from dm_document where folder('/System/Sysadmin/Reports', descend);
r_object_id       object_name
----------------  -----------
0900c35080002a1b  StateOfDocbase
0900c350800029d7  UpdateStats
0900c35080002a11  ContentWarning
0900c35080002a18  DBWarning
0900c3508000211e  ConsistencyChecker
5 documents founds

in getDfSession
Session created successfully in repository dmtest73 as user dmadmin

in API_selectdmtest73
SELECT-ing in repository dmtest73
Query is: SELECT r_object_id, object_name from dm_cabinet;
r_object_id       object_name
----------------  -----------
0c00c35080000107  Temp
0c00c3508000012f  Templates
0c00c3508000057b  dm_bof_registry
0c00c350800001ba  Integration
0c00c35080000105  dmadmin
0c00c35080000104  dmtest73
0c00c35080000106  System
0c00c35080000130  Resources
8 documents founds

After displaying the content of the dfc.properties file, dctmping attempts to connect to the given repository with the given credentials and runs both queries below (see function API_select() starting on line 68):

SELECT r_object_id, object_name from dm_document where folder('/System/Sysadmin/Reports', descend);
SELECT r_object_id, object_name from dm_cabinet;

As we don’t need a full listing for testing the connectivity, especially when it may be very large and take several minutes to complete, a maximum of 20 rows by query is returned.

Checking the global_registry

If a global_registry repository has been defined in the dfc.properties file, the credentials to access it are also listed in that file as illustrated below:

...
dfc.globalregistry.repository=my_global_registry
dfc.globalregistry.username=dm_bof_registry
dfc.globalregistry.password=AFAIKL8C2Y/2gRyQUV1R7pmP7hfBDpafeWPST9KKlQRtZVJ4Ya0MhLsEZKmWr1ok9+oThA==
...

Eventhough the password is encrypted, it is available verbatim to connect to the repository as shown below:

$ java -classpath .:$DOCUMENTUM/shared/config:$CLASSPATH dctmping dm_bof_registry 'AFAIKL8C2Y/2gRyQUV1R7pmP7hfBDpafeWPST9KKlQRtZVJ4Ya0MhLsEZKmWr1ok9+oThA=='
...
in getDfSession
Session created successfully in repository my_global_registry as user dm_bof_registry
 
in API_selectmy_global_registry
SELECT-ing in repository my_global_registry
Query is: SELECT r_object_id, object_name from dm_cabinet;
r_object_id       object_name
----------------  -----------
0c0f476f80000106  System
0c0f476f800005b4  dm_bof_registry
2 documents founds
...

Note that the utility does not check if the given docbase is a global_registry, so the connection attempt may raise an authentication error, but at least the given docbase has been proved to be reachable (or not).

Conclusion

This innocent little stand-alone utility can be easily included in any docker image and can assist in ascertaining at least 5 dependencies in the context of a post-deployment’s sanity check or of a problem troubleshooting:

1. the correct installation of the DFCs
2. the correct configuration of the dfc.properties
3. the correct initialization of $DOCUMENTUM and $CLASSPATH environment variables
4. the accessibility of the remote repository(ies)
5. and finally, the credentials to connect to said repository.

That is one big chunk of pre-requisites to check at once even before the DFCs clients start.
As a bonus, it also lists all the repositories potentially accessible through the docbrokers listed in the dfc.properties, which is useful in case the local machine hosts a mutualized service serving several repositories. Another freebie is that, if a global_registry is defined in the dfc.properties file, it can be checked too; actually, the utility could be improved to do that automatically (or manually in a subsequent re-run) as the needed credentials are given in that file and require no user interaction but, as they say, let’s leave that as an exercise for the reader.

L’article dctmping, A Documentum Repository Checker Utility est apparu en premier sur dbi Blog.

A rare but recurrent issue that customers sometimes encounter is to how to connect to each one of distinct repositories with the same name or same docbase id, or even both if one repository is a clone of the other one. The present connection resolution technique based on the dfc.properties file does not support this and only lets one connect to the first matching repository found. Well knowing this limitation, why were they created with the same name in the first place ? Just don’t and that pitfall is avoided. Actually however, this situation makes sense when the repositories are created by different teams, e.g. one TEST repository for each application in development, or developers’ personal repositories, maybe local on their laptops, or an application’s repository existing in different stages of its lifecycle, e.g. a PRODUCT_CATALOG repository in DEV, TUNI, INT, CTLQ, ACC, PROD, ARCHIVED, and maybe CLONE_1, CLONE_2, etc…
Now, at some point, a developer would need to access simultaneously, say, PRODUCT_CATALOG in DEV and PRODUCT_CATALOG in PROD in order to troubleshoot a problem, a plausible scenario. Another scenario is when a common, DFCs-based service must access docbases with the same name coming from different, independently managed applications. So, how to do that ?

Yes, how to do that ?

In this article, I presented a possible solution based on editing the dfc.properties file on-the-fly prior to connecting, so its docbroker_host and docbroker_port parameters would point to the target repository’s docbroker. As you know, that file is required to be able to connect to repositories. It is read by the DFCs when a connection request is done from a DFCs client, and the docbrokers listed in there are queried in turn until one of them replies with the information for the requested target that projects to it.
The problem with this algorithm is that it is not selective enough to tell apart distinct repositories with the same name. If the target could be specified like docbase[.server_instance][@machine[:port]], that could work. Actually, the above syntax, minus the [:port] part, is accepted but it does not work in our case. A typical returned error is:

DfNoServersException:: THREAD: main; MSG: [DM_DOCBROKER_E_NO_SERVERS_FOR_DOCBASE]error: "The DocBroker running on host (null:0) does not know of a server for the specified docbase (dmtest73@192.168.56.12)"; ERRORCODE: 100; NEXT: null

The hack described in the aforementioned article consisted in removing all the docbroker_host and docbroker_port pairs of parameters and inserting the exact one that is used by the repository of interest, so any ambiguity is lifted. Prior to the next connection, the work-around is repeated for the new target.
Wrappers around the command-line tools iapi and idql, wiapi respectively widql, do just that.
It works quite well for these tools but what about java DFCs clients ? I guess we could subclass the session manager’s getSession() or write a wrapper that applies the same work-around transparently and that was my intention at first when a customer raised the issue but I decided to do give OpenText’s knowledge base another try, in case an out of-the-box yet unknown parameter would solve this question with no programming needed, i.e. no customization of existing code. I was not left empty-handed as I found this article: “How can I dynamically switch between different Docbrokers using DFC?”, Article ID:KB8801087. Here is the full note:

How can I dynamically switch between different Docbrokers using DFC?

Article ID:KB8801087

Add FavoriteEmailNotifyPrint

Applies to

Documentum Foundation Classes 4.0

Summary

How can I dynamically switch between different Docbrokers using DFC?

Resolution
Normally, you are restricted to Docbases registered to theDocbroker specified in the DMCL.ini file. Modifying the apiConfigobject can still access Docbases that are not registered with thisDocbroker.

The steps involved are the following:
1.Get a session object for a docbase which is registered withthe docbroker specified in the dmcl.ini file.
2.Get an apiConfig object using the methodIDfSession.getClientConfig();
3.Set the 'primary_host" and "primary_port" attributes ofthis apiConfig object to a different docbroker.
4.Get a session for a different docbase registered with thisdocbroker.

For more details, please refer to the APIconfig object in theobject reference manual for Documentum 4i.
Legacy Article ID
ECD316940

It’s again a bit hacky and still requires a customization but at least it addresses the DFCs part. Although it is DFCs-oriented, as a proof of concept, let’s see first if we could make it work using the API from within iapi.

Testing from within iapi

In our test case, we have a docbase named dmtest73 with id 50000 projecting to docbroker host dmtest.cec on port 1489 and a different docbase with the same name dmtest73 but id 90005 projecting to the docbroker host docker on port 7489. Both docbrokers are present in the dfc.properties file. We want to be able to connect to each of those docbases. Let’s first try the normal, default connection:

$ iapi dmtest73 -Udmadmin -Pdmadmin
API> retrieve,c,dm_server_config
dump,c,l
USER ATTRIBUTES

  object_name                     : dmtest73
...
  owner_name                      : dmtest73
...
  acl_domain                      : dmtest73
...
  operator_name                   : dmtest73
...
  web_server_loc                  : dmtest.cec
...
SYSTEM ATTRIBUTES

  r_object_type                   : dm_server_config
  r_creation_date                 : 7/1/2019 19:26:18
  r_modify_date                   : 6/20/2021 23:55:33
...
  r_creator_name                  : dmtest73
...
  r_server_version                : 7.3.0000.0214  Linux64.Oracle
  r_host_name                     : dmtest.cec
  r_process_id                    : 908
  r_install_owner                 : dmadmin
...

# dump the docbase config too;
API> retrieve,s0,dm_docbase_config    
...
3c00c35080000103
API> dump,c,l
...
USER ATTRIBUTES

  object_name                     : dmtest73
  title                           : a v7.3 test repository
...
  owner_name                      : dmtest73
...
  acl_domain                      : dmtest73
...
  index_store                     : DM_DMTEST73_INDEX
...

SYSTEM ATTRIBUTES

  r_object_type                   : dm_docbase_config
  r_creation_date                 : 7/1/2019 19:26:18
  r_modify_date                   : 7/1/2019 17:33:50
...
  r_creator_name                  : dmtest73
...
  r_dbms_name                     : Oracle
  r_docbase_id                    : 50000
# we are in the docbase with id 50000;

So, this is the dmtest73 docbase that is accessed with the current docbroker definitions in dfc.properties.
Usually, only that docbase, with FQN dmtest73@dmtest.cec:1489, is reachable because its docbroker is listed earlier than dmtest73@docker:7489 in dfc.properties, as reported by the session’s apiconfig object:

dump,c,apiconfig
...
  dfc.docbroker.exclude.failure_th: 3
  dfc.docbroker.exclusion.time    : 30
  dfc.docbroker.host           [0]: dmtest.cec
                               [1]: dmtest.cec
                               [2]: docker
                               [3]: docker
                               [4]: docker
                               [5]: docker
  dfc.docbroker.port           [0]: 7289
                               [1]: 1489
                               [2]: 1489
                               [3]: 1589
                               [4]: 7489
                               [5]: 6489
  dfc.docbroker.protocol       [0]: rpc_static
                               [1]: rpc_static
                               [2]: rpc_static
                               [3]: rpc_static
                               [4]: rpc_static
                               [5]: rpc_static
  dfc.docbroker.search_order      : sequential
  dfc.docbroker.service        [0]: dmdocbroker
                               [1]: dmdocbroker
                               [2]: dmdocbroker
                               [3]: dmdocbroker
                               [4]: dmdocbroker
                               [5]: dmdocbroker
  dfc.docbroker.timeout        [0]: 0
                               [1]: 0
                               [2]: 0
                               [3]: 0
                               [4]: 0
                               [5]: 0
...

Here, 6 pairs of docbroker_host/docbroker_port were defined in the dfc.properties; each one has additional default parameters dfc.docbroker.protocol, dfc.docbroker.service and dfc.docbroker.timeout; they are all synchronized, i.e.
dfc.docbroker.host[i] uses port dfc.docbroker.port[i], dfc.docbroker.protocol[i], dfc.docbroker.service[i] with a timeout of dfc.docbroker.timeout[i].
The Note says to force the primary docbroker (i.e. the first one, corresponding to the values at index 0 of the dfc.docbroker% attributes) to the target docbase’s one; it doesn’t say anything about the other ones but we’ll remove them to eradicate any possibility of a fail over logic:

# empty the dfc.docbroker% attributes;
API> truncate,c,apiconfig,dfc.docbroker.host
truncate,c,apiconfig,dfc.docbroker.port
truncate,c,apiconfig,dfc.docbroker.protocol
truncate,c,apiconfig,dfc.docbroker.service
truncate,c,apiconfig,dfc.docbroker.timeout

# add the target docbase's docbroker;
API> append,c,apiconfig,dfc.docbroker.host
docker
append,c,apiconfig,dfc.docbroker.port
7489
append,c,apiconfig,dfc.docbroker.protocol
rpc_static
append,c,apiconfig,dfc.docbroker.service
dmdocbroker
append,c,apiconfig,dfc.docbroker.timeout
0

# verify the parameters;
API> dump,c,apiconfig
...
  dfc.docbroker.host           [0]: docker
  dfc.docbroker.port           [0]: 7489
  dfc.docbroker.protocol       [0]: rpc_static
  dfc.docbroker.search_order      : sequential
  dfc.docbroker.service        [0]: dmdocbroker
  dfc.docbroker.timeout        [0]: 0
...

# try to connect to dmtest73@docker:7489 now; 
API> connect,dmtest73,dmadmin,dmadmin
...
s1
# OK but where are we really ?
API> retrieve,s1,dm_server_config
...
3d015f9580000102
API> dump,c,l
USER ATTRIBUTES

  object_name                     : dmtest73
...
  owner_name                      : dmtest73c
...
  acl_domain                      : dmtest73c
...
  operator_name                   : dmtest73c
...
  web_server_loc                  : container73

SYSTEM ATTRIBUTES

  r_object_type                   : dm_server_config
  r_creation_date                 : 6/20/2021 02:52:36
  r_modify_date                   : 6/20/2021 00:59:43
...
  r_creator_name                  : dmtest73c
...
  r_server_version                : 16.4.0000.0248  Linux64.Oracle
  r_host_name                     : container73
  r_process_id                    : 13709
...

# dump the docbase config too;
API> retrieve,s1,dm_docbase_config
...
3c015f9580000103
API> dump,c,l
...
USER ATTRIBUTES

  object_name                     : dmtest73
  title                           : dmtest73 homonym silently
...
...
  acl_domain                      : dmtest73c
...
  index_store                     : dm_dmtest73c_index
...

SYSTEM ATTRIBUTES

  r_object_type                   : dm_docbase_config
  r_creation_date                 : 6/20/2021 02:52:36
  r_modify_date                   : 6/20/2021 01:27:22
...
...
  r_dbms_name                     : Oracle
  r_docbase_id                    : 90005
...
# we are now in the docbase with id 90005;

It works as proved by comparing the dm_server_config and dm_docbase_config objects, notably the dm_docbase_config.r_docbase_id.
So, what to think of such a solution ? It requires some cutting and pasting of statements but they could be saved into some sort of macros depending on the command-line client used (e.g. the GUI-based dqman). For iapi I’d definitively prefer starting it as described in the above article, e.g.

$ wiapi dmtest73:docker:7289
$ widql dmtest73:dmtest.cec:1489

but if several connections to homonym docbases must be opened at once in the same iapi working session, this is the only way.

The DFCs version

The solution from the Note:KB8801087 was for the DFCs in the first place. Let’s apply it in function getSessionExtended() (starting on line 46 in the listing below) with the following profile:

public void getSessionExtended(String repo, String user, String passwd) throws Exception

The function takes a docbase’s specification in repo, and a user name and password to connect to the repository. As we need a unambiguous syntax to define repo, let’s use the time proven one:

docbase_name[:docbroker_machine[:docbroker_port]]

where docbroker_machine:docbroker_port are the coordinates of the docbroker, possibly remote, the docbase docbase_name projects to. Obviously, two homonym docbases cannot project to the same docbroker (the docbroker would reject them all but the first one), but they could be running on the same machine and project to distinct docbrokers, locally or remotely.
An alternate syntax could be:

docbase_name@docbroker_machine:docbroker_port

If preferred, the change is easy in the function’s regular expression (see line 51 below):

Pattern re_docbase_docbroker_host_docbroker_port = Pattern.compile("^([^:]+)(@([^:]+)(:([^:]+))?)?$");

The function will first parse the extended repository syntax (starting on line 53), apply the Note’s hack (lines 81 to 87), and finally connect (line 96). To be sure we are in the right docbase, all the configuration objects will be dumped by the ancillary function dump_current_configs() (defined starting on line 124, called on line 182) as well. Starting on line 111, the apiconfig object is displayed to check the hack’s change. On lines 102 to 108, we find out the minimum field width to display the apiconfig’s attributes without truncation, as the dump() method does a lame job here.
If no docbroker is specified, then the function fails back to the DFCs default behavior so it remains backwards compatible.
In order to make the test possible, a main() function (starting on line 157) accepting and parsing command-line parameters (starting on line 167) is also provided.

// A test program for function getSessionExtended() to open sessions using an extended syntax;
// June 2021, cec@dbi-services.com;
// to compile: javac extendedConnection.java
// to execute: java -classpath .:$DOCUMENTUM/config:$CLASSPATH extendedConnection target_docbase user_name password
// the programm will attempt to connect to target_docbase using those credentials;
// target_docbase's syntax is:
//   docbase_name[:docbroker_machine[:docbroker_port]]
// if docbroker_machine is missing (and hence docbroker_port too), a session is opened through the default behavior from the dfc.properties file, i.e. following the order the docbrokers are listed in that file;
// otherwise, the docbrokers listed in dfc.properties are not used and the specified docbroker is;
// so the function is compatible with the default behavior of the DFCs;
import com.documentum.fc.client.IDfClient;
import com.documentum.fc.client.DfClient;
import com.documentum.fc.client.DfQuery;
import com.documentum.fc.client.IDfCollection;
import com.documentum.fc.client.IDfDocbaseMap;
import com.documentum.fc.client.IDfQuery;
import com.documentum.fc.client.IDfSession;
import com.documentum.fc.client.IDfSessionManager;
import com.documentum.fc.common.DfLoginInfo;
import com.documentum.fc.common.IDfLoginInfo;
import com.documentum.fc.common.IDfAttr;
import com.documentum.fc.client.IDfTypedObject;
import com.documentum.fc.client.IDfPersistentObject;
import java.io.RandomAccessFile;
import java.util.regex.Pattern;
import java.util.regex.Matcher;
import java.util.Enumeration;

public class extendedConnection {
   String gr_username = null;
   String gr_password = null;
   IDfSessionManager sessMgr = null;
   IDfSession idfSession;
 
   public void usage() {
   // output the utility extended repository's syntax;
      System.out.println("Usage:");
      System.out.println("   java [-Ddfc.properties.file=/tmp/dfc.properties] -classpath .:/app/dctm/config:$CLASSPATH docbase[:docbroker-host[:docbroker-port]] username password");
      System.out.println("Use the extended repository syntax (e.g. dmtest:docker:7289, vs. dmtest) to override the DFCs' default resolution mechanism.");
      System.out.println("Examples:");
      System.out.println("   java -classpath .:/app/dctm/config:$CLASSPATH dctmping dmtest73 dmadmin dmadmin");
      System.out.println("for using the docbrokers defined in the dfc.properties (classic usage)");
      System.out.println("   java -classpath .:/tmp:$CLASSPATH dctmping dmtest73:docker:7489 dmadmin dmadmin");
      System.out.println("for short-circuiting the docbrokers and using the repo's extended syntax");
   }
   public void getSessionExtended(String repo, String user, String passwd) throws Exception {
   // change the dfc.docbroker.host and dfc.docbroker.port to connect with more flexibility;
   // The target repository repo is defined though the following advanced syntax:
   //    docbase[:docbroker-host[:docbroker-port]]
      System.out.printf("getSessionExtended%n");
      Pattern re_docbase_docbroker_host_docbroker_port = Pattern.compile("^([^:]+)(:([^:]+)(:([^:]+))?)?$");

      Matcher check = re_docbase_docbroker_host_docbroker_port.matcher(repo);
      String docbase = null;
      String docbroker_host = null;
      String docbroker_port = null;
      if (check.find()) {
         docbase = check.group(1);
         docbroker_host = check.group(3);
         docbroker_port = check.group(5);
      } 
      else {
         System.out.println("Missing docbase name; the docbase is mandatory");
         usage();
      }
      if (docbroker_host != null) {
         System.out.println("host = " + docbroker_host);
         if (docbroker_port == null)
            docbroker_port = "1489";
         System.out.println("port = " + docbroker_port);
      } 
      else
         System.out.println("docbroker host is empty, using the dfc.properties");
      System.out.println("using the " + (docbroker_host != null ? (" docbroker host " + docbroker_host + ":" + docbroker_port) : "the dfc.properties"));

      IDfClient client = DfClient.getLocalClient();
      IDfTypedObject client_config = client.getClientConfig();

      if (docbroker_host != null) {
         // let's hack the session config to force the given docbroker[:port];
         client_config.truncate("dfc.docbroker.host", 0);
         client_config.appendString("dfc.docbroker.host", docbroker_host);
         client_config.truncate("dfc.docbroker.port", 0);
         client_config.appendString("dfc.docbroker.port", docbroker_port);
         client_config.truncate("dfc.docbroker.protocol", 1);
         client_config.truncate("dfc.docbroker.service", 1);
         client_config.truncate("dfc.docbroker.timeout", 1);
      }

      IDfLoginInfo login = new DfLoginInfo();
      login.setUser(user);
      login.setPassword(passwd);
      // as recommended, use the session manager;
      sessMgr = client.newSessionManager();
      sessMgr.setIdentity(docbase, login);
      idfSession = sessMgr.getSession(docbase);

      System.out.printf("session config:%n");
      int max_length = 0;
      // as the default presentation from dump() sucks too much due to the truncated attribut names, let's produce a non-truncated one:
      // first, iterate through the session config and find the longest one;
      for (Enumeration e = client_config.enumAttrs(); e.hasMoreElements() ;) {
         IDfAttr attr = (IDfAttr) e.nextElement();
         String name = attr.getName();
         String value = client_config.getString(name);
         if (null != value)
            max_length = max_length >= name.length() ? max_length : name.length();
      }
      // max_length contains now the length of the longest attribute name;
      // display the nicely formatted session config;
      for (Enumeration e = client_config.enumAttrs(); e.hasMoreElements() ;) {
         IDfAttr attr = (IDfAttr) e.nextElement();
         String name = attr.getName();
         String value = client_config.getAllRepeatingStrings(name, "n" + String.join("", new String(new char[max_length]).replace("", " ") + "  "));
         System.out.printf("%" + max_length + "s: %s%n", name, value);
      }
   }
 
   public void releaseSession() throws Exception {
   // quite obvious;
      sessMgr.release(idfSession);
   }
 
   public void dump_all_configs() throws Exception {
   // dump all the server and docbase configs defined in repository;
      System.out.printf("%nin dump_all_configs%n");
      String[] configs = {"select r_object_id from dm_server_config",
                          "select r_object_id from dm_docbase_config"};
      IDfQuery query = new DfQuery();
      for (String dql_stmt: configs) {
         System.out.println("executing " + dql_stmt);
         query.setDQL(dql_stmt);
         IDfCollection collection = null;
         String r_object_id = null;
         try {
            collection = query.execute(idfSession, IDfQuery.DF_READ_QUERY);
            while (collection.next()) {
               r_object_id = collection.getString("r_object_id");
               IDfPersistentObject obj = idfSession.getObjectByQualification("dm_sysobject where r_object_id = '" + r_object_id + "'");
               System.out.println("dumping object with id = " + r_object_id);
               System.out.println(obj.dump());
            }
         }
         catch(Exception e) {
            System.out.printf("Error in dumps_all_configs()%n");
            System.out.println(e.getMessage());
            e.printStackTrace();
            // continue as far as possible;
         }
         finally {
            if (collection != null)
               collection.close();
         }
      }
   }

   public static void main(String[] args) throws Exception {
      System.out.printf("%nextendedConnection started ...%n");
      extendedConnection dmtest = new extendedConnection();
      if (0 == args.length)
         System.exit(0);

      String docbase = null;
      String user    = null;
      String passwd  = null;
      // if any arguments are present, they must be the target docbase and the credentials to connect (username and password);
      if (args.length != 3) {
         System.out.println("Missing arguments. Usage: dctmping [target_docbase [user_name password]]");
         System.exit(1);
      }
      else {
         docbase = args[0];
         user    = args[1];
         passwd  = args[2];
      }

      try {
         // connect using the command-line parameters;
         dmtest.getSessionExtended(docbase, user, passwd);

         //dump all the server and docbase configs;
         dmtest.dump_all_configs();
      }
      catch(Exception e) {
         System.out.printf("Error while working in the docbase %s as user %sn", docbase, user);
         System.out.println(e.getMessage());
         e.printStackTrace();
      }
      finally {
         try {
            dmtest.releaseSession();
         }
         catch(Exception e) {}
      }
   }
}

Interestingly, the DFCs’ truncate method has an additional argument compared to its API’s equivalent, the index at which truncation should start. Also, using the DFCs, no opened session is needed prior to accessing the config temporary object, named here client config vs. API’s apiconfig.
Save this code into a file named extendedConnection.java in current directory.
To compile it:

javac extendedConnection.java

To execute it from current directory (paths may vary according to your installation):

export DOCUMENTUM=/app/dctm
export CLASSPATH=$DOCUMENTUM/dctm.jar:$DOCUMENTUM/dfc/dfc.jar
$ java -classpath .:$DOCUMENTUM/config:$CLASSPATH extendedConnection docbase username password

Example of output

Here is an example of the default behavior, i.e. when the target repo does not use the extended syntax repo:docbroker_host[:docbroker_port]:

$ java -classpath .:/app/dctm/config:$CLASSPATH extendedConnection dmtest73 dmadmin dmadmin

extendedConnection started ...
getSessionExtended
docbroker host is empty, using the dfc.properties
using the the dfc.properties
session config:
                                                  dfc.name: dfc
                                            dfc.config.dir: /app/dctm/config
                                           dfc.config.file: file:/app/dctm/config/dfc.properties
...
                                        dfc.docbroker.host: 
                                                            192.168.56.12
                                                            
                                                            192.168.56.15
                                                            192.168.56.15
                                                            192.168.56.15
                                        dfc.docbroker.port: 0
                                                            1489
                                                            0
                                                            7489
                                                            6489
                                                            1489
...
                                    dfc.docbroker.protocol: rpc_static
                                                            rpc_static
                                                            rpc_static
                                                            rpc_static
                                                            rpc_static
                                                            rpc_static
                                     dfc.docbroker.service: dmdocbroker
                                                            dmdocbroker
                                                            dmdocbroker
                                                            dmdocbroker
                                                            dmdocbroker
                                                            dmdocbroker
                                     dfc.docbroker.timeout: 0
                                                            0
                                                            0
                                                            0
                                                            0
                                                            0
...

in dump_all_configs
executing select r_object_id from dm_server_config
dumping object with id = 3d00c35080000102
USER ATTRIBUTES

  object_name                     : dmtest73
  title                           : 
  subject                         : 
...
  owner_name                      : dmtest73
...

SYSTEM ATTRIBUTES

  r_object_type                   : dm_server_config
  r_creation_date                 : 7/1/2019 7:26:18 PM
...
  r_creator_name                  : dmtest73
...
  r_server_version                : 7.3.0000.0214  Linux64.Oracle
  r_host_name                     : dmtest.cec
...

executing select r_object_id from dm_docbase_config
dumping object with id = 3c00c35080000103
USER ATTRIBUTES

  object_name                     : dmtest73
  title                           : a v7.3 test repository
...
  acl_domain                      : dmtest73
...
  index_store                     : DM_DMTEST73_INDEX
...

SYSTEM ATTRIBUTES

  r_object_type                   : dm_docbase_config
  r_creation_date                 : 7/1/2019 7:26:18 PM
...
  r_creator_name                  : dmtest73
...
  r_docbase_id                    : 50000
...

This is the dmtest73 docbase with id 50000, like in the iapi example seen before.
Make sure $CLASSPATH includes the dfc.jar, or dctm.jar if going through the manifest in it is preferred.
Next, an example of accessing a docbase with the same name as previously but on another host, using the extended syntax:

$ java -classpath .:/app/dctm/config:$CLASSPATH extendedConnection dmtest73:docker:7489 dmadmin dmadmin

extendedConnection started ...
getSessionExtended
host = docker
port = 7489
using the  docbroker host docker:7489
session config:
                                                  dfc.name: dfc
                                            dfc.config.dir: /app/dctm/config
                                           dfc.config.file: file:/app/dctm/config/dfc.properties
...
                                        dfc.docbroker.host: docker
                                        dfc.docbroker.port: 7489
...
                                    dfc.docbroker.protocol: rpc_static
                                     dfc.docbroker.service: dmdocbroker
                                     dfc.docbroker.timeout: 0
...

in dump_all_configs
executing select r_object_id from dm_server_config
dumping object with id = 3d015f9580000102
USER ATTRIBUTES

  object_name                     : dmtest73
  title                           : 
  subject                         : 
...

SYSTEM ATTRIBUTES

  r_object_type                   : dm_server_config
  r_creation_date                 : 6/20/2021 2:52:36 AM
...
  r_creator_name                  : dmtest73c
...
  r_server_version                : 16.4.0000.0248  Linux64.Oracle
  r_host_name                     : container73
...

executing select r_object_id from dm_docbase_config
dumping object with id = 3c015f9580000103
USER ATTRIBUTES

  object_name                     : dmtest73
  title                           : dmtest73 homonym silently
...
  acl_domain                      : dmtest73c
...
  index_store                     : dm_dmtest73c_index
...

SYSTEM ATTRIBUTES

  r_object_type                   : dm_docbase_config
  r_creation_date                 : 6/20/2021 2:52:36 AM
...
  r_creator_name                  : dmtest73c
...
  r_docbase_id                    : 90005
...

The reached docbase has the 90005, as expected.
As a final example, let’s connect to the default dmtest73 again but this time using the extended syntax:

$ java -classpath .:/app/dctm/config:$CLASSPATH dctmping dmtest73:dmtest.cec:1489 dmadmin dmadmin

extendedConnection started ...

getSessionExtended
host = dmtest.cec
port = 1489
using the  docbroker host dmtest.cec:1489
session config:
                                                  dfc.name: dfc
                                            dfc.config.dir: /app/dctm/config
                                           dfc.config.file: file:/app/dctm/config/dfc.properties
...
                                  dfc.docbroker.debug.host: 
                                  dfc.docbroker.debug.port: 0
..
                                        dfc.docbroker.host: dmtest.cec
                                        dfc.docbroker.port: 1489
..
                                    dfc.docbroker.protocol: rpc_static
                                     dfc.docbroker.service: dmdocbroker
                                     dfc.docbroker.timeout: 0
...

in dump_all_configs
executing select r_object_id from dm_server_config
dumping object with id = 3d00c35080000102
USER ATTRIBUTES

  object_name                     : dmtest73
  title                           : 
  subject                         : 
...
  operator_name                   : dmtest73
...
  web_server_loc                  : dmtest.cec
...

SYSTEM ATTRIBUTES

  r_object_type                   : dm_server_config
  r_creation_date                 : 7/1/2019 7:26:18 PM
...
  r_creator_name                  : dmtest73
...
  r_server_version                : 7.3.0000.0214  Linux64.Oracle
  r_host_name                     : dmtest.cec
...

executing select r_object_id from dm_docbase_config
dumping object with id = 3c00c35080000103
USER ATTRIBUTES

  object_name                     : dmtest73
  title                           : a v7.3 test repository
...
  acl_domain                      : dmtest73
...
  index_store                     : DM_DMTEST73_INDEX
...

SYSTEM ATTRIBUTES

  r_object_type                   : dm_docbase_config
  r_creation_date                 : 7/1/2019 7:26:18 PM
...
  r_creator_name                  : dmtest73
...
  r_docbase_id                    : 50000
...

The reached docbase has id 50000 as expected.
The client config shows here the same output as sessionconfig from within iapi, which shouldn’t come as a surprise given that iapi invokes the DFCs behind the scenes.
As with the alternative in the aforementionned article, this work-around even allows to completely remove the docbroker_host/docbroker_port pairs from the dfc.properties file as it does not use them and imposes its own direct, flattened resolution mechanism (“just contact the damn given docbroker”). This can be verified with a stripped-down dfc.properties file (e.g. /tmp/dfc.properties) with no such entries and invoking the test program thusly:

$ java -Ddfc.properties.file=/tmp/dfc.properties -classpath .:$CLASSPATH extendedConnection dmtest73:docker:7489 dmadmin dmadmin

Of course, in such a case, the default resolution mechanism won’t work if attempted to be used through the normal repository syntax, and the DFCs will complain with an error as shown below:

$ java -Ddfc.properties.file=/tmp/dfc.properties -classpath .:$CLASSPATH extendedConnection dmtest73 dmadmin dmadmin

extendedConnection started ...
getSessionExtended
docbroker host is empty, using the dfc.properties
using the the dfc.properties
Error while working in the docbase dmtest73 as user dmadmin
[DM_DOCBROKER_E_NO_DOCBROKERS]error:  "No DocBrokers are configured"
DfServiceException:: THREAD: main; MSG: [DM_DOCBROKER_E_NO_DOCBROKERS]error:  "No DocBrokers are configured"; ERRORCODE: 100; NEXT: null
	at com.documentum.fc.client.DfServiceException.newNoDocbrokersException(DfServiceException.java:44)
...

Conclusion

Although this work-around smells like a hack, it is a very effective one. From within iapi or any DFCs client with a slightly smarter connection function, this recurrent yet pesky limitation has finally found a solution. However, while it is acceptable for in-house development where source code is available, third-party developers might not want to bother with an ad hoc customization to fix a niche problem; so some per$ua$ion work might be needed here too. Let’s hope that OpenText, in breaking with a 30 years-old code immobilism in this area, will delight us soon with a really transparent solution which, who knows, supports an enhanced repository syntax such as the proposed one.

L’article Connecting to Repositories with the Same Name and/or ID est apparu en premier sur dbi Blog.

Have you ever faced an issue where dmqdocbroker, iapi, idql and the likes aren’t able to communicate at all with any Docbroker (connection broker)? Here, I’m not talking about potentially wrong hostname, port or connect modes, which might prevent you to reach a Docbroker if it’s not configured properly because this will still most likely reply to you with an error message… I’m really talking about the utility/binaries that cannot communicate anymore, it’s like all messages are sent to the void and nothing will ever respond (is that a black-hole I’m seeing?)!

Earlier this month, I suddenly had this behavior at one of our customer on two out of dozens of Documentum Servers. Everything seemed to be up&running, all the processes were there:

[dmadmin@cs-0 ~]$ ps -ef
UID      PID PPID C  STIME TTY       TIME CMD
dmadmin 7005    1 0  14:11 ?     00:00:00 ./dmdocbroker -port 1489 -init_file $DOCUMENTUM/dba/Docbroker.ini
dmadmin 7014    1 0  14:11 ?     00:00:00 ./dmdocbroker -port 1487 -init_file $DOCUMENTUM/dba/DocbrokerExt.ini
dmadmin 7077    1 0  14:11 ?     00:00:07 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 7087    1 0  14:11 ?     00:00:07 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini
dmadmin 7100 7077 0  14:11 ?     00:00:00 $DM_HOME/bin/mthdsvr master 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 7077 GR_REPO $DOCUMENTUM/dba/log
dmadmin 7101 7100 0  14:11 ?     00:00:04 $DM_HOME/bin/mthdsvr worker 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 0 GR_REPO $DOCUMENTUM/dba/log
dmadmin 7102 7087 0  14:11 ?     00:00:00 $DM_HOME/bin/mthdsvr master 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 7087 REPO1 $DOCUMENTUM/dba/log
dmadmin 7121 7102 0  14:11 ?     00:00:03 $DM_HOME/bin/mthdsvr worker 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 0 REPO1 $DOCUMENTUM/dba/log
dmadmin 7122 7100 0  14:11 ?     00:00:03 $DM_HOME/bin/mthdsvr worker 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 1 GR_REPO $DOCUMENTUM/dba/log
dmadmin 7123 7077 0  14:11 ?     00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 7124 7077 0  14:11 ?     00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 7144 7102 0  14:11 ?     00:00:04 $DM_HOME/bin/mthdsvr worker 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 1 REPO1 $DOCUMENTUM/dba/log
dmadmin 7148 7087 0  14:11 ?     00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini
dmadmin 7149 7087 0  14:11 ?     00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini
dmadmin 7165 7100 0  14:11 ?     00:00:04 $DM_HOME/bin/mthdsvr worker 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 2 GR_REPO $DOCUMENTUM/dba/log
dmadmin 7166 7077 0  14:11 ?     00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 7167 7102 0  14:11 ?     00:00:03 $DM_HOME/bin/mthdsvr worker 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 2 REPO1 $DOCUMENTUM/dba/log
dmadmin 7168 7087 0  14:11 ?     00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini
dmadmin 7169 7100 0  14:11 ?     00:00:04 $DM_HOME/bin/mthdsvr worker 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 3 GR_REPO $DOCUMENTUM/dba/log
dmadmin 7187 7077 0  14:11 ?     00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 7190 7102 0  14:11 ?     00:00:03 $DM_HOME/bin/mthdsvr worker 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 3 REPO1 $DOCUMENTUM/dba/log
dmadmin 7194 7087 0  14:11 ?     00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini
dmadmin 7210 7100 0  14:11 ?     00:00:03 $DM_HOME/bin/mthdsvr worker 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 4 GR_REPO $DOCUMENTUM/dba/log
dmadmin 7213 7077 0  14:11 ?     00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 7215 7102 0  14:11 ?     00:00:04 $DM_HOME/bin/mthdsvr worker 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 4 REPO1 $DOCUMENTUM/dba/log
dmadmin 7225 7087 0  14:11 ?     00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini
dmadmin 7334    1 0  14:12 ?     00:00:00 /bin/sh $JMS_HOME/server/startMethodServer.sh
dmadmin 7336 7334 0  14:12 ?     00:00:00 /bin/sh $JMS_HOME/bin/standalone.sh
dmadmin 7447 7336 21 14:12 ?     00:02:57 $JAVA_HOME/bin/java -D[Standalone] -server -XX:+UseCompressedOops -server -XX:+UseCompressedOops -Xms8g -Xmx8g -XX:MaxMetaspaceSize=512m -XX
dmadmin 7695 7077 0  14:12 ?     00:00:04 ./dm_agent_exec -enable_ha_setup 1 -docbase_name GR_REPO.GR_REPO -docbase_owner dmadmin -sleep_duration 0
dmadmin 7698 7087 0  14:12 ?     00:00:04 ./dm_agent_exec -enable_ha_setup 1 -docbase_name REPO1.REPO1 -docbase_owner dmadmin -sleep_duration 0
dmadmin 7908 7077 0  14:13 ?     00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 7918 7087 0  14:13 ?     00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini
dmadmin 8269 7077 0  14:21 ?     00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 8270 7077 0  14:21 ?     00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini
dmadmin 8327 6370 0  14:27 pts/1 00:00:00 ps -ef
[dmadmin@cs-0 ~]$

However, I could see the communication issues by looking at the Repository log because it would show that the AgentExec was actually not connected, even after almost 20 minutes:

[dmadmin@cs-0 ~]$ cd $DOCUMENTUM/dba/log/REPO1/agentexec/
[dmadmin@cs-0 agentexec]$ date
Mon Apr 12 14:29:03 UTC 2021
[dmadmin@cs-0 agentexec]$
[dmadmin@cs-0 agentexec]$ tail -8 ../../REPO1.log
2021-04-12T14:11:56.453407      7087[7087]      0000000000000000        [DM_WORKFLOW_I_AGENT_START]info:  "Workflow agent worker (pid : 7194, session 010f12345000000c) is started sucessfully."
2021-04-12T14:11:57.455899      7087[7087]      0000000000000000        [DM_SERVER_I_START]info:  "Sending Initial Docbroker check-point "

2021-04-12T14:11:57.547764      7087[7087]      0000000000000000        [DM_MQ_I_DAEMON_START]info:  "Message queue daemon (pid : 7225, session 010f123450000456) is started sucessfully."
2021-04-12T14:11:58.348442      7223[7223]      010f123450000003        [DM_DOCBROKER_I_PROJECTING]info:  "Sending information to Docbroker located on host (cs-0.domain.com) with port (1490).  Information: (Config(REPO1), Proximity(1), Status(Open), Dormancy Status(Active))."
2021-04-12T14:11:58.661666      7223[7223]      010f123450000003        [DM_DOCBROKER_I_PROJECTING]info:  "Sending information to Docbroker located on host (cs-0.domain.com) with port (1488).  Information: (Config(REPO1), Proximity(1), Status(Open), Dormancy Status(Active))."
2021-04-12T14:11:58.959490      7223[7223]      010f123450000003        [DM_DOCBROKER_I_PROJECTING]info:  "Sending information to Docbroker located on host (cs-1.domain.com) with port (1490).  Information: (Config(REPO1), Proximity(2), Status(Open), Dormancy Status(Active))."
Mon Apr 12 14:12:55 2021 [INFORMATION] [AGENTEXEC 7698] Detected during program initialization: Version: 16.4.0200.0256  Linux64
[dmadmin@cs-0 agentexec]$
[dmadmin@cs-0 agentexec]$ # Previous startup from the AgentExec logs showing that it didn't start yet
[dmadmin@cs-0 agentexec]$ tail -2 agentexec.log
Sat Apr 10 19:20:30 2021 [INFORMATION] [LAUNCHER 23135] Detected during program initialization: Version: 16.4.0200.0256 Linux64
Sun Apr 11 19:20:26 2021 [INFORMATION] [LAUNCHER 2890] Detected during program initialization: Version: 16.4.0200.0256 Linux64
[dmadmin@cs-0 agentexec]$

The interesting part is that the Repositories have all been started properly and projected to the Docbroker. However, any client from the Documentum Server locally wouldn’t be able to connect to the Docbroker. Even more interesting, this was actually a HA environment with 2 CS. The Documentum Server hosting the Primary CS (I will call it cs-0) had the issue while the Documentum Server hosting the Remote CS (I will call it cs-1) had no problem. Executing the dmqdocbroker on the cs-0 to ping the Docbroker of the cs-0 never gave a response, however doing the exact same command to ping the Docbroker of the cs-0 but from the cs-1 host did work without any problem and showed the correct projection of the repositories:

## on cs-0 (Documentum Server hosting the PCS)
[dmadmin@cs-0 ~]$ hostname -f
cs-0.domain.com
[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping
^C
real 0m53.513s
user 0m6.132s
sys 0m0.578s
[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ time echo quit | iapi REPO1.REPO1 -Udmadmin -Pxxx
^C
real 0m46.431s
user 0m6.241s
sys 0m0.575s
[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ time echo quit | iapi REPO1.cs-1_REPO1 -Udmadmin -Pxxx
^C
real 0m35.694s
user 0m6.163s
sys 0m0.582s
[dmadmin@cs-0 ~]$

## on cs-1 (Documentum Server hosting the RCS)
[dmadmin@cs-1 ~]$ hostname -f
cs-1.domain.com
[dmadmin@cs-1 ~]$
[dmadmin@cs-1 ~]$ time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0200.0080
Using specified port: 1489
Successful reply from docbroker at host (cs-0) on port(1490) running software version (16.4.0200.0256 Linux64).

real 0m3.499s
user 0m6.950s
sys 0m0.469s
[dmadmin@cs-1 ~]$
[dmadmin@cs-1 ~]$ time echo quit | iapi REPO1.cs-1_REPO1 -Udmadmin -Pxxx

OpenText Documentum iapi - Interactive API interface
Copyright (c) 2018. OpenText Corporation
All rights reserved.
Client Library Release 16.4.0200.0080

Connecting to Server using docbase REPO1.cs-1_REPO1
[DM_SESSION_I_SESSION_START]info: "Session 010f1234501b635c started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0200.0256 Linux64.Oracle
Session id is s0
API> Bye

real 0m5.032s
user 0m7.401s
sys 0m0.487s
[dmadmin@cs-1 ~]$
[dmadmin@cs-1 ~]$ time echo quit | iapi REPO1.REPO1 -Udmadmin -Pxxx

OpenText Documentum iapi - Interactive API interface
Copyright (c) 2018. OpenText Corporation
All rights reserved.
Client Library Release 16.4.0200.0080

Connecting to Server using docbase REPO1.REPO1
[DM_SESSION_I_SESSION_START]info: "Session 010f1234501b6506 started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0200.0256 Linux64.Oracle
Session id is s0
API> Bye

real 0m5.315s
user 0m7.976s
sys 0m0.515s
[dmadmin@cs-1 ~]$

This shows that the issue isn’t the Docbroker or the Repositories themselves but rather the utility/binaries present on the cs-0 that cannot open communication channels with the local Docbroker, for some reasons… Even after setting debugging on the Docbroker, I could see communications when the dmqdocbroker utility was used on the cs-1 host but nothing was showing-up if the same command was used on the cs-0 host instead. You can enable some logs for the Docbroker by adding “trace=true” into the Docbroker.ini file and you can also add some other traces by setting the following environment variables (value can be 1 or 10 for example) and then restart the Docbroker: “export DM_DOCBROKER_TRACE=1; export DM_DEBUG_BROKER=1; export DM_TRANS_LOG=1“. Additionally, you can also add options to the launch script, just like for the Repository part: “-odocbroker_trace -onettrace_all_option -oxxx“.

Unfortunately, the dmqdocbroker utility uses the dmawk binary and the iapi/idql are also binaries so it’s rather difficult to debug further without the source code… After some testing/debugging, I found something rather hard to believe… All the binaries of the Documentum Server looked OK, they were no changes done in the past few weeks and the files were the same (same hash) than on the cs-1 for example. As you probably know, dmqdocbroker/iapi/idql will use the dfc.properties from the folder “$DOCUMENTUM_SHARED/config/” (with $DOCUMENTUM_SHARED=$DOCUMENTUM forced, starting in 16.4). Therefore, I have been looking into this folder for anything that might disrupt the proper behavior of the utility/binaries. All the files in this folder were 100% identical between cs-0 and cs-1, except for the encrypted password of the dm_bof_registry as well as the dfc.keystore since both of these are generated once. This would mean that the issue wouldn’t be there, but it was. I started looking into other areas to try to find the root cause but nothing was working. Then, I came back to the config folder and simply tried to empty it… Somehow, the dmqdocbroker was working again, magically! I mean, it printed many errors because the files log4j.properties, dfc.properties and dfc.keystore weren’t there but it replied something… What to do then? Well, I went step by step, putting back the files one by one, as they are supposed to be, and then executing the dmqdocbroker again to see if it stops working.

The files dfc.properties, log4j.properties, dfcfull.properties, dfc.keystore and all the cache folders were restored properly and the dmqdocbroker was still working without any problem… So what the hell? That’s more or less all of the files, isn’t it? True, that’s all the files, minus the dbor ones: dbor.properties and dbor.properties.lck. At this customer, these files are empty because no configuration was needed. It would be very hard to believe that this could be the issue, right? Well, have a look for yourself:

[dmadmin@cs-0 ~]$ cd $DOCUMENTUM_SHARED/config/
[dmadmin@cs-0 config]$ ls -l
total 140
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26 2020 ServerApps
drwxr-x--- 9 dmadmin dmadmin  4096 Jul 26 2020 Shared
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26 2020 acs
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26 2020 bpm
-rwxr-x--- 1 dmadmin dmadmin     0 Jul 22 2020 dbor.properties
-rw-rw-r-- 1 dmadmin dmadmin     0 Jul 26 2020 dbor.properties.lck
-rw-rw-r-- 1 dmadmin dmadmin  2152 Jul 26 2020 dfc.keystore
-rw-rw-r-- 1 dmadmin dmadmin   481 Jul 26 2020 dfc.properties
-rw-rw---- 1 dmadmin dmadmin    70 Jul 22 2020 dfc.properties.bak.0
-rwxr-x--- 1 dmadmin dmadmin   242 Jul 26 2020 dfc.properties.bak.1
-rw-rw-r-- 1 dmadmin dmadmin   271 Jul 26 2020 dfc.properties.bak.2
-rw-rw-r-- 1 dmadmin dmadmin   323 Jul 26 2020 dfc.properties.bak.3
-rw-rw-r-- 1 dmadmin dmadmin   481 Jul 26 2020 dfc.properties.bak.4
-rw-rw-r-- 1 dmadmin dmadmin   482 Jul 26 2020 dfc.properties.bak.5
-rwxrwx--- 1 dmadmin dmadmin 79268 Jul 22 2020 dfcfull.properties
-rwxr-x--- 1 dmadmin dmadmin  1242 Jul 26 2020 log4j.properties
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ # With the initial content, dmqdocbroker isn't working
[dmadmin@cs-0 config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:43:28 UTC 2021
^C
real    0m22.718s
user    0m6.401s
sys     0m0.853s
Wed Apr 14 07:43:51 UTC 2021
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ mkdir test
[dmadmin@cs-0 config]$ mv * test/
mv: cannot move 'test' to a subdirectory of itself, 'test/test'
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ ls -l
total 4
drwxr-x--- 6 dmadmin dmadmin 4096 Apr 14 07:44 test
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ # With the folder empty, dmqdocbroker is "working" (errors but expected ones)
[dmadmin@cs-0 config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:45:15 UTC 2021
0 [main] ERROR com.documentum.fc.common.impl.logging.LoggingConfigurator  - Problem locating log4j configuration
1 [main] WARN com.documentum.fc.common.impl.logging.LoggingConfigurator  - Using default log4j configuration
3 [main] ERROR com.documentum.fc.common.impl.preferences.PreferencesManager  - [DFC_PREFERENCE_LOAD_FAILED] Failed to load persistent preferences from null
java.io.FileNotFoundException: dfc.properties
        at com.documentum.fc.common.impl.preferences.PreferencesManager.locateMainPersistentStore(PreferencesManager.java:378)
        at com.documentum.fc.common.impl.preferences.PreferencesManager.readPersistentProperties(PreferencesManager.java:329)
        at com.documentum.fc.common.impl.preferences.PreferencesManager.<init>(PreferencesManager.java:37)
        ...
2862 [main] ERROR com.documentum.fc.client.security.impl.IdentityManager  - [DFC_SECURITY_IDENTITY_INIT] no identity initialization or incomplete identity initialization
DfException:: THREAD: main; MSG: ; ERRORCODE: ff; NEXT: null
        at com.documentum.fc.client.security.impl.JKSKeystoreUtil.creteNewKeystoreFile(JKSKeystoreUtil.java:425)
        at com.documentum.fc.client.security.impl.JKSKeystoreUtil.createNewKeystore(JKSKeystoreUtil.java:209)
        at com.documentum.fc.client.security.impl.DfcIdentityKeystore.applyDfcInitPolicy(DfcIdentityKeystore.java:95)
        ...
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0200.0080
Using specified port: 1489
Successful reply from docbroker at host (cs-0) on port(1490) running software version (16.4.0200.0256 Linux64).

real    0m3.763s
user    0m6.265s
sys     0m0.672s
Wed Apr 14 07:45:19 UTC 2021
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ ls -l
total 12
drwxr-x--- 8 dmadmin dmadmin 4096 Apr 14 07:45 documentum
-rw-r----- 1 dmadmin dmadmin 3245 Apr 14 07:45 log4j.log
drwxr-x--- 6 dmadmin dmadmin 4096 Apr 14 07:44 test
-rw-r----- 1 dmadmin dmadmin    0 Apr 14 07:45 trace.log
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ rm -rf documentum/ log4j.log  trace.log
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ mv test/log4j.properties ./
[dmadmin@cs-0 config]$ mv test/dfc.properties ./
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ # With the folder empty exept for log4j.properties and dfc.properties files, dmqdocbroker is working
[dmadmin@cs-0 config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:47:17 UTC 2021
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0200.0080
Using specified port: 1489
Successful reply from docbroker at host (cs-0) on port(1490) running software version (16.4.0200.0256 Linux64).

real    0m4.280s
user    0m8.161s
sys     0m0.729s
Wed Apr 14 07:47:21 UTC 2021
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ ls -l
total 20
drwxr-x--- 8 dmadmin dmadmin 4096 Apr 14 07:47 Shared
-rw-r----- 1 dmadmin dmadmin 2153 Apr 14 07:47 dfc.keystore
-rw-rw-r-- 1 dmadmin dmadmin  481 Jul 26  2020 dfc.properties
-rwxr-x--- 1 dmadmin dmadmin 1242 Jul 26  2020 log4j.properties
drwxr-x--- 6 dmadmin dmadmin 4096 Apr 14 07:46 test
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ rm -rf Shared/ dfc.keystore
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ mv test/dfc.keystore ./
[dmadmin@cs-0 config]$ mv test/dfcfull.properties ./
[dmadmin@cs-0 config]$ mv test/dfc.properties* ./
[dmadmin@cs-0 config]$ mv test/log4j.properties* ./
[dmadmin@cs-0 config]$ mv test/ServerApps ./
[dmadmin@cs-0 config]$ mv test/Shared ./
[dmadmin@cs-0 config]$ mv test/acs ./
[dmadmin@cs-0 config]$ mv test/bpm ./
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ ls -l
total 140
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26  2020 ServerApps
drwxr-x--- 9 dmadmin dmadmin  4096 Jul 26  2020 Shared
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26  2020 acs
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26  2020 bpm
-rw-rw-r-- 1 dmadmin dmadmin  2152 Jul 26  2020 dfc.keystore
-rw-rw-r-- 1 dmadmin dmadmin   481 Jul 26  2020 dfc.properties
-rw-rw---- 1 dmadmin dmadmin    70 Jul 22  2020 dfc.properties.bak.0
-rwxr-x--- 1 dmadmin dmadmin   242 Jul 26  2020 dfc.properties.bak.1
-rw-rw-r-- 1 dmadmin dmadmin   271 Jul 26  2020 dfc.properties.bak.2
-rw-rw-r-- 1 dmadmin dmadmin   323 Jul 26  2020 dfc.properties.bak.3
-rw-rw-r-- 1 dmadmin dmadmin   481 Jul 26  2020 dfc.properties.bak.4
-rw-rw-r-- 1 dmadmin dmadmin   482 Jul 26  2020 dfc.properties.bak.5
-rwxrwx--- 1 dmadmin dmadmin 79268 Jul 22  2020 dfcfull.properties
-rwxr-x--- 1 dmadmin dmadmin  1242 Jul 26  2020 log4j.properties
drwxr-x--- 2 dmadmin dmadmin  4096 Apr 14 07:51 test
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ ls -l test/
total 0
-rwxr-x--- 1 dmadmin dmadmin 0 Jul 22  2020 dbor.properties
-rw-rw-r-- 1 dmadmin dmadmin 0 Jul 26  2020 dbor.properties.lck
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ # With the full folder except the dbor files, dmqdocbroker is still working
[dmadmin@cs-0 config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:51:30 UTC 2021
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0200.0080
Using specified port: 1489
Successful reply from docbroker at host (cs-0) on port(1490) running software version (16.4.0200.0256 Linux64).

real    0m3.501s
user    0m6.632s
sys     0m0.666s
Wed Apr 14 07:51:34 UTC 2021
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ mv test/dbor.properties* ./
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ # With the dbor files back, dmqdocbroker isn't working anymore
[dmadmin@cs-0 config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:51:56 UTC 2021
^C
real    0m30.682s
user    0m5.001s
sys     0m0.424s
Wed Apr 14 07:52:27 UTC 2021
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ mv dbor.properties.lck test/
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ # Removing just the dbor files again, dmqdocbroker is working again
[dmadmin@cs-0 config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:52:36 UTC 2021
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0200.0080
Using specified port: 1489
Successful reply from docbroker at host (cs-0) on port(1490) running software version (16.4.0200.0256 Linux64).

real    0m3.185s
user    0m5.546s
sys     0m0.578s
Wed Apr 14 07:52:39 UTC 2021
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ ll
total 140
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26  2020 ServerApps
drwxr-x--- 9 dmadmin dmadmin  4096 Jul 26  2020 Shared
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26  2020 acs
drwxr-x--- 7 dmadmin dmadmin  4096 Jul 26  2020 bpm
-rwxr-x--- 1 dmadmin dmadmin     0 Jul 22  2020 dbor.properties
-rw-r----- 1 dmadmin dmadmin     0 Apr 14 07:52 dbor.properties.lck
-rw-rw-r-- 1 dmadmin dmadmin  2152 Jul 26  2020 dfc.keystore
-rw-rw-r-- 1 dmadmin dmadmin   481 Jul 26  2020 dfc.properties
-rw-rw---- 1 dmadmin dmadmin    70 Jul 22  2020 dfc.properties.bak.0
-rwxr-x--- 1 dmadmin dmadmin   242 Jul 26  2020 dfc.properties.bak.1
-rw-rw-r-- 1 dmadmin dmadmin   271 Jul 26  2020 dfc.properties.bak.2
-rw-rw-r-- 1 dmadmin dmadmin   323 Jul 26  2020 dfc.properties.bak.3
-rw-rw-r-- 1 dmadmin dmadmin   481 Jul 26  2020 dfc.properties.bak.4
-rw-rw-r-- 1 dmadmin dmadmin   482 Jul 26  2020 dfc.properties.bak.5
-rwxrwx--- 1 dmadmin dmadmin 79268 Jul 22  2020 dfcfull.properties
-rwxr-x--- 1 dmadmin dmadmin  1242 Jul 26  2020 log4j.properties
drwxr-x--- 2 dmadmin dmadmin  4096 Apr 14 07:52 test
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ diff dbor.properties.lck test/dbor.properties.lck
[dmadmin@cs-0 config]$

So as you can see above (it’s rather long but I wanted to put all the evidences I gathered because I still cannot believe this is the cause of the issue), just removing/renaming the empty file “dbor.properties.lck” which was there, untouched, since almost 9 months is sufficient to have the dmqdocbroker/iapi/idql working again… Trying to put back the old empty file, the issue will come back. It’s the “same” file, same content (empty), same file format, everything… The only difference would be the Inode of course and the creation/modification dates.

After some more investigations, the issue appeared to be on the NAS behind which was still having a lock on the file, somehow. For information, I also had the same behavior on a second environment but with the file “$DOCUMENTUM/config/ServerApps/identityInterprocessMutex.lock” this time… So if that even happen to you, take a look at these lock files under $DOCUMENTUM/config and make sure there are no problems with the storage.

L’article Documentum – dmqdocbroker/iapi/idql not working because of dbor.properties.lck est apparu en premier sur dbi Blog.

At a customer, the D2-REST (16.5.1) application hosted on WebLogic Server 12c started showing 500 Internal Server Errors, after a customer release including many things. The error was rather simple to replicate since opening the D2-REST Product info page was sufficient (https://<host>/D2-REST/product-info). The URL was returning the following:

At the same time, on the logs:

2021-04-26 06:46:20,340 UTC [ERROR] ([ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)') - com.emc.documentum.rest.util.LogHelper        : LogId: 9b360f83-335a-413e-87e3-481ba5cbf168, Status: 500, code: E_INTERNAL_SERVER_ERROR, message: An internal server error occurs.
org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.NoSuchMethodError: com.google.common.base.Objects.firstNonNull(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:982)
        at com.emc.documentum.rest.servlet.RestDispatcherServlet.doDispatch(RestDispatcherServlet.java:33)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:286)
        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)
        at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)
        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:350)
        at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
        at com.emc.documentum.rest.filter.ApplicationFilter.doFilter(ApplicationFilter.java:33)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
        at com.emc.documentum.d2.rest.filter.AppValidationFilter.doFilter(AppValidationFilter.java:35)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
        at com.emc.documentum.d2.rest.filter.AppInfoFilter.doFilter(AppInfoFilter.java:39)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
        at com.emc.documentum.rest.security.filter.RepositoryNamingFilter.doFilter(RepositoryNamingFilter.java:40)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
        at com.emc.documentum.rest.filter.RestCorsFilter.doFilterInternal(RestCorsFilter.java:47)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
        at com.emc.documentum.rest.filter.CompressionFilter.doFilter(CompressionFilter.java:73)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
        at com.emc.documentum.rest.log.MessageLoggingFilter.doFilter(MessageLoggingFilter.java:69)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
        at com.emc.documentum.rest.security.filter.ExceptionHandlerFilter.doFilterInternal(ExceptionHandlerFilter.java:31)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3706)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3672)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:344)
        at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
        at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
        at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
        at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2443)
        at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2291)
        at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2269)
        at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1705)
        at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1665)
        at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:272)
        at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
        at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
        at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
        at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
        at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:652)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:360)
Caused by: java.lang.NoSuchMethodError: com.google.common.base.Objects.firstNonNull(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
        at com.emc.documentum.d2fs.controller.D2AppInfoController.attribute(D2AppInfoController.java:160)
        at com.emc.documentum.d2fs.controller.D2AppInfoController.getProductInfo(D2AppInfoController.java:94)
        at com.emc.documentum.d2fs.controller.D2AppInfoController.get(D2AppInfoController.java:65)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
        ... 72 common frames omitted
2021-04-26 06:46:20,414 UTC [INFO ] ([ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)') - com.emc.documentum.rest.util.LogHelper        : XMLOutputFactory loaded com.ctc.wstx.stax.WstxOutputFactory.
2021-04-26 06:46:20,416 UTC [INFO ] ([ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)') - com.emc.documentum.rest.util.LogHelper        : XMLInputFactory loaded com.ctc.wstx.stax.WstxInputFactory.
2021-04-26 06:46:20,451 UTC [INFO ] ([ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)') - com.emc.documentum.rest.util.LogHelper        : Class com.emc.documentum.rest.config.DataBindingRuntime addLastPropertySource rest-api-data-binding.properties.

The recently deployed release contained many things but looking into it in details, the most promising suspect was the Oracle WebLogic Server PSU (+coherence patch) from April 2021. Based on the logs, this looked like a GUAVA (Google core libraries for Java) related issue. Usually, the D2-REST application would be using its own application libraries but it might happen that for some security reasons, the configuration would be changed to force WebLogic to use the Oracle provided ones instead. This would be in order to keep the third-party libraries up-to-date, as much as possible, to reduce the potential security issues. At this customer, rollback the PSU would be a rather important security problem. After looking into the details, it was clear that the method mentioned above has been deleted in GUAVA 21.0 (deprecated in 20.0). On the other hand, D2 16.5.1 comes with GUAVA 13.0.1 by default and D2-REST (+ D2-Smartview) comes with GUAVA 20.0. As part of the April PSU, this library was probably upgraded to 21.0 (I didn’t find any confirmation). Therefore, I tried to force D2-REST to re-use its internal GUAVA libraries instead (while keeping the others from WebLogic) by adding a new line inside the “<prefer-application-packages>” section:

[weblogic@wsd2rest-0 ~]$ cd $APPLICATIONS/D2-REST/WEB-INF/
[weblogic@wsd2rest-0 WEB-INF]$ cat weblogic.xml
<?xml version="1.0" encoding="UTF-8"?>

<weblogic-web-app>
  ...
  <container-descriptor>
    <!--prefer-web-inf-classes>true</prefer-web-inf-classes-->
    <prefer-application-packages>
      <package-name>org.slf4j</package-name>
      <package-name>com.google.common.*</package-name>
    </prefer-application-packages>
    <!--show-archived-real-path-enabled>true</show-archived-real-path-enabled-->
  </container-descriptor>
  ...
</weblogic-web-app>
[weblogic@wsd2rest-0 WEB-INF]$

Adding the line 11 above forces WebLogic to load the application specific packages instead of its own. After a Managed Server restart, the issue was gone, which confirms that the April PSU was the culprit:

Since we force WebLogic to not use its own jar files for some Google libraries, that means that potential security issues related to these jar files are obviously re-opened… However, at some point, you have a choice to make between being secure but having a non-working application OR potentially having some flaws but a working application. It’s obviously possible to go one-step further and instead of using “<package-name>com.google.common.*</package-name>“, which is rather generic, use a more refined definition of the package so that the scope affected is smaller.

The same applies to D2-Smartview as well since it is also a REST client, so it relies heavily on such packages…

L’article Documentum – E_INTERNAL_SERVER_ERROR on D2-REST Product page related to GUAVA libraries after WebLogic PSU est apparu en premier sur dbi Blog.

If you have ever been working on a sensitive Documentum environment (they are all sensitive, are they not?!), you might already have worked on hardening your Web Servers. One of these aspects is to have a specific set of HTTP Security Headers. In this blog, I will talk about one in particular, which is the Content-Security-Policy (CSP).

The recommendations are usually to setup a set of headers. Here is an example of header names and values (that are/should be considered secure):

• X-XSS-Protection: 1; mode=block
• X-Content-Type-Options: nosniff
• Content-Security-Policy: default-src ‘none’; script-src ‘self’; connect-src ‘self’; img-src ‘self’; style-src ‘self’;
• X-Frame-Options: SAMEORIGIN
• Cache-Control: no-cache, no-store
• Pragma: no-cache
• Strict-Transport-Security: max-age=63072000; includeSubDomains

In case you have never heard of the CSP, I like the documentation that Mozilla provides, it is very clear and provides all the necessary information for you to understand how things works as well as what you can and cannot configure.

The configuration of the CSP is really application dependent because it controls what the browser should be allowed to execute/fetch/render based on the value of the HTTP Header. With the above example, a lot of things will be disabled completely because the default-src is set to ‘none’ and everything that isn’t specifically defined in the HTTP Header will fallback with the value of the default-src parameter. This means that, for example, the browser will not even allow the load of a font from a ttf file (some application like D2 tries to load ttf files). Everything set with ‘self’ means that if it’s a resource coming from the same server (same scheme, host/dns/domain, and port), then it will be allowed. For other details, I would strongly suggest you look at the Mozilla documentation.

Applying all other HTTP Security Headers to D2 shouldn’t cause too many issues but applying the CSP as depicted in the example will completely break it. Here is a screenshot of the Google Chrome console with the “recommended” settings from a security point of view (Content-Security-Policy: default-src ‘none’; script-src ‘self’; connect-src ‘self’; img-src ‘self’; style-src ‘self’;):

Regarding the CSP, the usage of the ‘unsafe-inline’, ‘unsafe-eval’ or ‘data:’ directives are usually considered insecure. Unfortunately, most application (D2 isn’t an exception) will require some of these directives in order to work, as you can see on the above screenshot. There is always the option to use the ‘nonce-*’ or the hash value but that will require you to configure each and every resource, one by one… When you have hundreds of applications to manage that each tries to load dozens of different resources, that will most likely become an issue. Therefore, you will most probably end-up with a more relaxed configuration. Let’s try D2 with a more realistic CSP based on the above errors (Content-Security-Policy: default-src ‘none’; script-src ‘self’ ‘unsafe-inline’ ‘unsafe-eval’; connect-src ‘self’; img-src ‘self’; style-src ‘self’ ‘unsafe-inline’; font-src ‘self’; manifest-src ‘self’; frame-src ‘self’;):

That’s still not enough and that’s the purpose of this blog. The configuration, until now, is rather simple: you configure your Web Server, reload, and look for errors on the Chrome console. However, as you can see in the second screenshot, there is a problem with the D2 WSCTF plugin.

When D2 is configured to use the WSCTF plugin, it will actually execute a piece of code on your workstation that is being accessed by D2 (by the browser) through a socket using the associated protocol (WebSockets Secure – “wss://”). Therefore, this needs to be added into the allowed connection source using “connect-src wss:”. Unless I’m mistaken, I don’t think it is possible to filter this configuration further. However, doing that isn’t sufficient, it will still fail with the latest error shown in the previous screenshot: Refused to frame ” because it violates the following Content Security Policy directive: “frame-src ‘self'”. The frame ” is actually also because of the WSCTF plugin, to avoid redirections at the browser level when D2 talks to the plugin. Documentum created its own custom protocol that is being used for that purpose and that’s what is still missing.

In order to fix this issue and allow the WSCTF plugin to work, the needed configuration is “frame-src dctmctf:”. This might be documented somewhere but I have never seen it before. To find that, I have been looking at the JavaScript code being executed in the browser (by putting a breakpoint) and it gave me the following:

As shown, the frame being started begins with “dctmctf:” and therefore, allowing the frame source on that scheme is fixing the issue (yes all the messages are in red, meaning it’s an “ERROR” but that’s how D2 prints these info messages…):

Therefore, in case you are using D2 (and a lot of other applications), a more realistic CSP configuration will most probably be something like:

Content-Security-Policy: default-src ‘none’; script-src ‘self’ ‘unsafe-inline’ ‘unsafe-eval’; connect-src ‘self’ wss:; img-src ‘self’ data:; style-src ‘self’ ‘unsafe-inline’; font-src ‘self’; manifest-src ‘self’; frame-src ‘self’ dctmctf:;

As mentioned at the beginning of this blog, CSP is really application dependent. Unfortunately, most apps aren’t built with CSP in mind and therefore you must make concessions to be able to strengthen your Web Servers without breaking your applications.

L’article Documentum – Applying Content-Security-Policy (CSP) on D2 while using WSCTF plugin est apparu en premier sur dbi Blog.

A few weeks ago, an issue was identified at a customer which would cause the value of “dm_server_config.r_server_version” to be reset to the GA version number (16.4 P00) instead of the currently deployed patch (which is 16.4 P26) and that would happen randomly on some of the Content Servers but not all. The different Content Servers are all deployed on a Kubernetes Cluster, are using a single image (which contains the patched binaries) and are using replicas to provide High Availability. That really means that it’s the same image everywhere but despite that, some of them would, at some point, end-up with the P00 shown.

Every time the pods would start/restart, the “dm_server_config.r_server_version” would properly display the P26 version, without exception. I spent several hours doing testing on that issue, but I was never able to replicate the issue by simply restarting the Content Server. At startup, a Content Server will update the value of “dm_server_config.r_server_version” with the currently used binaries (returned by the “documentum -v” command). I tried enabling the RPC and SQL traces after some discussion with the OpenText Support, but it didn’t show anything useful.

Since it appeared randomly and since I couldn’t replicate the issue by restarting the Content Server, I just simply let an environment up&running without any user activity on it and I was checking every day the value of the “dm_server_config.r_server_version”. During the week, nothing happened, the P26 was constantly shown and the dm_server_config object wasn’t updated in any way. However, after the weekend, it was suddenly showing P00 so I started looking into the details:

[morgan@k8s_master ~]$ kubectl get pod cs-0
NAME   READY   STATUS    RESTARTS   AGE
cs-0   1/1     Running   0          6d22h
[morgan@k8s_master ~]$
[morgan@k8s_master ~]$ kubectl describe pod cs-0 | grep Started
      Started:      Mon, 31 Jan 2022 10:53:48 +0100
[morgan@k8s_master ~]$
[morgan@k8s_master ~]$ kubectl exec -it cs-0 -- bash -l
[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ date; iapi ${DOCBASE_NAME} -Udmadmin -Pxxx << EOC
> retrieve,c,dm_server_config
> dump,c,l
> EOC
Mon Feb  7 07:57:00 UTC 2022
...
SYSTEM ATTRIBUTES

  r_object_type                   : dm_server_config
  r_creation_date                 : 7/15/2021 11:38:28
  r_modify_date                   : 2/6/2022 03:19:25
  ...
  r_server_version                : 16.4.0000.0248  Linux64.Oracle
  r_host_name                     : cs-0
  ...

API> Bye

[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ grep --color "16.4.0" $DOCUMENTUM/dba/log/${DOCBASE_NAME}.log
    OpenText Documentum Content Server (version 16.4.0260.0296  Linux64.Oracle)
2022-01-31T09:59:36.943158      6994[6994]      0000000000000000        [DM_FULLTEXT_T_QUERY_PLUGIN_VERSION]info:  "Loaded FT Query Plugin: /app/dctm/server/product/16.4/bin/libDsearchQueryPlugin.so, API Interface version: 1.0, Build number: HEAD; Feb 14 2018 04:27:20, FT Engine version: xPlore version 16.4.0160.0089"
Mon Jan 31 10:00:50 2022 [INFORMATION] [AGENTEXEC 8215] Detected during program initialization: Version: 16.4.0260.0296  Linux64
2022-02-06T03:04:00.114945      23067[23067]    0000000000000000        [DM_FULLTEXT_T_QUERY_PLUGIN_VERSION]info:  "Loaded FT Query Plugin: /app/dctm/server/product/16.4/bin/libDsearchQueryPlugin.so, API Interface version: 1.0, Build number: HEAD; Feb 14 2018 04:27:20, FT Engine version: xPlore version 16.4.0160.0089"
2022-02-06T03:19:25.601602      31553[31553]    0000000000000000        [DM_FULLTEXT_T_QUERY_PLUGIN_VERSION]info:  "Loaded FT Query Plugin: /app/dctm/server/product/16.4/bin/libDsearchQueryPlugin.so, API Interface version: 1.0, Build number: HEAD; Feb 14 2018 04:27:20, FT Engine version: xPlore version 16.4.0160.0089"
[morgan@k8s_master ~]$

As you can see on the above output, the modification date of the dm_server_config was Sunday 6-Feb-2022 at 03:19:25 UTC while the repository started on the Monday 31-Jan-2022 at 09:59 UTC. Until the Friday 4-Feb-2022, the returned version was P26 (16.4.0260.0296) but then after the weekend, it was P00 (16.4.0000.0248). The grep command above was used initially to verify that the repository started with the P26, that’s the very first line of the Repository log file: “OpenText Documentum Content Server (version 16.4.0260.0296 Linux64.Oracle)”. However, on the Sunday morning, suddenly there are two new messages shown related to the FT Query Plugin initialization. This usually means that the Content Server was re-initialized and that’s indeed what happened:

[dmadmin@cs-0 ~]$ grep "DM_SESSION_I_INIT_BEGIN.*Initialize Server Configuration" $DOCUMENTUM/dba/log/${DOCBASE_NAME}.log
2022-01-31T09:59:34.302520      6994[6994]      0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Server Configuration."
2022-02-06T03:03:56.846840      23067[23067]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Server Configuration."
2022-02-06T03:19:23.644453      31553[31553]    0000000000000000        [DM_SESSION_I_INIT_BEGIN]info:  "Initialize Server Configuration."
[dmadmin@cs-0 ~]$

Therefore, something must have triggered the reinit and that would most probably be a job. One that would have run around 03:04 UTC and one around 03:20 UTC. Looking at the dm_sysobject created around that time gave me two good candidates of jobs that would have performed the reinit of the Content Server and therefore that might be the cause of the switch from P26 to P00:

API> ?,c,select r_object_id, r_creation_date, r_modify_date, object_name from dm_sysobject where r_creation_date>=date('06.02.2022 03:00:00','dd.mm.yyyy hh:mi:ss') and r_creation_date<=date('06.02.2022 03:25:00','dd.mm.yyyy hh:mi:ss');
r_object_id       r_creation_date            r_modify_date              object_name
----------------  -------------------------  -------------------------  ----------------------------------
090f45158029f8a6  2/6/2022 03:03:13          2/6/2022 03:03:14          2/6/2022 03:03:11 dm_Initialize_WQ
090f45158029f8a9  2/6/2022 03:04:14          2/6/2022 03:04:15          2/6/2022 03:03:44 dm_DMClean
090f45158029f8aa  2/6/2022 03:04:11          2/6/2022 03:04:11          Result.dmclean
090f45158029f8ad  2/6/2022 03:04:15          2/6/2022 03:04:16          2/6/2022 03:04:14 dm_WfmsTimer
090f45158029f8b4  2/6/2022 03:11:46          2/6/2022 03:11:47          2/6/2022 03:11:42 dm_Initialize_WQ
090f45158029f8b7  2/6/2022 03:12:52          2/6/2022 03:12:53          2/6/2022 03:12:12 DocumentTraining
090f45158029fc1e  2/6/2022 03:16:15          2/6/2022 03:16:15          2/6/2022 03:16:13 dm_Initialize_WQ
090f45158029fc21  2/6/2022 03:22:17          2/6/2022 03:22:17          2/6/2022 03:19:13 dm_DMFilescan
090f45158029fc22  2/6/2022 03:22:14          2/6/2022 03:22:14          Result.dmfilescan
090f45158029fc28  2/6/2022 03:23:48          2/6/2022 03:23:48          2/6/2022 03:23:43 dm_Initialize_WQ
(10 rows affected)

As shown above, the first reinit was most probably triggered by the dm_DMClean job while the second one was most probably coming from the dm_DMFilescan job: if you look at the start time of these jobs (check the object_name) and the completion time (check the Result line), then the reinit is right in the middle of it. Just in case, looking at the “a_last_completion” for these two jobs also confirmed the same:

API> ?,c,select r_object_id, a_last_completion, a_next_invocation, object_name from dm_job where object_name in ('dm_DMClean','dm_DMFilescan');
r_object_id       a_last_completion          a_next_invocation          object_name
----------------  -------------------------  -------------------------  -------------
080f45158000035b  2/6/2022 03:04:14          2/13/2022 03:00:00         dm_DMClean
080f45158000035c  2/6/2022 03:22:17          2/13/2022 03:15:00         dm_DMFilescan

Knowing that I got two good candidates, I obviously had to try manually to reproduce the issue. Therefore, I restarted the repository to get back to P26:

[dmadmin@cs-0 ~]$ date; iapi ${DOCBASE_NAME} -Udmadmin -Pxxx << EOC
> ?,c,select r_object_id, r_creation_date, r_modify_date, r_server_version from dm_server_config;
> ?,c,select r_object_id, a_last_completion, a_next_invocation, object_name from dm_job where object_name in ('dm_DMClean','dm_DMFilescan');
> EOC
Mon Feb  7 08:45:13 UTC 2022
...
Session id is s0
API> 
r_object_id       r_creation_date            r_modify_date              r_server_version
----------------  -------------------------  -------------------------  --------------------------------
3d0f451580000102  7/15/2021 11:38:28         2/7/2022 08:41:02          16.4.0260.0296  Linux64.Oracle
(1 row affected)

API> 
r_object_id       a_last_completion          a_next_invocation          object_name
----------------  -------------------------  -------------------------  -------------
080f45158000035b  2/6/2022 03:04:14          2/13/2022 03:00:00         dm_DMClean
080f45158000035c  2/6/2022 03:22:17          2/13/2022 03:15:00         dm_DMFilescan
(2 rows affected)

API> Bye
[dmadmin@cs-0 ~]$

Then, I ran the dm_DMClean (updated the a_next_invocation and window_interval so that the job can start), checked that it performed the reinit of the Content Server and verified the “dm_server_config.r_server_version” value:

[dmadmin@cs-0 ~]$ date; iapi ${DOCBASE_NAME} -Udmadmin -Pxxx << EOC
> ?,c,select r_object_id, r_creation_date, r_modify_date, r_server_version from dm_server_config;
> ?,c,select r_object_id, a_last_completion, a_next_invocation, object_name from dm_job where object_name in ('dm_DMClean','dm_DMFilescan');
> EOC
Mon Feb  7 08:50:39 UTC 2022
...
Session id is s0
API> 
r_object_id       r_creation_date            r_modify_date              r_server_version
----------------  -------------------------  -------------------------  --------------------------------
3d0f451580000102  7/15/2021 11:38:28         2/7/2022 08:41:02          16.4.0260.0296  Linux64.Oracle
(1 row affected)

API> 
r_object_id       a_last_completion          a_next_invocation          object_name
----------------  -------------------------  -------------------------  -------------
080f45158000035b  2/7/2022 08:49:19          2/8/2022 03:00:00          dm_DMClean
080f45158000035c  2/6/2022 03:22:17          2/8/2022 03:15:00          dm_DMFilescan
(2 rows affected)

API> Bye
[dmadmin@cs-0 ~]$

The reinit of the Content Server happened but it didn’t change the “dm_server_config.r_modify_date”, maybe because it was already showing P26 so nothing had to be updated? The only thing that changed is the “dm_job.a_last_completion” obviously, since the job ran. This means that the dm_DMClean is probably not the culprit, so I did the same for the dm_DMFilescan:

[dmadmin@cs-0 ~]$ date; iapi ${DOCBASE_NAME} -Udmadmin -Pxxx << EOC
> ?,c,select r_object_id, r_creation_date, r_modify_date, r_server_version from dm_server_config;
> ?,c,select r_object_id, a_last_completion, a_next_invocation, object_name from dm_job where object_name in ('dm_DMClean','dm_DMFilescan');
> EOC
Mon Feb  7 08:59:23 UTC 2022
...
Session id is s0
API> 
r_object_id       r_creation_date            r_modify_date              r_server_version
----------------  -------------------------  -------------------------  --------------------------------
3d0f451580000102  7/15/2021 11:38:28         2/7/2022 08:52:34          16.4.0000.0248  Linux64.Oracle
(1 row affected)

API> 
r_object_id       a_last_completion          a_next_invocation          object_name
----------------  -------------------------  -------------------------  -------------
080f45158000035b  2/7/2022 08:49:19          2/8/2022 03:00:00          dm_DMClean
080f45158000035c  2/7/2022 08:58:44          2/8/2022 03:15:00          dm_DMFilescan
(2 rows affected)

API> Bye
[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ grep Report $DOCUMENTUM/dba/log/${DOCBASE_NAME}/sysadmin/DMFilescanDoc.txt
DMFilescan Report For DocBase REPO01 As Of 2/7/2022 08:52:22
Report End  2/7/2022 08:58:43
[dmadmin@cs-0 ~]$

As you can see above, the dm_DMFilescan did change the “dm_server_config.r_server_version” to P00 and therefore the “dm_server_config.r_modify_date” was also updated. Checking the dm_DMFilescan job report shows that it took around 6min to complete and the update of the dm_server_config object happened around 10s after the start of the job.

Therefore, the reason why the “dm_server_config.r_server_version” is being changed “randomly” from P26 back to P00 isn’t actually random but it’s due to the execution of the dm_DMFilescan job. On HA environment, since this job can run on any of the available Content Servers, it gave a sense of randomness but it’s not. The same information was provided to OpenText and the bug CS-136387 was opened for the same.

While doing further checks to try to understand the root cause, I saw the following on the method’s dmbasic scripts:

[dmadmin@cs-0 ~]$ cd $DM_HOME/bin/
[dmadmin@cs-0 bin]$
[dmadmin@cs-0 bin]$ documentum -v
OpenText Documentum Release Version: 16.4.0260.0296  Linux64.Oracle
[dmadmin@cs-0 bin]$
[dmadmin@cs-0 bin]$ ls -ltr dmfilescan* dmclean*
-rwxr-x--- 1 dmadmin dmadmin 13749675 Nov 11 13:38 dmfilescan
-rwxr-x--- 1 dmadmin dmadmin 13769063 Nov 11 13:38 dmclean.patch.bak
-rwxr-x--- 1 dmadmin dmadmin 13866290 Nov 11 13:39 dmclean
[dmadmin@cs-0 bin]$
[dmadmin@cs-0 bin]$ strings dmfilescan | grep "16.4"
16.4.0000.0248
[dmadmin@cs-0 bin]$
[dmadmin@cs-0 bin]$ strings dmclean.patch.bak | grep "16.4"
16.4.0000.0248
[dmadmin@cs-0 bin]$
[dmadmin@cs-0 bin]$ strings dmclean | grep "16.4"
16.4.0260.0296
[dmadmin@cs-0 bin]$

As shown above, the reason is the Documentum patching of the binaries:

• dmclean: the dmbasic script is being updated properly and the version number that seems to be hardcoded inside it reflects the P26
• dmfilescan: the dmbasic script isn’t being updated by the patch (there is no “*.patch.bak” file) and therefore it still contains the P00 hardcoded version

L’article Documentum – r_server_version reset to P00 instead of correct patch version est apparu en premier sur dbi Blog.

This is part II of the article, see Part I here

The global_parameters file

Here is an example of this file: dbi services.

Besides the location of the software packages and installation directory, this mandatory sourceable bash file contains all the settings required to create a seed docbase and any instantiation of it. Those settings have the following syntax:

stem_parameter-name=value

e.g.:
DCTM0_DOCBASE=seed
DCTM0_DOCBASE_ID=1
DCTM0_SERVICE_NAME=${DCTM0_DOCBASE}
DCTM0_SERVICE_PORT=50000
DCTM0_RDBMS=oracle
DCTM0_DATABASE_OWNER=${DCTM0_DOCBASE}
DCTM0_DATABASE_PASSWORD=${DCTM0_DOCBASE}
[[ ${DCTM0_RDBMS} == "postgres" ]] && DCTM0_DB_SERVER_PORT=5432
DCTM0_ROOT=${dctm_root}/${DCTM0_DOCBASE}
DCTM0_HOST=cs1
DCTM0_INSTALL_OWNER=${dctm_owner}
DCTM0_INSTALL_PASSWORD=${DCTM0_INSTALL_OWNER}
DCTM0_DOCBROKER_NAME=docbroker
DCTM0_DOCBROKER_PORT=1489
DCTM0_SERVER_HTTP_PORT=9080
DCTM0_SERVER_HTTP_MEMORY="-Xms512m -Xmx512m"

Here, a docbase named seed is defined with an id of 1, a TCP/IP port of 50000, etc. Its stem is DCTM0. Although it may look overly complicated, this prefix trick allows to define several docbases at once in global_parameters without parameter name clash since the prefix, or stem, make them unique. This way, a history of the docbases can also be kept in the same file. However, it is mostly useful in that it allows to create or instantiate several docbases at the same time, see later for more about this. For the curious ones, the stem idea comes from the Rexx programming language where stems are used to implement multi-dimensional arrays, trees, lists, records, etc., an original take on data structure syntax.

Of course, these indirections must be resolved to access the values; this is done in function global_parameter.set_role() by using bash’s dereferencing syntax, e.g.:

local tmp=${stem}_DOCBASE_ID; tmp=${!tmp}
echo "${title} docbase id is : ${tmp}"

On line 1, the name of the variable of interest is computed, e.g. DCTM10_DOCBASE_ID for ${stem} set to DCTM10; next, this variable name is dereferenced (or expanded in bash’s parlance) into ${tmp} using the ${!var} notation.

At any one time, we only need to know the docbase to be created or the seed docbase and the docbase to be instantiated into. Their respective settings will be stored in the variables ${ACTIVE_*} and ${SEED_*} after expansion.

global_parameters takes up to 2 parameters when invoked:

Usage
. global_parameters [seed_repo [new_repo]]

where:
seed_repo is the stem of the repository to be created or used as a seed;
it defaults to DCTM0 and is used in create_docbase.sh and instantiate_docbase.sh.
for creation, it points to the specs of the docbase to be created and that can be used later as a seed;
for instantiation, seed_repo is the stem of the repository to be taken as the seed during the instantiation and new_repo is the one of the docbase to instantiate.
all repositories must have their specs defined in this global_parameters file.

global_parameters is usually invoked by the scripts create_docbase.sh and instantiate_docbase.sh with the same command-line parameters they receive. There is no real need to invoke it directly since a tailored environment for each repository after its instantiation can be set using the more adequate function swr() (standing for SWitch Repository) defined in dmadmin’s ~/.profile by create_docbase.sh, more on this later. This also prevents polluting the environment with the settings of all the docbases defined in there. However, being a bash file, it also allows to define several useful functions:

function show_repo() stem [title]
function show_seed()
function show_active()
function set_role() (SEED | ACTIVE) stem

show_repo() stem [title] displays the parameters of the repository as dereferenced from the given stem, with title as explanatory string, which may be empty. E.g.:

$ show_repo DCTM10 "stem DCTM10 defines:"
stem DCTM10 defines: docbase name is : repo10
stem DCTM10 defines: docbase id is : 100011
stem DCTM10 defines: docbase service name is : repo10
stem DCTM10 defines: docbase service port is : 50100
stem DCTM10 defines: docbase host is : cs2
stem DCTM10 defines: docbase root directory is : /u01/dctm/repo10
stem DCTM10 defines: docbase owner is : dmadmin
stem DCTM10 defines: installer password is : dmadmin
stem DCTM10 defines: docbase docbroker name : docbrokerrepo010
stem DCTM10 defines: docbase docbroker port : 1507
stem DCTM10 defines: RDBMS is : postgres
stem DCTM10 defines: database owner is : repo10
stem DCTM10 defines: database password is : repo10
stem DCTM10 defines: database server port is : 5500
stem DCTM10 defines: docbase http server port : 9760
stem DCTM10 defines: docbase http server memory : -Xms512m -Xmx1024m
Scripts'dir is /mnt/shared/blog-seed-docbase

$ show_repo DCTM1
docbase name is : repo01
docbase id is : 10001
docbase service name is : repo01
docbase service port is : 50010
docbase host is : cs2
docbase root directory is : /u01/dctm/repo01
docbase owner is : dmadmin
installer password is : dmadmin
docbase docbroker name : docbroker
docbase docbroker port : 1489
RDBMS is : oracle
database owner is : repo01
database password is : repo01
database server port is : N/A
docbase http server port : 9180
docbase http server memory : -Xms512m -Xmx1024m
Scripts'dir is /mnt/shared/blog-seed-docbase

show_repo() can be manually invoked to verify the repository settings before they are used by create_docbase.sh and instantiate_docbase.sh.

In addition to the defined stems, the roles SEED and ACTIVE are considered stems too but are set dynamically by set_role(). Thus:

$ set_role SEED DCTM1
SEED docbase name is : repo01
SEED docbase id is : 10001
SEED docbase service name is : repo01
SEED docbase service port is : 50010
SEED docbase host is : cs2
SEED docbase root directory is : /u01/dctm/repo01
SEED docbase owner is : dmadmin
SEED installer password is : dmadmin
SEED docbase docbroker name : docbroker
SEED docbase docbroker port : 1489
SEED RDBMS is : oracle
SEED database owner is : repo01
SEED database password is : repo01
SEED database server port is : N/A
SEED docbase http server port : 9180
SEED docbase http server memory : -Xms512m -Xmx1024m
Scripts'dir is /mnt/shared/blog-seed-docbase

$ show_repo SEED Seed
Seed docbase name is : repo01
Seed docbase id is : 10001
Seed docbase service name is : repo01
Seed docbase service port is : 50010
Seed docbase host is : cs2
Seed docbase root directory is : /u01/dctm/repo01
Seed docbase owner is : dmadmin
Seed installer password is : dmadmin
Seed docbase docbroker name : docbroker
Seed docbase docbroker port : 1489
Seed RDBMS is : oracle
Seed database owner is : repo01
Seed database password is : repo01
Seed database server port is : N/A
Seed docbase http server port : 9180
Seed docbase http server memory : -Xms512m -Xmx1024m
Scripts'dir is /mnt/shared/blog-seed-docbase

$ set_role ACTIVE DCTM10
ACTIVE docbase name is : repo10
ACTIVE docbase id is : 100011
ACTIVE docbase service name is : repo10
ACTIVE docbase service port is : 50100
ACTIVE docbase host is : cs2
ACTIVE docbase root directory is : /u01/dctm/repo10
ACTIVE docbase owner is : dmadmin
ACTIVE installer password is : dmadmin
ACTIVE docbase docbroker name : docbrokerrepo010
ACTIVE docbase docbroker port : 1507
ACTIVE RDBMS is : postgres
ACTIVE database owner is : repo10
ACTIVE database password is : repo10
ACTIVE database server port is : 5500
ACTIVE docbase http server port : 9760
ACTIVE docbase http server memory : -Xms512m -Xmx1024m
Scripts'dir is /mnt/shared/blog-seed-docbase

$ show_repo ACTIVE Active
Active docbase name is : repo10
Active docbase id is : 100011
Active docbase service name is : repo10
Active docbase service port is : 50100
Active docbase host is : cs2
Active docbase root directory is : /u01/dctm/repo10
Active docbase owner is : dmadmin
Active installer password is : dmadmin
Active docbase docbroker name : docbrokerrepo010
Active docbase docbroker port : 1507
Active RDBMS is : postgres
Active database owner is : repo10
Active database password is : repo10
Active database server port is : 5500
Active docbase http server port : 9760
Active docbase http server memory : -Xms512m -Xmx1024m
Scripts'dir is /mnt/shared/blog-seed-docbase

# same effect with:
$ . global_parameters DCTM1 DCTM10
SEED docbase name is : repo01
SEED docbase id is : 10001
SEED docbase service name is : repo01
SEED docbase service port is : 50010
SEED docbase host is : cs2
SEED docbase root directory is : /u01/dctm/repo01
SEED docbase owner is : dmadmin
SEED installer password is : dmadmin
SEED docbase docbroker name : docbroker
SEED docbase docbroker port : 1489
SEED RDBMS is : oracle
SEED database owner is : repo01
SEED database password is : repo01
SEED database server port is : N/A
SEED docbase http server port : 9180
SEED docbase http server memory : -Xms512m -Xmx1024m
Scripts'dir is /mnt/shared/blog-seed-docbase

ACTIVE docbase name is : repo10
ACTIVE docbase id is : 100011
ACTIVE docbase service name is : repo10
ACTIVE docbase service port is : 50100
ACTIVE docbase host is : cs2
ACTIVE docbase root directory is : /u01/dctm/repo10
ACTIVE docbase owner is : dmadmin
ACTIVE installer password is : dmadmin
ACTIVE docbase docbroker name : docbrokerrepo010
ACTIVE docbase docbroker port : 1507
ACTIVE RDBMS is : postgres
ACTIVE database owner is : repo10
ACTIVE database password is : repo10
ACTIVE database server port is : 5500
ACTIVE docbase http server port : 9760
ACTIVE docbase http server memory : -Xms512m -Xmx1024m
Scripts'dir is /mnt/shared/blog-seed-docbase

set_role() is invoked by global_parameters with the following logic:

• when global_parameters is invoked with one or no parameter, the repository the stem (or DCTM0 if no stem was given) resolves into receives the ACTIVE role;
• when 2 stems are passed as parameters, the docbase the first stem resolves into receives the SEED role, and the one the second stem resolves into receives the ACTIVE role;

Obviously, the SEED role is only used in the script instantiate_docbase.sh and refers to the model docbase. The ACTIVE role is used in both scripts to refer to the docbase to be created or instantiated.

Several settings relate to the supported RDBMS, Oracle or PostgreSQL:

if [[ "${ACTIVE_RDBMS}" == "oracle" ]]; then
   cs_package=documentum_server_22.4_linux64_oracle.tar
   oracle_download_url=https://download.oracle.com/otn_software/linux/instantclient/217000
   oracle_ic_basic=instantclient-basic-linux.x64-21.7.0.0.0dbru.zip
   oracle_ic_sqlplus=instantclient-sqlplus-linux.x64-21.7.0.0.0dbru.zip
   db_server_host_account=oracle
   db_server_host_password=${db_server_host_account}
   db_server_host_alias=db
   db_server_host_ip_address=192.168.0.21
   db_listener_port=1521
   # although distinct concepts, the connect string and the service name must be equal to satisfy the migration utility;
   db_connect_string=pdb1
   db_service_name=pdb1
   db_sys_account=sys
   db_sys_password=${db_sys_account}
   db_datafile_root=/u02/oradata/ORCL/${db_service_name}
   db_remote_connect_string=orcl
elif [[ "${ACTIVE_RDBMS}" == "postgres" ]]; then
   cs_package=documentum_server_22.4_linux64_postgres.tar
   postgresql_download_url=https://ftp.postgresql.org/pub/source
   jdbc_postgresql_download_url=https://jdbc.postgresql.org/download
   postgresql_custom_package=postgresql-15.1.tar.gz
   # set this to yes or no if a compilation is needed to produce the custom package;
   postgresql_compile=no
   postgresql_jdbc_package=postgresql-42.5.1.jar
   db_server_host_alias=localhost
   db_connect_string=${ACTIVE_DOCBASE}
else
   echo "Invalid or Unsupported RDBMS [${ACTIVE_RDBMS}]"
  rc=1
fi

The packages to download and their URLs are specified there. For the Oracle RDBMS, we will use the InstantClient and SQL*Plus. The InstantClient includes the JDBC drivers required by the Migration Utility. For the PostgreSQL RDBMS, we download the source package and the latest JDBC drivers from the global_properties.${*postgresql_download_url} locations.

For Oracle, we need to connect to the host to perform some tasks, e.g. schema export and import, cleanup; thus, we need the host’s IP address and account. For PostgreSQL, this is not needed as the database is always local (embedded). For Oracle, we also need the sys credentials to manage accounts in the existing database. If this is delegating too much power, especially if the database is shared among several unrelated applications, the database accounts need to be created in advance and the scripts be adapted consequently. When using Oracle RDBMS, a database exclusively dedicated to the instantiated repositories and shared among them is the ideal situation. In this context, a full access to the sys account is justified but this may depend on your organization. Evidently, open-source, free software offers much more flexibility in this regard.

Finally, we declare the name of the connect string to use to connect to the database; for Oracle, it will be the SQL*Net alias that will be defined in the tnsnames.ora file; for PostgreSQL, it will be the name of the INI section that will be defined in dmadmin‘s ~/.odbc.ini file since the database is accessed through the ODBC API. More on these parameters in the paragraph dedicated to the database.

A trendy alternative to this parameter=value file format could be json or yaml files, with an intermediate conversion to bash syntax using the parsing tools jq and yq. Another excellent alternative would consist in storing each docbase’s settings in its own file and provide the file(s) to source as command-line arguments. That would allow to not use the stem trick if found too complicated. There are many ways to jungle between different sets of configurations but the stem one is quite a simple and easy one to use as it is pure, executable bash and therefore does not need any intermediate translation; moreover, it centralizes into one unique file all the docbase definitions to be used. Adapt as you see fit.

See part III here

L’article A quick repository creation utility (part II) est apparu en premier sur dbi Blog.

Some things never change: gravity, human foolishness, and the length of content servers’ installation time. Indeed, have you ever noticed how long it takes to create a new Documentum repository ? I’m not talking of the time spent interactively behind a keyboard to type commands or answer prompts from the installation and configuration tools, those are just tedious, but of the time actually spent by those tools to deliver the new repository once all the settings have been provided. Whether it happens in a VM or a container, interactive or automated, the final approach is the same slow one. Most of it is spent inside the database creating tables (414 tables), views (974 views) and indexes (536 indexes, including the implicit ones for non-nullity constraints), for a total of 1928 user objects in ContentServer v22.2, and populate them; plus, initializing the new repository with system objects, which will ultimately also affects the database. Comparatively, not so much time is spent at copying and configuring files around, even though there are about 13’000 of them in the CS v22.2. Clearly, if the database part could be sped up, it would make the whole process much less time-consuming. A lot of time, in the order of several total minutes, is also spent waiting for processes to start or stop, e.g. the method server; removing these inefficient delays could also substantially reduce the overall installation time.

So, here is a challenge: is there a way to speed up the creation of a repository by optimizing some of its steps ? As you can guess, there is, partly thanks to a tool Documentum has made available since Content Server v7.3, the Migration Utility.

In this article, I propose to use an out of the box docbase as a seed repository, copy it along with the Documentum binaries anywhere it is deemed necessary, on the same host or on a different one, and launch the Documentum’s Migration Utility to change its name, docbase id, host name and/or server name as needed. The procedure, let’s call it henceforth docbase instantiation, can be repeated as many times as needed, even in parallel (which is normally not recommended by the vendor), whenever new, fresh, distinct repositories are necessary, either temporarily or permanently.

Besides the technical challenge, the use cases are aplenty: quickly instantiate a docbase to rehearse a migration, apply an update or patch procedure, deploy a development docbase, test a Documentum’s feature inside a fresh repository, test an application’s deployment, rollback some heavy customizations, unit testing or application testing, etc., basically anything that pertains to development and testing. To be fair, nothing precludes applying this procedure in production once one has gained sufficient confidence with the docbases churned out this way.

To wet your appetite, let’s just say that on my 8-year old, puffing and panting but sturdy laptop, instantiating a new repository out of the seed one with a new name and id takes less than 6 minutes, considerably less that the 25 minutes of a standard, even silent, creation on the same machine. On a much more recent 4-year old machine, with a lighter virtualization configuration (linux containers vs. VMs), its took 2.5 minutes down from 15 minutes. On the same infrastructure but using a private and local PostgreSQL database instead of a remote Oracle one, this time drops down to less than 1 minute. Did I get your attention yet ? If so, read on !

Of course, the absolute timings don’t mean much as they are subject to the infrastructure in use but the ratio does: at least a remarkable 4-fold on old hardware, and a more than 15 times speed increase with a local PostgreSQL database on more recent hardware/infrastructure. Even on faster hardware, where the speed gain is less appealing, the automated instantiating procedure is still convenient thanks to its simplicity.

The seed docbase approach

So far, we used to clone repositories using long and daunting procedures (see e.g. Knowledge Base articles KB8966391, KB8715468, KB0528105, and KB0499567 at OpenText). While renaming a docbase is explained in OTX note KB7713200, before the Migration Utility there was no official in-place way to change a docbase id. Such a change must be applied to all the docbase’s objects metadata and all the references to object ids in configuration files (e.g. in server.ini) on disk. The risk is to forget or misinterpret some embedded docbase id in some strings in the database or in disk files; thus, only the vendor could really propose an exhaustive and reliable way to do it. Without the Migration Utility, the only safe way at our disposal to do that is to first create an empty repository with the new name and id, and importing in there the original repository’s documents, quite a complicated and boring process. Consequently, it is easier to just copy a whole docbase somewhere else and use it from there. Such clones are nonetheless quite practical as long as they don’t attempt to project to the same docbroker(s) (i.e. they don’t have any common docbroker) and their docbrokers are not listed in the same client’s dfc.properties file. A DFC client wishing to connect to any of the clones would either use a tailored dfc.properties file with the clone docbroker’s host as the sole occurrence of the clone docbase (plus, optionally, any other docbroker hosts of non conflicting repositories) as discussed in the article Connecting to a Repository via a Dynamically Edited dfc.properties File (part I) or use the enhancement presented in the article Connecting to Repositories with the Same Name and/or ID, which require some code editing and is therefore not possible in closed-source applications.

Experience shows that while stand-alone clones are acceptable at first, sooner or later they must be accessed together by the same client (e.g. some third-party service such as a pdf rendition service), and that’s where the name and id restrictions kick in. Failure to remember their common origin yields strange, hard to diagnose error messages, e.g. non-existing objects with ids seemingly coming out of thin air.

To remove all these limitations, the repository’s name and id have to be changed, which is the Migration Utility’s main purpose. Once done, as many clone repositories as needed can run on any machine, the same machine or different ones, and project to common docbrokers, or dedicated ones listed in the same client’s dfc.properties file. This enhancement was missed for so long that it is a real relief to finally have it, so kudos to OpenText.

Besides the docbase’s name and id, the Migration Utility also allows to change the host name, the owner name, server name and/or password, and all these from one single place. The most time consuming part is the id change because each and every repository object has an id (or more) containing the 5 hexadedimal digits of the docbase id which must be corrected. However, in a freshly created docbase containing less than 1’800 objects, this step is completed quickly at the database level. The Migration Utility itself can be used against any repository, not necessarily an empty one but of course the more objects it contains (e.g. in a production repository), the longer it can potentially take to complete, with a considerable stress on the database if the update operation is transactional. Here, as we are starting with an empty docbase, the migration cannot go any faster and, in case of a failure, the instantiation procedure using the seed can be restarted confidently as many times as needed after performing some clean up to roll back the changes done so far.

In order to prevent distracting issues, the copied repository must run under the same O/S version as the original one, which won’t be an limitation if the host is the same as the seed’s (e.g. identical machine or a precisely configured O/S such as in containers); the RDBMS’s version can be more relaxed as Documentum only uses very basic functionalities from it (tables, views and indexes, and foreign keys, not much more, although Oracle’s index-organized tables were also used in a few occasions), but it must be the same RDBMS software as the seed docbase’s (i.e. either Oracle or PostgreSQL in both the seed and the copy; the other RDBMS supported by Documentum, DB2 and SQLServer, were not tested). In this project, we worked in the following environments:

• O/S

Oracle Linux 8.4 in a VirtualBox VM; also tested with Ubuntu Linux 22.0.4 (Jammy Jellyfish) in a linux container managed by Proxmox.

• RDBMS

Oracle RDBMS v12.1.0.1.0 (12c) on a remote VM, the same database for both the seed and clone docbases but distinct schemas of course.

Also tested with Oracle RDBMS v21.3.0.0.0 in a linux container running Oracle Linux 8.6, and with PostgreSQL v15.1, the most recent as of this writing.

• Database connectivity

Oracle InstantClient 21.7.0.0 and ODBC drivers v42.5.1 for PostgreSQL.

For the Migration Utility: Oracle JDBC drivers included with the InstantClient and JDBC v42.5.1 for PostgreSQL.

• JDK

AWS Corretto JDK v11.0.14.

• Content Server

Content Server v22.2 and v22.4.

Some of the above products’ versions are not implicitly mentioned in the system requirement guides, which does not mean they don’t work, only that they were not tested, which is understandable given the large number of possible products’ and versions’ combinations. In this project, we seize the opportunity to test the latest available versions of each component, and downgrade in case an issue was encountered, a way to stay on the bleeding edge of the platform.

Since a docbase consists of the Documentum binaries, configuration files, content files (collectively grouped under the “Documentum files”) and a database schema, each of these parts of a seed repository must be addressed. Once the seed docbase along with its docbroker and method server have been created and shut down, its logs and caches will be deleted, and the ${DOCUMENTUM} tree and the content files will be moved into a tar ball. When using an Oracle database, the docbase’s schema will be exported and the dump file added to the archive, When using a PostgreSQL database, the whole tree including binaries and data, will be added to the archive. The compressed archive can then be stored on a convenient location (e.g. a network drive, a git or nexus repository, etc.) ready to be used by anyone, mostly developers or administrators. After that, the seed docbase can be removed from the host and from the database (if using Oracle) as it is no longer needed. Should it ever be, it can be restored from the tar ball or instantiated out of itself since, unsurprisingly, the instantiation process is idempotent.

On the receiving end, the instantiating procedure will explode the tarball to another root directory, possibly on another machine after having created an O/S account for the content server files’ owner. Next, if an Oracle database is used, it will create a db account for the new docbase and import the dumped data into the db schema. If a PostgreSQL database is used, no data import is necessary since the whole database, binaries included, has been extracted from the tarball; only some quick renaming of the database, schema and user are performed. Next, a few adjustments will be applied in the restored configuration files to match their new location and schema, and finally the Migration Utility will be run. A few final corrections will be applied and, optionally, a check of the db schema and files under ${DOCUMENTUM} too.

This approach implies that the Documentum binaries are considered part of the repository, just like its documents, which is a common situation in dedicated VMs and containers. One pro of this approach is that it takes care of the installation of the binaries as well, they are simply extracted from the tarball, a relief knowing how long the serverSetup.sh program can take to execute. When using PostgreSQL as the RDBMS, the database becomes part of the seed package, binaries and data, and is treated as if it were a component of the repository. Maybe one day, OpenText will propose an optional installation of a PostgreSQL database aside the content server, just like it currently does with tomcat and jboss as the method servers; being open source, there are no legal deterrents in doing so.

A single parameter file, global_parameters, defines all the parameters for the creation of the seed docbase and its future instantiations. Actually, there is nothing special in a seed docbase, it is just a role; any docbase created by the create_docbase.sh script can later be used as a seed. If only the creation of a docbase is wanted, just by setting suitable parameters in that file, an automated installation of the binaries and a repository creation is possible the traditional – but slow – way. Even without the benefits of a full instantiation, this is already quite a useful by-product of the project.

The complete procedure consists of the following general steps:

1. Edit the global_parameters file and provide all the required settings for the machine that will host the repositories;

2. Enroll a machine for the seed docbase’s installation, see pre-requisites.sh;

3. Edit the global_parameters file and provide all the required settings for the repositories to be created, the seed and any instantiation of it to come;

4. Create a repository, archive its ${DOCUMENTUM} directory along with its extracted database schema or full database if using PostgreSQL, see script create_docbase.sh; this is done only once and the docbase will be the model – or seed – for all future instantiations;

5. For each docbase to instantiate, execute the script instantiate_docbase.sh with as parameters the docbase to use as the seed and the docbase to instantiate from it. The detailed parameters will be taken from global_parameters. If the docbases must be created on different hosts, enroll those hosts first using pre-requisites.sh;

Let’s now see all the above scripts. In order to save space, only code snippets are included in the current article. See dbi services github for the complete scripts.

The prerequisites

The script is accessible here: dbi services’ github.

Prior to running the scripts, the following conditions must be met:

1. An existing directory location to hold the global_parameters file and the scripts; at the start of the scripts’ execution, the current working directory will be changed there. This directory can be a mounted volume or it can be copied onto the machine where the seed docbase and/or the instantiations will be created; it will be referred to as global_parameters.${scripts_dir}. The notation global_parameters.${parameter_name} means the value of parameter_name as defined in the global_parameters file;

2. The global_parameters file edited to suit the needs (see next paragraph); that’s where all the parameters below are taken from;

3. An installation volume global_parameters.${binaries_root}. If it does not exist, it is created under /. If a mounted volume must be used, be sure to mount it first so the installation directory gets created there;

4. Download into global_parameters.${dctm_software} the Content Server from OpenText, e.g. documentum_server_22.4_linux64_oracle.tar and documentum_server_22.4_linux64_postgres.tar. The other packages get downloaded anonymously by create_docbase.sh as needed but the content server needs a logged in session at OpenText (and a contractual account).

If the PostgreSQL RDBMS is used, nothing is required at the database level since the scripts have a complete control over this locally installed software.

However, if the selected RDBMS is Oracle, the requirements are clearly more complicated. An existing database is required that is accessible through global_parameters.${db_server_host_ip_address} plus its service name (global_parameters.${db_service_name}) and listener port (global_parameters.${db_listener_port}); the InstantClient’s local tnsnames.ora will be created and filled in with that information. Also, an account is required on the database host’s, typically oracle (cf. global_parameters.${db_server_host_account}/global_parameters.${db_server_host_password}), to remotely launch the dump and load utilities, and transfer the dump file from/to the database host.

Moreover, the sys account’s credentials global_parameters.${db_sys_account}/global_parameters.${db_sys_password} is needed to manipulate the repositories’schemas while connected using global_parameters.${db_remote_connect_string}.

Note that the server host account and the sys account are unnecessary if the steps to perform under those accounts are delegated to the DBAs, which is a must in some large organizations where personnel’s’ roles are very specialized. Therefore, some coordination with them is needed, which may make the whole process not as straightforward and fast as intended. The schema’s dumps after the seed docbases are created and their subsequent loads when instantiating new docbases can also be done by the DBAs. The same applies to the schemas if they are created beforehand. Adapt the scripts as needed by the actual situation.

After the above conditions are satisfied where applicable, the script prerequisites.sh can be invoked as root to enroll the machine with the settings from global_parameters. It will perform the following actions:

1. Set the FQDN of the machine (global_parameters.${dctm_machine} and global_parameters.${dctm_domain}, and append it in /etc/hosts;

2. Set IP aliases for the content server machine (cs) and the database (global_parameters.${db_server_host_alias} for global_parameters.${db_server_host_ip_address});

2. Install sshpass; this program allows a non-interactive ssh connection without being prompted by taking the password from an environment variable or the command-line and passing it to ssh. It is used in order to fully automate the scripts’ execution only when the RDBMS is a supposedly remote Oracle database. If PostgreSQL is used, sshpass is not necessary since those databases are always local;

3. Install a few other utilities such as unzip, gawk and curl used by the scripts. Additionally, tcl and expect are also installed although they may not be necessary since the Documentum programs serverSetup.bin and dm_launch_server_config_program.sh are invoked non interactively, i.e. during silent installations;

4. Create an installation owner account on current machine (defined as global_parameters.${dctm_machine}), usually dmadmin, defined as global_parameters.${dctm_owner}. In order to simplify the scripting and to execute the Documentum root tasks, dmadmin is made sudoer with no password required; This is not an issue as most of the time dmadmin runs on a fully dedicated VM or container and cannot hurt anybody except itself, which is already possible with no additional privilege. Anyway, the privilege can be revoked later once the procedure is completed;

5. Create the global_parameters.${binaries_root} sub-directory where the repositories will be created on the enrolled machine. Make sure the parent volume is large enough to contain the docbases to create locally;

Again, points 1 to 4 depends on the organization; sometimes, the machines (e.g. VMs) are delivered already configured; sometimes the root account is a no go or under some strict conditions. Therefore, those steps are only applicable in private machines under complete control of an individual. Adapt the procedure as needed.

There are also a few commented out kernel settings related to limits as recommended by Documentum in the OpenText Documentum System Upgrade and Migration Guide for CS22.2. They may apply in VMs or non-virtualized environments. In our Promox linux container environment, some of them may only be set at the host’s level. Activate them as needed, although in a light developer’s environment those limits will probably never be hit.

See Part II here

L’article A quick repository creation utility (part I) est apparu en premier sur dbi Blog.

This is part III of the article, see Part I here and Part II here.

Creating the seed docbase

See dbi services for its source. Hereafter, only some salient steps are presented, refer to the script for the details.

As its name implies, the script create_docbase.sh is used to create a docbase optionally to be used later as a seed. Its usage is:

$ ./create_docbase.sh [stem]

where the optional stem in global_parameters defaults to DCTM0 and points to the docbase’s required settings. It is executed as the installation owner, usually dmadmin.

After the global_parameters file is sourced with the given stem, the installation directory is created and all the needed software packages (the JDK, the Oracle Instant Client and SQL*Plus if the Oracle RDBMS is chosen, the PostgreSQL source tarball global_parameters.${postgresql_package} otherwise) are downloaded into the directory global_parameters.${dctm_software}, only if they are not there yet. After their downloading, the packages are expanded and the software installed and configured as advised in the OpenText documentation. For example, the anonymous algorithm for secure communication with the content server is reinstated in the JDK from its default disabled state.

If the selected RDBMS is PostgreSQL, its installation is completely customized. In effect, the platform’s standard PostgreSQL package is normally installed system-wide by root with pieces scattered in several locations, e.g. /var/lib and /usr/lib. As we want to embed this RDBMS with the repository and under the dmadmin ownership, this is not convenient; we want all the pieces in one directory tree root. In order to make this possible, we need to compile the source code specifying the installation directory ${ACTIVE_DOCBASE_ROOT}/postgresql. The binary and data files will all be installed under this directory, which will later be included in the compressed tar ball when the script completes executing. As the compilation takes several minutes and even though it only needs to be done once, a pre-compiled tarball global_parameters.${postgresql_custom_package} is available so this step can be skipped, unless global_parameters.${postgresql_compile} is set to “yes“.

After the PostgreSQL binaries have been extracted or compiled, the ODBC connectivity is configured. Surprisingly, Documentum’s configuration program imperatively needs a connect string named postgres and defined in /etc/odbc.ini as if it were hard-coded. Apparently, it reads this file explicitly instead of relying on the ODBC API which looks for any specified connect string first in ~/.odbc.ini and lastly in /etc/odbc.ini. Also, it requires the running PostgresSQL server to be listening on the default port 5432 instead of any port defined in the ini file. Strangely enough, and this is visible when using the configuration program interactively, ~/.odbc.ini is correctly used and validated but only until the program effectively starts creating the docbase, where it switches to /etc/odbc.ini and the weird behavior. The documentation mentions changes to be done in ODBC.INI but does not say where it is (it’s in /etc but ~/.odbc.ini should be used preferably). Also, it incorrectly says that the connect string should be [MyPostgres] whereas it should be anything, although only [postgres] works at this stage. All this suggests that the ODBC part is not well mastered by developpers of the installation program, or that it was ported without much attention from some other O/S. Fortunately, these idiosyncrasies are abandoned after the docbase is created and all the parameters set in the installer owner’s ~/.odbc.ini are honored, e.g. custom connect string and database server port.

Here is how /etc/odbc.ini must look for the docbase creation to start:

[postgres]
Description = PostgreSQL connection to postgres
Driver = PostgreSQL Unicode
Database = postgres
Servername = ${db_server_host_alias}
UserName = ${dctm_owner}
# hard-coded in the server configuration tool;
Port = 5432

and here is the final ~/.odbc.ini file defined after the docbase creation, including some of the recommended database settings:

[${db_connect_string}]
Description = PostgreSQL connection to ${ACTIVE_DOCBASE}
Driver = PostgreSQL Unicode
Database = ${ACTIVE_DOCBASE}
Servername = ${db_server_host_alias}
UserName = ${dctm_owner}
Port = ${ACTIVE_DB_SERVER_PORT}
Protocol = $(echo ${postgresql_custom_package} | sed -E 's/.+-([0-9]+).*$/\1/')
ReadOnly = No
RowVersioning = No
ShowSystemTables = No
ShowOidColumn = No
FakeOidIndex = No
UpdateableCursors = Yes

The shell variables get expanded when the file is saved.

Another requirement with the postgreSQL RDBMS is that the installer demands a directory named db_${ACTIVE_DOCBASE}_dat.dat for the datafile to be created beforehand; if not found, the installer fails.

When using the Oracle RDBMS, a global_parameters.${dctm_root}/docbase/oracle/instantclient_21_7/network/admin/tnsnames.ora file gets created with the following content:

${db_connect_string} =
   (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)
                             (HOST = ${db_server_host_alias})(PORT = ${db_listener_port}
                             )
                  )
                  (CONNECT_DATA =
                     (SERVER = DEDICATED)
                     (SERVICE_NAME = ${db_service_name})
                  )
   )

Next, the data and index tablespaces are created and finally the schema account with the required grants. Here is a summary of the database requirements for Oracle:

select TABLESPACE_NAME, ALLOCATION_TYPE from dba_tablespaces order by 1;
TABLESPACE_NAME          ALLOCATION_TYPE
------------------------------ ---------------------------
DCTM1_DATA            SYSTEM
DCTM1_INDEX           SYSTEM
...
SEED_DATA             SYSTEM
SEED_INDEX            SYSTEM

SQL> select privilege from dba_sys_privs where grantee = upper('dctm1');
GRANTEE                        PRIVILEGE                      ADMIN_OPT COMMON
------------------------------ ------------------------------ --------- ---------
DCTM1                          CREATE VIEW                    NO        NO
DCTM1                          CREATE ANY VIEW                NO        NO
DCTM1                          CREATE SEQUENCE                NO        NO
DCTM1                          CREATE PROCEDURE               NO        NO
DCTM1                          CREATE TABLE                   NO        NO

SQL> select * from dba_role_privs where grantee = upper('dctm1');
GRANTEE                        GRANTED_ROLE                   ADMIN_OPT DEFAULT_R COMMON
------------------------------ ------------------------------ --------- --------- ---------
DCTM1                          CONNECT                        NO        YES       NO
DCTM1                          RESOURCE                       NO        YES       NO
DCTM1                          SELECT_CATALOG_ROLE            NO        YES       NO

SQL> select TABLESPACE_NAME, STATUS, ALLOCATION_TYPE, SEGMENT_SPACE_MANAGEMENT from user_tablespaces;
TABLESPACE_NAME                STATUS                      ALLOCATION_TYPE             SEGMENT_SPACE_MANA
------------------------------ --------------------------- --------------------------- ------------------
DCTM1_DATA                     ONLINE                      SYSTEM                      AUTO
DCTM1_INDEX                    ONLINE                      SYSTEM                      AUTO

After the database steps have been completed, the documentum’s installer program serverSetup.bin is launched with the following generated on-the-fly response file:

INSTALLER_UI=silent
KEEP_TEMP_FILE=true
common.installOwner.password=dmadmin
SERVER.SECURE.ROOT_PASSWORD=root

# documentum binaries;
SERVER.DOCUMENTUM=${ACTIVE_ROOT}/documentum
APPSERVER.SERVER_HTTP_PORT=${ACTIVE_SERVER_HTTP_PORT}
APPSERVER.SECURE.PASSWORD=tomcat

# java;
PATH_TO_JAVA=$(ls -d ${ACTIVE_ROOT}/java/amazon-corretto-11*-linux-x64)

This step takes several minutes to complete but it does not matter; as said previously, the real optimization occurs in the instantiation step, not in the installation of the binaries or the creation of the model docbase; those steps are done only once (unless it is done outside the context of this project).

Method servers are dedicated to each docbase and receive a distinct base port number defined as global_properties.${stem_SERVER_HTTP_PORT}. Later during the instantiation, their memory sizing will be configured as set in global_properties.${stem_SERVER_HTTP_MEMORY}.

It has been noticed that the method server creates several cache directories (e.g. felix-cache and activemq-data) in the directory it is started from; to move them out of the way and prevent cluttering, the command cd ${DM_JMS_HOME}/temp is later inserted in startMethodServer.sh before tomcat’s startup.sh script is invoked.

After the binaries have been installed, a connectivity check using documentum’s dmdbtest is performed to confirm that the database is reachable via SQL*Net for Oracle or via ODBC for PostgreSQL. This test differs from the ones previously done directly at the database level in that it is performed by documentum. Its output resembles the following one:

if [[ "${ACTIVE_RDBMS}" == "oracle" ]]; then
   ${DOCUMENTUM}/product/*/bin/dmdbtest -Dxx -S${db_connect_string} -U${ACTIVE_DOCBASE} -P${ACTIVE_DOCBASE}
else
   ${DOCUMENTUM}/product/*/bin/dmdbtest -Dpostgres -Spostgres -U${dctm_owner} -Pxxx
fi
Database successfully opened.
Test table successfully created.
Test view successfully created.
Test index successfully created.
Insert into table successfully done.
Index successfully dropped.
View successfully dropped.
Database case sensitivity test successfully past.
Table successfully dropped.

Note that the above test needs an extra step when using Oracle RDBMS because although the sqlplus executable is installed in Instant Client’s ${ORACLE_HOME} and ${ORACLE_HOME} is in the ${PATH}, the installer expects it in the non-existing ${ORACLE_HOME}/bin. So, that directory must be created beforehand and sqlplus symlinked there. Evidently, sqlplus’ expected location is hard-coded and the ${ORACLE_HOME} environment variable is either not passed to the java programs invoked by the installer script, or ignored when looking for it. This quirk can be half-forgiven as it is a documented requirement.

Since several docbases can potentially be instantiated and run on the same machine, an environment file named docbase_name.env for each of them is prepared to allow switching easily between them. This file notably defines the ${DOCUMENTUM} and ${PATH} variables, invokes documentum’s own dm_set_server_env.sh, and should be sourced prior to working with a given docbase. To simplify this, the bash function swr() (which stands for SWitch Repository) has been defined in the installation owner’s ~/.profile. Other useful management functions such as sur()and sdr(), have also been defined in docbase_name.env, see down below.

Next, another response file is prepared to install the docbroker. docbrokers too are dedicated to each docbase and defined in the global_properties file with parameters stem_DOCBROKER_NAME and stem_DOCBROKER_PORT. Make sure there is no port conflict when creating/instantiating new docbases. Here is an example of this response file:

KEEP_TEMP_FILE=true
PATH_TO_JAVA=$(ls -d ${ACTIVE_ROOT}/java/amazon-corretto-11*-linux-x64)
INSTALLER_UI=silent
common.aek.algorithm=AES_128_CBC

# docbroker;
SERVER.CONFIGURATOR.BROKER=TRUE
SERVER.DOCBROKER_ACTION=CREATE
SERVER.DOCBROKER_PORT=${ACTIVE_DOCBROKER_PORT}
SERVER.DOCBROKER_NAME=${ACTIVE_DOCBROKER_NAME}
SERVER.PROJECTED_DOCBROKER_HOST=$(hostname)
SERVER.PROJECTED_DOCBROKER_PORT=${ACTIVE_DOCBROKER_PORT}
SERVER.DOCBROKER_CONNECT_MODE=native
SERVER.USE_CERTIFICATES=false

Make sure the docbroker name does not contains any non alphanumeric characters as the installer is quite restrictive in this regard. For example, the name docbroker_01 is rejected because it contains an underscore.

Documentum’s dm_launch_server_config_program.sh is then invoked with that response file and the docbroker gets created and started along with the method server.

Next, another response file is prepared for the repository. That one is quite large so, please, refer to the code on git here dbi services.

After the configuration program is invoked with the above response file and completes, the documentum processes are shut down and, if using Oracle, the resulting seed’s database schema will be extracted using the traditional exp utility. Although the exp and imp utilities have been superseded by the data pump for a while now, they do their job well enough for repositories’ schemas. Of course, the more modern data pump can also be used instead under certain conditions, but it may need some preliminary work on the database’s side to accommodate it. The same applies if using the Instant Client’s exp and imp utilities when these tools’ version differs from the database’s. For example, one of our test database is a v12.1.0.1.0 one but there are no downloadable exp and imp tools with that version for the Instant Client as they started being included only in v12.2.0.x (cf. https://download.oracle.com/otn/linux/instantclient/122010/instantclient-basic-linux.x64-12.2.0.1.0.zip and https://download.oracle.com/otn/linux/instantclient/122010/instantclient-tools-linux.x64-12.2.0.1.0.zip). And since those versions differ from the database’s, a PL/SQL package must be executed so they can be used. All this is quite complicated and to simplify the data extraction and import tasks, we will use the exp and imp utilities bundled with the RDBMS; those will always work as-is so no additional download nor configuration are necessary. However, those tools need to be invoked from within a remote session, hence the settings global_parameters.${db_server_host_*}, unless those steps are delegated to the DBAs. As already discussed, Oracle RDBMS is a proprietary, closed-source and quite complex software. Starting with the licensing, it requires some careful planning. Sometimes, databases are even shared between several applications. Therefore, it is mostly managed by specialized people in an organization and installed in dedicated machines. For those reasons, the Oracle part takes up a large chunk of the procedure dedicated to the RDBMS and, for faster and more agile configuration, the leaner but sufficient PostgreSQL RDBMS is preferred. As precedently written, the postgreSQL source is compiled so it can be embedded with the docbase that uses it and the whole binaries and database get copied into the tar ball, which is a considerable simplification at a measly cost of 276 Mb.

After the compressed tar ball is produced, it is renamed to ${ACTIVE_DOCBASE}_${ACTIVE_RDBMS}.tgz and moved to the ${scripts_dir} if one with the same name does not exist there yet (otherwise, the move is not performed), ready to be instantiated as needed.

A useful by-product of this project, although it won’t benefit from the optimized provision time, the generic script create_docbase.sh creates any docbase whose parameters are set in global_parameters. Once created, this docbase can be used stand-alone, or taken a snapshot of and used as a seed for future instantiations, or both. It can even be uncompressed in several places on the same machine or on different machines as distinct clones, provided resource conflicts are resolved and the usual well-known adjustments post-cloning are applied. The script can create several docbases to be used as models for different purposes after further customizations; edit the script as required.

It is possible to specify as many docbases to be created (or instantiated, see later) in global_parameters.${dctm_machine} as needed by specifying their stems on the command-line. So, if several docbases need be created, even concurrently, define them all in global_parameters and invoke the creation script as follows:

# Example:
$ for i in {1..10}; do
   ./create_docbase.sh "DCTM${i}" &
done
where DCTM1_* to DCTM10_* are the respective docbases's stem "pointing" to the settings defined in the global_properties file.

# Another example with unrelated stems:
./create_docbase.sh bookstore &
./create_docbase.sh product_catalog &
./create_docbase.sh research_papers &

Concurrent docbase creation is another useful by-product of the project and is possible because each created docbase is normally stand-alone (i.e. no dependency on other services such as the docbroker or the method server) and has a distinct ${DOCUMENTUM} directory under global_parameters.${dctm_root}/docbase_name.

Granted, when using PostgreSQL , since the docbase creation demands a server listening on port 5432, there may be a short period of time when a server gets commonly used by several instances of dm_launch_server_config_program.sh. Also, multiple database server instances get started on port 5432, with only the first one succeeding. But this should not be an issue since the databases and their locations are distinct. This does not happen at instantiation time as the PostgreSQL server is always started with the custom port.

The additional cost is 2 Gb of disk space since the documentum binaries are not shared between the repositories (i.e. each repository has its own ${DOCUMENTUM} and its own copy of the binaries), but disk space is so cheap these days that this is not prohibitive. When using Oracle, the above concurrency behavior does not apply.

As mentioned before, create_docbase.sh sets up a sourceable script to define a working docbase’s environment and several useful functions, global_properties.${dctm_root}/docbase/docbase.env. In particular, it defines:

${ACTIVE_DOCBASE}, ${ACTIVE_ROOT}, ${JAVA_HOME}, ${DOCUMENTUM} and ${PATH}, and sources documentum’s own dm_set_server_env.sh. ${ACTIVE_DOCBASE} is the name of the current repository and ${ACTIVE_ROOT} the root directory that contains that docbase’s ${JAVA_HOME}, ${DOCUMENTUM} and the postgres root directory if that RDBMS is used. ${ACTIVE_ROOT} is also equal to global_properties.${dctm_root}/docbase.

In addition to those environment variables, several management functions are defined too; see further down below.

For more comfort, the environment file also changes the shell’s prompt to something nicer, e.g.:

dmadmin@cs2:[/u01/dctm/repo03][repo03] $

The prompt shows the currently logged user and the machine it is logged on, the current working directory, and the current active docbase, which is useful to prevent manipulating the wrong docbase when several of them coexist in the machine.

See Part IV here

L’article A quick repository creation utility (part III) est apparu en premier sur dbi Blog.

This is part IV of the article, see here for Part I here, Part II here, and Part III here.

Instantiating a new docbase

See dbi services.

Roughly speaking, the instantiation process uncompresses a seed, or model, docbase from its compressed tar ball and runs the Migration Utility against it to adapt its name and id. Those actions are performed by the script instantiate_docbase.sh. Its usage is:

$ instantiate_docbase.sh seed_repo new_repo

where seed_repo and new_repo are stems pointing to the settings of the respective docbase defined in the global_parameters file. The easy-to-remember syntax is the same as that of the cp command, for a similar concept.

Instantiation is always performed locally so if it needs to be done on a machine different from the one where the model docbase was created, the new machine needs first to be enrolled (see script pre-requisites.sh) and the script invoked from there.

The procedure looks in the ${DOCUMENTUM} tree and changes all the references to the seed docbase so they match the target docbase, including the settings such as port numbers, docbroker names, host names, http server memory sizing, sym links, etc, and then tackles the metadata’s big part. For this to be successful, make sure the seed docbase’s definition in the global_parameter file has not changed and still matches its installation in the compressed tar ball.

For an Oracle RDBMS, a new schema is created with the required tablespaces and grants, the dump file that was produced by the exp utility is copied onto the database’s host, slightly edited (references to the seed’s schema are changed to the new instance’s schema) and imported into the new schema. Next, as the creation of some views’ fails with “IMP-00008: unrecognized statement in the export file:” errors, the view creation statements (almost 1000 of them) are extracted from the dump file (although this is a binary file, the SQL statements it contains can be extracted as text and edited), corrected and re-executed separately, as shown here:

# extract the view creation statements from export file and create the views;
strings ${seed_dmp_file} | sed -r 's/^[ \t\$]*(\(*)(SELECT)(.+)/\1\2\3/I' | gawk '{if (sub(/^CREATE VIEW /, "CREATE or REPLACE VIEW ", \$0)) {print; getline; pos = 2000; if (length(\$0) > pos) {while (substr(\$0, --pos, 1) != ","); print substr(\$0, 1, pos); print substr(\$0, pos + 1)}else print; print "/"}}END{print "quit"}' >> create_views.sql

# count the views to be created;
grep "CREATE or REPLACE VIEW" create_views.sql | wc -l
# 977 in CS 22.2;
# 983 in CS 22.4;

# create the views;
sqlplus ${ACTIVE_DATABASE_OWNER}/${ACTIVE_DATABASE_PASSWORD}@${db_remote_connect_string} @create_views.sql

The view text needs to be reformatted to work around the weird legacy format of the dump file.

For good measure, all the new schema’s compilable objects are recompiled and the schema’s statistics computed. A verification of the schema’s object validity is performed:

set echo on
set pagesize 10000
set linesize 100
col object_name format a30
col object_type format a30
select object_name, object_type from user_objects where status = 'INVALID';
select object_name, object_type, status from user_objects;
quit
eoq
# output:
no rows selected
OBJECT_NAME                    OBJECT_TYPE                    STATUS
------------------------------ ------------------------------ -------
D_1F00271080000195             INDEX                          VALID
D_1F00271080000241             INDEX                          VALID
DMI_CHANGE_RECORD_S            TABLE                          VALID
D_1F002710800001A8             INDEX                          VALID
DMI_DD_ATTR_INFO_R             TABLE                          VALID
...
DM_INDEXES                     VIEW                           VALID
DM_RESYNC_DD_ATTR_INFO         VIEW                           VALID
DM_RESYNC_DD_TYPE_INFO         VIEW                           VALID
DMI_DD_ATTR_INFO_DDEN          VIEW                           VALID

1931 rows selected.

For a PostgreSQL RDBMS, things are much less complicated. The server port is adapted and the server started. Next, the names of the schema, database and user are adapted and the ownership of each of the schema’s objects is displayed as a verification. Example of output:

  datname  
-----------
 postgres
 template1
 template0
 repo02
(4 rows)

                                   List of roles
 Role name |                         Attributes                         | Member of 
-----------+------------------------------------------------------------+-----------
 dmadmin   | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
 repo02    | No inheritance                                             | {}

You are now connected to database "repo02" as user "dmadmin".
ALTER SCHEMA
You are now connected to database "postgres" as user "dmadmin".
ALTER DATABASE
NOTICE:  MD5 password cleared because of role rename
ALTER ROLE
ALTER ROLE
ALTER ROLE

You are now connected to database "repo03" as user "repo03".
                                   List of roles
 Role name |                         Attributes                         | Member of 
-----------+------------------------------------------------------------+-----------
 dmadmin   | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
 repo03    | No inheritance                                             | {}

 current_user 
--------------
 repo03
(1 row)

                                                version                                                
-------------------------------------------------------------------------------------------------------
 PostgreSQL 15.1 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, 64-bit
(1 row)

  datname  
-----------
 postgres
 template1
 template0
 repo03
(4 rows)

 schemaname |              objectname               | len | objectowner | objecttype 
------------+---------------------------------------+-----+-------------+------------
 repo03     | adm_turbo_size                        |  14 | repo03      | TABLE
 repo03     | creation_date_message_id_ui           |  27 | repo03      | INDEX
 repo03     | d_1f00271280000001                    |  18 | repo03      | INDEX
 repo03     | d_1f00271280000002                    |  18 | repo03      | INDEX
...
 repo03     | dm_federation_log                     |  17 | repo03      | TABLE
 repo03     | dm_federation_r                       |  15 | repo03      | TABLE
 repo03     | dm_federation_rp                      |  16 | repo03      | VIEW
 repo03     | dm_federation_rv                      |  16 | repo03      | VIEW
 repo03     | dm_federation_s                       |  15 | repo03      | TABLE
...
 repo03     | status_no_idx                         |  13 | repo03      | INDEX
(1932 rows)

The above output shows the repo03 account and its roles, a database named repo03 and docbase’s objects owned by user repo03. Also, repo03 was instantiated from repo02 as the seed.

As for Oracle, the connectivity is also adapted here but instead of the SQL*Net’s tnsnames.ora, the installation owner’s ~/.odbc.ini file is adapted to match the target docbase’s settings and the reference to it in the server.ini file is adapted too. That concludes the RDBMS part.

The new instance’s password is also saved using the documentum’s dm_encrypt_password utility, and tested using dmdbtest. At this point, the Migration Utility step can be started.

The Migration Utility

The Migration Utility is located in ${DOCUMENTUM}/product/22.2/install/external_apps/MigrationUtil. This tool takes its instructions from the xml configuration file ${DM_HOME}/install/external_apps/MigrationUtil/config.xml. The following relevant attributes’ values are edited to match the target docbase’s settings:

• the type of RDBMS, the listener’s (for Oracle) or the server’s port (for PostgreSQL)
• the database host name
• the install owner’s password
• the docbase’s name and database’s password
• the old and new docbase ids
• the new docbase server name (this is not a machine name but the server object’s one in dm_server_config)
• the docbase’s machine name
• the installation owner’s name

Here is an example of the config.xml file generated while instantiating repo02 into bookstore:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
   <comment>Database connection details</comment>
   <entry key="dbms">postgres</entry> <!-- This would be either sqlserver, oracle, db2 or postgres -->
   <entry key="tgt_database_server">localhost</entry> <!-- Database Server host or IP -->
   <entry key="port_number">6000</entry> <!-- Database port number -->
   <entry key="InstallOwnerPassword">dmadmin</entry>
   <entry key="isRCS">no</entry> <!-- set it to yes, when running the utility on secondary CS -->
   <entry key="TomcatPath"></entry> <!-- Optional. set it to appropriate tomcat path if DM_JMS_HOME environment variable is not set -->
   <!-- <comment>List of docbases in the machine</comment> -->
   <entry key="DocbaseName.1">repo02</entry>
   <entry key="DocbaseName.2">Docbase2</entry>
   <entry key="DocbaseName.3"></entry>
   <!-- <comment>docbase owner password</comment> -->
   <entry key="DocbasePassword.1">bookstore</entry>
   <entry key="DocbasePassword.2">Docbase owner password2</entry>
   <entry key="DocbasePassword.3"></entry>
   <entry key="ChangeDocbaseID">yes</entry> <!-- To change docbase ID or not -->
   <entry key="Docbase_name">repo02</entry> <!-- has to match with DocbaseName.1 -->
   <entry key="NewDocbaseID">20000</entry> <!-- New docbase ID -->
   <entry key="ChangeServerName">yes</entry>
   <entry key="NewServerName.1">bookstore</entry>
   <entry key="NewServerName.2"> </entry>
   <entry key="ChangeDocbaseName">yes</entry>
   <entry key="NewDocbaseName.1">bookstore</entry>
   <entry key="NewDocbaseName.2"> </entry>
   <entry key="ChangeHostName">no</entry>
   <entry key="HostName">Old Host name </entry>
   <entry key="NewHostName">New Host name </entry>
   <entry key="ChangeInstallOwner">no</entry>
   <entry key="InstallOwner">Old Install Owner </entry>
   <entry key="NewInstallOwner"> New Install Owner </entry>
   <entry key="NewInstallOwnerPassword">New Install Owner password </entry>
   <entry key="DockerSeamlessUpgrade">no</entry>
   <entry key="PrimaryHost">Primary Host name </entry>
   <entry key="SecondaryHost">Secondary Host name </entry>
   <entry key="PrimaryServerConfig">Primary Server name </entry>
   <entry key="SecondaryServerConfig">Secondary Server name </entry>
   <entry key="DocbaseService">Docbase Service name </entry>
</properties>

Since the Migration Utility is a java application, it uses JDBC to connect to the database; thus, prior to launching it, a JDBC connection to the database is attempted in order to check the connectivity parameters. To this effect, a simple but generic utility has been specially written for this project, jdbc_tester_generic.java (see article A Generic JDBC Tester), located in ${scripts_dir}. Its usage is:

$ java -cp jdbc_driver_jar:path_to_the_tester jdbc_tester_generic jdbc_driver user_name password jdbc_url sql_statement

# Example for a PostgreSQL database:
$ export CLASSPATH=${dctm_software}/${postgresql_jdbc_package}:$CLASSPATH
$ java -cp $CLASSPATH:${scripts_dir} jdbc_tester_generic org.postgresql.Driver dmadmin xxx jdbc:postgresql://localhost:6000/bookstore "select 'now in postgresql db: ' || to_char(current_timestamp(0), 'DD/MM/YYYY HH24:MI:SS') as now;"
The command-line parameters are:
arg 0 = org.postgresql.Driver
arg 1 = dmadmin
arg 2 = xxx
arg 3 = jdbc:postgresql://localhost:6000/bookstore
arg 4 = select 'now in postgresql db: ' || to_char(current_timestamp(0), 'DD/MM/YYYY HH24:MI:SS') as now;
Loading driver [org.postgresql.Driver]:
JDBC Driver org.postgresql.Driver successfully loaded
Connecting to url [jdbc:postgresql://localhost:6000/bookstore] as user [dmadmin]:
Successful connection to database
Executing SQL query [select 'now in postgresql db: ' || to_char(current_timestamp(0), 'DD/MM/YYYY HH24:MI:SS') as now;]:
Retrieving the result set:
now in postgresql db: 08/01/2023 19:00:53
JDBC connection successfully tested, quitting ...

# Example for an Oracle database;
$ ln -s ${ORACLE_HOME}/ojdbc8.jar ${ORACLE_HOME}/ojdbc.jar
$ export CLASSPATH=${ORACLE_HOME}/ojdbc.jar:$CLASSPATH
# using Oracle thin:
$ java -cp $CLASSPATH:${scripts_dir} jdbc_tester_generic oracle.jdbc.driver.OracleDriver seed seed jdbc:oracle:thin:@//db:1521/pdb1 "select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual"
The command-line parameters are:
arg 0 = oracle.jdbc.driver.OracleDriver
arg 1 = seed
arg 2 = seed
arg 3 = jdbc:oracle:thin:@//db:1521/pdb1
arg 4 = select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual
Loading driver [oracle.jdbc.driver.OracleDriver]:
JDBC Driver oracle.jdbc.driver.OracleDriver successfully loaded
Connecting to url [jdbc:oracle:thin:@//db:1521/pdb1] as user [seed]:
Successful connection to database
Executing SQL query [select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual]:
Retrieving the result set:
now in Oracle db: 08/01/2023 18:51:51
JDBC connection successfully tested, quitting …

# using Oracle thick, OCI-based drivers and the SQL*Net alias:
$ java -cp $CLASSPATH:${scripts_dir} jdbc_tester_generic oracle.jdbc.driver.OracleDriver seed seed jdbc:oracle:oci:@pdb1 "select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual"

The command-line parameters are:
arg 0 = oracle.jdbc.driver.OracleDriver
arg 1 = seed
arg 2 = seed
arg 3 = jdbc:oracle:oci:@pdb1
arg 4 = select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual
Loading driver [oracle.jdbc.driver.OracleDriver]:
JDBC Driver oracle.jdbc.driver.OracleDriver successfully loaded
Connecting to url [jdbc:oracle:oci:@pdb1] as user [seed]:
Successful connection to database
Executing SQL query [select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual]:
Retrieving the result set:
now in Oracle db: 08/01/2023 18:52:37
JDBC connection successfully tested, quitting ...

This utility works for any RDBMS as long as its JDBC drivers are provided. Note how simpler the syntax for Oracle JDBC Thick is compared to Oracle Thin’s or PosgreSQL’s, as its required parameters are actually read from the tnsnames.ora using the service name as an index, which allows to reuse an existing configuration instead of extracting each required piece of information from it. Unfortunately, but understandably since it is Oracle-specific, it is not used by the Migration Utility in favor of the more universal, pure java one that works with any RDBMS.

Finally, the Migration Utility is launched and the requested changes are performed. Each one has its own log file so it is easy to check them for any error. As an example, here is the log for the change docbase id step:

Reading config.xml from path: config.xmlReading server.ini parameters
Retrieving server.ini path for docbase: repo02
Found path: /u01/dctm/bookstore/documentum/dba/config/repo02/server.ini
Set the following properties:
Docbase Name:repo02
Docbase ID:10002
New Docbase ID:20000
DBMS: postgres
DatabaseName: bookstore
SchemaOwner: bookstore
ServerName: localhost
PortNumber: 6000
DatabaseOwner: bookstore
-------- PostgreSQL JDBC Connection Testing ------
jdbc:postgresql://localhost:6000/bookstore
Connected to database
Utility is going to modify Objects with new docbase ID
Fri Jan 06 13:52:47 UTC 2023
-----------------------------------------------------------
Processing tables containing r_object_id column
-----------------------------------------------------------
-------- PostgreSQL JDBC Connection Testing ------
jdbc:postgresql://localhost:6000/bookstore
Connected to database
-----------------------------------------------------------
Update the object IDs of the Table: dm_replication_events with new docbase ID:4e20
-----------------------------------------------------------
Processing objectID columns
-----------------------------------------------------------
Getting all ID columns from database
-----------------------------------------------------------
Processing ID columns in each documentum table
Column Name: job_id
Update the ObjectId columns of the Table: with new docbase ID
-----------------------------------------------------------
Update the object IDs of the Table: dm_state_extension_s with new docbase ID:4e20
-----------------------------------------------------------
Processing objectID columns
-----------------------------------------------------------
Getting all ID columns from database
-----------------------------------------------------------
Processing ID columns in each documentum table
Column Name: r_object_id
Update the ObjectId columns of the Table: with new docbase ID
...

where ID 4e20 is the hexadecimal representation of the bookstore’s docbase id 20000. We can deduce that all the columns containing ids of all the tables of the bookstore schema are scanned for the old value as a substring and replaced with the new value as a substring. It is understandable that these changes can put quite a heavy load on the database and are quite touchy since any failure can corrupt its integrity. Hopefully, they are done in an efficient manner. However, in our case with an out of the box seed docbase, there are very few objects to change so this is not a concern and the utility completes very quickly.

Although the resulting docbase is perfectly usable, the migration utility leaves lots of things unchanged, e.g. the dm_location’s root directories on disk (they still contain the seed docbase’s hexadecimal id), the name of the index tablespace (if Oracle is used), the ACLs’ owner (i.e. the ACLs’ domain is still the seed’s database owner; this does not jeopardize the database but is not justified in an empty docbase; actually, by keeping them unchanged, potential applications that refer to acl_domain/object_name pairs are protected), etc. Thus, the documentum services are started and a final corrective pass is executed to normalize all those remaining traces of the prior incarnation.

A few jobs, such as the dm_ContentWarning, dm_StateOfDocbase and dm_ConsistencyChecker, are forced to be executed as soon as possible so some more post-instantiation checks can be done. The db-crawler script presented in the article db-crawler, a database search utility for Documentum is also launched in the background at the end against the Oracle schema to check if there are still remains of the former seed docbase. Likewise, for PostgreSQL schemas, the pg_dump utility is launched and the resulting text file is grepped for any occurrence of the former seed docbase which is basically what the db-crawler does, except much quicker.

Like with create_docbase.sh, and for the same reason, multiple concurrent instantiations may be launched simultaneously, as follows:

# Example:
for i in {1..10}; do
   time ./instantiate_docbase.sh SEED "DCTM${i}" &
done
where SEED and DCTM1_* to DCTM10_* are the respective docbases's stem "pointing" to the settings defined in the global_properties file.

# Another example:
./instantiate_docbase.sh DCTM2 bookstore &
./instantiate_docbase.sh SEED sales &
./instantiate_docbase.sh PUBLICATIONS research_papers &

See Part V here

L’article A quick repository creation utility (part IV) est apparu en premier sur dbi Blog.

This is part V of the article, see Part I here, Part II here, Part III here and Part IV here.

Timings

As said before, absolute timings vary a lot because they depends on many factors, starting with the hardware in use. Thus, the given values are just examples of what can be achieved in a simple setup such as a laptop, and they are nonetheless quite impressive. The relevant information here is to compare the creation times with the instantiation times when 1, 2, … 5 and 10 docbases are created at the same time.

It is well understood that Oracle’s slower timings are not caused by Oracle itself but by the more complex procedure to duplicate the docbase’s schema, vs. nothing to duplicate in PostgreSQL since the database gets included with the content server and customized in place. Maybe, similar results as PostgreSQL could be obtained with a local Oracle Express Edition RDBMS, to be tried sometime if the stingy 2 Gb RAM and 12 Gb data limits are acceptable (note that Oracle XE is likely not certified for Documentum, likely because of its limits), although there is no way in Oracle to rename a schema barring exporting/importing it, which we already do here except mostly remotely.

The fastest test bed we had access to was composed of:

• Hardware: DELL XPS15 2019, i9 9980HK 8 cores, 2TB SSD, 64GB RAM
• Virtualization: Proxmox Virtual Environment 7.2-3

The following 2 linux container hosts were created:

• oracle-8, 4 cores, 10 GiB RAM, Oracle Linux Server 8.6 and Oracle RDBMS v21.3.0.0.0
• cs2, 8 cores, 20 GiB RAM, Ubuntu 20.04.5 LTS for the Content Server v22.4

In both machines, the amount of memory was large enough to prevent the 1 GiB large swap to be used with the ensuing dramatic slowness.

The table below gives the average and average maximum times to first create a seed repository and then instantiate 1 to 5 + 10 concurrent times per repository. Averages were based on 3 to 5 executions, depending on the variability of the timings (the more variable timings, the more executions were performed).

As an example, the values in bold means we instantiated 10 docbases with Oracle RDBMS as the database and it took on average 14m43s per docbase, whereas the longest time was 15m7s in average. We can therefore tell that in average it took 15m7s to create that set of 10 docbases on the same machine. Had we created them the normal and sequential way, it would have taken on average 10 * 21m20s, i.e. almost 3.5 hours or 14 times more. That’s a lot of long coffee breaks.

When creating a single docbase by instantiating it, it took in average 3m7s. The concurrency placed a toll on the performances, but of course this could be alleviated somewhat by adding more resources to the containers.

With PostgreSQL RDBMS as the database, the gain is even better: it took as little as 57s to instantiate one single docbase on average, and 2m49s per docbase to create 10 docbases on the same machine, and overall 3m11s until the last one was ready. Had we created them the normal and sequential way, it would have taken 10 * 18m22s, i.e. almost 3 hours or more than 60 times more ! The instantiation procedure is between 3 and 5 times faster with PostgreSQL than with Oracle but both are so fast that it does not make any practical difference, at least for a few docbases.

As expected, measured times are quite variable so only the ratios between instantiation and creation times are really comparative. Still, at the very least, the absolute timings prove that is is possible to have excellent performance on some out of the box, run-of-the-mill hardware, such as developer’s laptops.

Notes

¹The command was:

$ time ./create_docbase.sh DCTM0 2>&1 | tee /tmp/dctm0.log

with the settings for the stem DCTM0 defined in global_properties.

²The commands for n concurrent instantiations was for instance:

$ for i in {1..n}; do
   time ./instantiate_docbase.sh DCTM0 ORA_DCTM${i} 2>&1 | tee /tmp/ORA_DCTM${i}.log &
done

with the settings for the stems *DCTM1 to *DCTM10 defined in global_properties.

³After an increase of the number of cores from 4 to 20 and of the memory from 10 GiB to 20 GiB in oracle-8, and of the number of cores from 8 to 20 and of the memory from 20 GiB to 30 GiB in cs2. Also, make sure there is enough disk space in both oracle-8 and cs2, e.g. for 10 repositories, 50 GiB for the Oracle data files and 50 GiB for the documentum files.

Management commands

Now that creating docbases has become such an expedite and mundane task (docbases are a commodity at this point), chances are that they’ll start piling up in the same machine and therefore we need a way to manage them easily. At the minimum, such functions must be able to:

• switch among the installed docbases
• show the currently selected docbase and its instantiation parameters
• start/stop/query the status each of a docbase’s component
• restart a docbase, a docbroker or a method server
• navigate to folders of interest in a docbase’s directory tree
• view/tail a docbase’s logs if interest
• connect to the currently selected docbase and to its underlying database using native tools
• cleanly remove an instantiated docbase or all of them at once

Like swr() glimpsed precedently, several functions and bash aliases have been created to fulfill the above needs.

Here are they are, defined in each created or instantiated repository’s environment file ${dctm_root}.${repo}/${repo}.env:

# function shr, for SHow Repository;
# displays the currently selected or given repository's info;
# Usage:
# shr[repository_name]
shr()

# alias whr for WHich Repository to show the currently selected repository;
alias whr=shr

# function sur, for StartUp Repository, to start a repository's all processes;
# it depends on function swr above;
# Usage:
# sur [repository_name]
# when invoked without parameters, the currently selected repository environment is started up;
# when invoked with a parameter, it becomes the current selected repository and its environment is started up;
sur()

# function sdr, for ShutDown Repository, to stop a repository's all processes;
# it depends on function swr above;
# Usage:
# sdr [repository_name]
# when invoked without parameters, the currently selected repository environment is shut down;
# when invoked with a parameter, it becomes the current selected repository and its environment is started up;
sdr()

# function str, for STatus Repository, to show the status of a repository;
# it depends on functions swr;
# Usage:
# rst [repository_name]
# when invoked without parameters, the currently selected repository environment is queried;
# when invoked with a parameter, it becomes the current selected repository and its environment is started up;
rst()

# bounced, for bounce Docbase, stop/start the repository;
alias bounced

# bounceb, for bounce Broker, stop/start the docbroker by minimizing the lost connectivity to the repository;
alias bounceb

# bouncems, for bounce Method Server;
alias bouncems

# cd to directories of interest;
alias croot
alias cdroot
alias cdctm
alias cdba
alias clog

# view files/logs of interest;
# less server.ini
alias lconf

# less repository’s log;
alias llog

# less tomcat’s catalina.out;
alias lcat

# less the method server log;
alias lms

# tails logs of interest; also doable from within the less utility with the F command;
# tail the repository’s log;
alias tlog

# tails tomcat’s catalina.out;
alias tcat

# tails the method server log;
alias tms

# interactive utilities;
# for the repositories;
alias iiapi
alias iidql

# for PostgreSQL
alias iisql
alias ipsql

# for Oracle;
alias isqlp

# function rmr, for Remove Repository, to remove a repository by wiping off its working directory;
# it depends on functions swr and uses sdr defined above;
# Usage:
# rmr [repository_name]
# when invoked without parameters, the currently selected repository environment is removed;
# when invoked with a parameter, it becomes the current selected repository and is removed;
rmr()

Several global functions are also defined in ~/.profile:

# alias lsr for LS Repository;
# list the existing, instanciated repositories;
alias lsr

# function swr, for SWitch Repository;
# tool to switch between instanciated repositories on the same machine and ${dctm_root};
# current working directory is moved to the selected repository's ${DOCUMENTUM};
# Usage:
# swr [repository_name]
# when invoked without parameters, the currently sourced repository's environment is displayed, without first refreshing it;
#
swr()

# function sura, for Start Up Repository All;
# start all the existing, instantiated repositories;
# Usage:
#    sura
#
sura()

# function sdra, for Shut Down Repository All;
# shut down all existing, instantiated repositories;
# Usage:
#    sdra
#
sdra()

# function shra, for Show Repository All;
# show all the existing, instantiated repositories;
# Usage:
#    shra
#
shra()

# function stra, for STatus Repository All;
# show the status of all existing, instantiated repositories;
# Usage:
#    stra
#
stra()

# function rmra, for RM Repository All;
# remove all existing, instantiated repositories;
# Usage:
# rmra
#
rmra()

swr() [repo] is the first function to invoke when starting a new shell and no repository is selected yet. It will switch to the docbase repo. If unsure about repo, a list of existing instantiated repositories can be obtained through the alias lsr which simply list the directories in ${dctm_root}, e.g.:

$ lsr
repo02
repo03
repo04
repo05
repo06
repo07
repo08
repo09
repo10
repo12

$ swr repo12
Switching to repository repo12 ...
ACTIVE_DOCBASE=repo12
ACTIVE_ROOT=/u01/dctm/repo12
CLASSPATH=/u01/dctm/repo12/documentum/product/22.4/dctm-server.jar:/u01/dctm/repo12/documentum/dctm.jar:/u01/dctm/repo12/documentum/dfc/bcprov-jdk15on.jar:/u01/dctm/repo12/documentum/config:/u01/dctm/repo12/documentum/product/22.4/bin:
DM_HOME=/u01/dctm/repo12/documentum/product/22.4
DM_JMS_HOME=/u01/dctm/repo12/documentum/tomcat9.0.65
DOCUMENTUM=/u01/dctm/repo12/documentum
DOCUMENTUM_SHARED=/u01/dctm/repo12/documentum
JAVA_HOME=/u01/dctm/repo12/documentum/java64/JAVA_LINK
LD_LIBRARY_PATH=/u01/dctm/repo12/documentum/product/22.4/bin:/u01/dctm/repo12/documentum/dfc:/u01/dctm/repo12/documentum/fulltext/dsearch:/u01/dctm/repo12/postgresql/lib:
PATH=/u01/dctm/repo12/documentum/java64/JAVA_LINK/bin:/u01/dctm/repo12/documentum/product/22.4/bin:/u01/dctm/repo12/documentum/dba:/u01/dctm/repo12/postgresql/bin:/u01/dctm/repo12/documentum/tomcat9.0.65/bin:/u01/dctm/repo12/java/amazon-corretto-11.0.17.8.1-linux-x64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
POSTGRESQL_HOME=/u01/dctm/repo12/postgresql
PS1=\[\033[0;32m\]\u@\h:\[\033[36m\][\w][repo12]\[\033[0m\] $ 
dfcpath=/u01/dctm/repo12/documentum/dfc
Repository repo12's environment is:
Active docbase name is                  : repo12
Active docbase id is                    : 100013
Active docbase service name is          : repo12
Active docbase service port is          : 50120
Active docbase host is                  : cs2
Active docbase version                  : 22.4.0000.0120
Active docbase root directory is        : /u01/dctm/repo12
Active installer owner is               : dmadmin
Active installer password is            : dmadmin
Active docbase docbroker name           : docbroker010
Active docbase docbroker port           : 1511
Active docbase http server base port    : 9800
Active docbase http server memory       : "-Xms512m -Xmx1024m"
JAVA_HOME                               : /u01/dctm/repo12/documentum/java64/JAVA_LINK
JAVA_VERSION                            :
openjdk version "11.0.17" 2022-10-18 LTS
OpenJDK Runtime Environment Corretto-11.0.17.8.1 (build 11.0.17+8-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.17.8.1 (build 11.0.17+8-LTS, mixed mode)
dctm_root                               : /u01/dctm
Scripts'dir is                          : /mnt/shared/blog-seed-docbase
CLASSPATH                               : /u01/dctm/repo12/documentum/product/22.4/dctm-server.jar:/u01/dctm/repo12/documentum/dctm.jar:/u01/dctm/repo12/documentum/dfc/bcprov-jdk15on.jar:/u01/dctm/repo12/documentum/config:/u01/dctm/repo12/documentum/product/22.4/bin:
DM_HOME                                 : /u01/dctm/repo12/documentum/product/22.4
DOCUMENTUM                              : /u01/dctm/repo12/documentum
DM_JMS_HOME                             : /u01/dctm/repo12/documentum/tomcat9.0.65
PATH                                    : /u01/dctm/repo12/documentum/java64/JAVA_LINK/bin:/u01/dctm/repo12/documentum/product/22.4/bin:/u01/dctm/repo12/documentum/dba:/u01/dctm/repo12/postgresql/bin:/u01/dctm/repo12/documentum/tomcat9.0.65/bin:/u01/dctm/repo12/java/amazon-corretto-11.0.17.8.1-linux-x64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LD_LIBRARY_PATH                         : /u01/dctm/repo12/documentum/product/22.4/bin:/u01/dctm/repo12/documentum/dfc:/u01/dctm/repo12/documentum/fulltext/dsearch:/u01/dctm/repo12/postgresql/lib:
Active database owner is                : repo12
Active database password                : repo12
Active database connection string       : repo12
Active database server port             : 5422
DATABASE_TYPE                           : Postgresql

dmadmin@cs2:[/u01/dctm/repo12][repo12] $ 

It may help to compare functions defined in ~/.profile with static methods in OO programming, whereas functions and aliases defined in ${dctm_root}.${repo}/${repo}.env are similar to instance methods: the former apply to any repository and the latter to the current one, unless a different one is explictly specified. Note that if one of ~/.bash_profile or ~/.bash_login is present, ~/.profile is ignored; Ubuntu tends to use ~/.profile whereas Red Hat derivatives use ~/.bash_profile, so adapt this as needed in your linux variant.

The repository-related functions takes optional docbase names, not stems, because the latter don’t apply to existing docbases and are exclusively used in the create_docbase.sh and instantiate_docbase.sh scripts.

These functions are mostly useful when several repositories have been instantiated on the same machine and one needs to switch quickly among them (see ~/.profile’s swr() above) to perform certain tasks such as show the current one (shr()), start it up (sur()), shut it down (sdr()), check the status of its services (str()), restart services (bounced(), bouceb() and bouncems()), check the logs and configuration (lconf, [lt]log, [lt]cat, [lt]ms), navigate directories of interest (croot, cdroot, cdctm, cdba, clog), launch utilities (iiapi, iidql, iisql, ipsql, isqlp), or wipe it off altogether (rmr()). In effect, removing an instantiated repository should not be any more complicated than instantiating a new one. rmr() also takes care of the Oracle schema if one is used; to do so, it first transfers the script rm_user.sh to the Oracle host and then invokes it. This script removes the database user ${ACTIVE_DATABASE_OWNER}, and drops its tablespaces and datafiles in a safely manner, or so. For PostgreSQL, no such script is needed since the embedded database is removed when the ${ACTIVE_ROOT} directory tree is deleted.

The alias bounceb that stops/starts a docbroker is quite clever and deserves some explanation. When a docbroker is stopped, the docbase that targets it becomes unreachable and stays so until its next projection period, even up to several seconds after the docbroker is back on-line. In order to minimize that unavailability delay, an iapi session is first started while listening to a temporary named pipe, connects to the docbase and is sent to the background; then, the docbroker is restarted, a reinit command is written into the pipe and gets processed by iapi so the docbase projects immediately to the freshly started docbroker. Finally, the quit command is sent to iapi through the named pipe to exit the tool, and the pipe is removed. This works because iapi keeps its connection to the docbase even when the docbroker is stopped. In effect, the docbroker is only used to allow clients to establish a session with the docbases that project onto it; once connected, they don’t need it any more. In that respect, the docbroker has a function similar to an Oracle listener.

Conclusion

While the original idea was simply to speed up the creation of docbases, some of its deliverables can be used as stand-alone utilities, e.g. the script create_docbase.sh, the management aliases and functions.

Those scripts can be used as the basis or greatest common divisor of a more ambitious tool to streamline the installation and operation of Documentum software, e.g. xPlore, DA, etc., on any environment such as VMs, linux or OCI containers.

However useful, the scripts’ main purpose is only to give a practical example of the approach. They were written with a clear context in mind: an individual with full control of their infrastructure (e.g. a powerful enough laptop with virtualization software such as Proxmox), which is not always the case, especially with commercial software. Thus, be prepared to do some adaptations to suit different, likely more rigid contexts, and fix a few bugs at the same time too.

Many enhancements are possible, e.g. use OCI containers, convert any docbase installation into a seed one, etc.; there is enough work to keep one busy during those long, freezing winter nights. Incidentally, it is not impossible that some of them will be covered in future articles, so watch this space.

L’article A quick repository creation utility (part V) est apparu en premier sur dbi Blog.

It may come as a surprise but Documentum SELECT DQL statement has a pagination feature. This largely overlooked capability is implemented with the RETURN_RANGE hint and works quite well.
While it is not such a big deal to programmatically implement a missing feature, it is generally more efficient to use its built-in implementation if it is available. In this article, I’ll show how to paginate through a result set both using a coded implementation and using the RETURN_RANGE hint.
Firstly, we need data we can browse through. It is not hard to generate random strings of characters or numbers but we want them to be plausible, i.e. a fake IP v4 address is not just a 4-group suite of numbers but each number must also be less than 256 and each group is dot-separated; a job title or a company name must look like a real job or company and, at the very least, must be pronounceable. This is not an absolute must but it not only makes using them a more enjoyable experience but sometimes a correct format is expected and necessary for the data to be processed correctly.
Many sources of fake data are available on-line, e.g. this site [sample-csv-files](https://github.com/datablist/sample-csv-files/tree/main/files) has several types of data delivered as easy-to-parse csv files (e.g customers, organizations,people, etc.). For programmers, the [Faker python package](https://faker.readthedocs.io/en/stable/index.html), offers a very rich spectrum of fake data and we will use it too.
Since we work with Documentum, a document management system, using fake tabular data for attributes is not the only factor to consider; document contents in various formats such as pdf, Office, Autodesk or simple text are also to be considered sometimes. Here however, in the context of pagination through metadata, contents are not relevant and we don´t need fake data for them. We will only create content-less documents with fake data as their attributes.

RETURN_RANGE syntax

The OpenText DQL Reference Guide, Appendix A.13, documents that hint in a rather sketchy manner:

RETURN_RANGE starting_row ending_row [optimize_top_row] 'sorting_clause'

The hint specifies which rows are returned by a query sorted by the returned values of specified properties. This hint is provided as a general way to paginate the results of a query.
starting_row specifies the starting row number to return from the qualified rows. It starts at 1.
ending_row specifies the ending row number to return from the qualified rows.
optimize_top_rows provides the top rows for optimization and is optional.
sorting_clause specifies the attribute and its sorting sequence used to determine the range. It defines the sequence of the qualified results. It has the format:

'attribute_name [ASC|DESC] [,attribute_name [ASC|DESC]...]'

It uses ascending by default and is compulsory. Note the mandatory enclosing single quotes.

Thus, starting_row and ending_row define a sliding windows through a result set and by varying their values n .. n + nb_rows – 1, it is possible to navigate page-wise through a result set. The sequence:
1 .. nb_rows
nb_rows + 1 .. 2* nb_rows
2 * nb_rows + 1 .. 3 * nb_rows
…
n * nb_rows + 1 .. (n + 1) * nb_rows
with n = 0, 1, 2 …
determines the 0-based, nb_rows-high page n to reach.

Before we show examples of use, we need the aforementioned fake data. We will generate documents with their metadata filled either by the Faker module or from the website. The following python program populate.py, emits api statements to create content-less documents with content type organization defined as organization(Organization Id,Name,Website,Country,Description,Founded,Industry,Number of employees) as described in organizations samples:

#!/usr/bin/python3
# cec@dbi-services, January 2024;
# Purpose: generate Documentum api statement to populate a repository with content-less documents with fake metadata;
# The generated script can later be fed to iapi for execution;
# Usage:
#    ./populate.py [--csv-url|-c] [--nb-docs|-n]
# where:
#   the optional --csv-url option tells the program to get its csv-formatted data from the given url. By default, the Faker python module is used to internally generate the data;
# Examples:
#  ./populate.py > ./populate.lst 
#  scp ./populate.lst dmadmin@dmtest:/tmp/.
#  ssh dmadmin@dmtest 'bash -l -c "/home/dmadmin/documentum/product/current/bin/iapi dmtest -Udmadmin -Pxxx -R/tmp/populate.lst"'
# or populate the docbase in 1 step:
#    ./populate.py | ssh dmadmin@dmtest 'bash -l -c "/home/dmadmin/documentum/product/current/bin/iapi dmtest -Udmadmin -Pxxx"'
# or just copy the program onto the CS or a CS-client machine, and work directly from there:
#    ./populate.py | /home/dmadmin/documentum/product/current/bin/iapi dmtest -Udmadmin -Pxxx
#   
WORKING_CABINET = 'PAGINATE'
DEFAULT_NB_DOCS = 500

# as the Faker module does not provide values for them, the global constant INDUSTRIES has been manually populated with data from the site as shown below:
# curl -s https://media.githubusercontent.com/media/datablist/sample-csv-files/main/files/organizations/organizations-10000.csv | gawk -v FS="," '{if (!match($8, /^[0-9]+$/)) industries[$8]++}END{for (i in industries) print i | "sort" }' | gawk 'BEGIN{printf "INDUSTRIES = ["} {printf("%s%s", NR > 1 ? ", " : "", "\"" $0 "\"")}END{print "]"}'
INDUSTRIES = ["Accounting", "Airlines / Aviation", "Alternative Dispute Resolution", "Alternative Medicine", "Animation", "Apparel / Fashion", "Architecture / Planning", "Arts / Crafts", "Automotive", "Aviation / Aerospace", "Banking / Mortgage", "Biotechnology / Greentech", "Broadcast Media", "Building Materials", "Business Supplies / Equipment", "Capital Markets / Hedge Fund / Private Equity", "Chemicals", "Civic / Social Organization", "Civil Engineering", "Commercial Real Estate", "Computer Games", "Computer Hardware", "Computer Networking", "Computer / Network Security", "Computer Software / Engineering", "Construction", "Consumer Electronics", "Consumer Goods", "Consumer Services", "Cosmetics", "Dairy", "Defense / Space", "Design", "Education Management", "E - Learning", "Electrical / Electronic Manufacturing", "Entertainment / Movie Production", "Environmental Services", "Events Services", "Executive Office", "Facilities Services", "Farming", "Financial Services", "Fine Art", "Fishery", "Food / Beverages", "Food Production", "Fundraising", "Furniture", "Gambling / Casinos", "Glass / Ceramics / Concrete", "Government Administration", "Government Relations", "Graphic Design / Web Design", "Health / Fitness", "Higher Education / Acadamia", "Hospital / Health Care", "Hospitality", "Human Resources / HR", "Import / Export", "Individual / Family Services", "Industrial Automation", "Industry", "Information Services", "Information Technology / IT", "Insurance", "International Affairs", "International Trade / Development", "Internet", "Investment Banking / Venture", "Investment Management / Hedge Fund / Private Equity", "Judiciary", "Law Enforcement", "Law Practice / Law Firms", "Legal Services", "Legislative Office", "Leisure / Travel", "Library", "Logistics / Procurement", "Luxury Goods / Jewelry", "Machinery", "Management Consulting", "Maritime", "Marketing / Advertising / Sales", "Market Research", "Mechanical or Industrial Engineering", "Media Production", "Medical Equipment", "Medical Practice", "Mental Health Care", "Military Industry", "Mining / Metals", "Motion Pictures / Film", "Museums / Institutions", "Music", "Nanotechnology", "Newspapers / Journalism", "Non - Profit / Volunteering", "Oil / Energy / Solar / Greentech", "Online Publishing", "Other Industry", "Outsourcing / Offshoring", "Package / Freight Delivery", "Packaging / Containers", "Paper / Forest Products", "Performing Arts", "Pharmaceuticals", "Philanthropy", "Photography", "Plastics", "Political Organization", "Primary / Secondary Education", "Printing", "Professional Training", "Program Development", "Public Relations / PR", "Public Safety", "Publishing Industry", "Railroad Manufacture", "Ranching", "Real Estate / Mortgage", "Recreational Facilities / Services", "Religious Institutions", "Renewables / Environment", "Research Industry", "Restaurants", "Retail Industry", "Security / Investigations", "Semiconductors", "Shipbuilding", "Sporting Goods", "Sports", "Staffing / Recruiting", "Supermarkets", "Telecommunications", "Textiles", "Think Tanks", "Tobacco", "Translation / Localization", "Transportation", "Utilities", "Venture Capital / VC", "Veterinary", "Warehousing", "Wholesale", "Wine / Spirits", "Wireless", "Writing / Editing"]

def parse_cmd_line():
   import argparse
   import textwrap
   parser = argparse.ArgumentParser(formatter_class = argparse.RawDescriptionHelpFormatter,
                                    description = textwrap.dedent("""\
This script generates Documentum API statements to create content-less documents with fake metadata.
The generated script can later be fed to iapi for execution.
"""))
   parser.add_argument("--csv-url", "-c",
                       type = str,
                       dest = "csv_url",
                       required = False,
                       default = "",
                       help = "It tells the program to get its csv-formatted data from the given url. By default, the Faker python module is used to internally generate the data.")

   parser.add_argument("--nb-docs", "-n",
                       type = int,
                       dest = "nb_docs",
                       required = False,
                       default = DEFAULT_NB_DOCS,
                       help = "Number of content-less fake documents to create. It defaults to 100.")
   args = parser.parse_args()
   return args.csv_url, args.nb_docs   

def setup_test(cabinet_name):
   print(f'''
set,c,apiconfig,dfc.date_format
d-M-yyyy

create,c,dm_cabinet
set,c,l,object_name
{cabinet_name}
save,c,l
reset,c,l

?,c,create type organization (doc_index integer, organization_id character(15), organization_name character(255), website character(255), country character(100), description character(100), founded date, industry character(100), number_of_employees integer) with supertype dm_document publish
''')

def to_api(cabinet_name, org):
   print(f'''
create,c,organization
set,c,l,subject
Fake document for testing pagination in a result set through the RETURN_RANGE hint;
set,c,l,object_name
{org["organization_name"]}
set,c,l,doc_index
{org["doc_index"]}
set,c,l,organization_id
{org["organization_id"]}
set,c,l,organization_name
{org["organization_name"]}
set,c,l,website
{org["website"]}
set,c,l,country
{org["country"]}
set,c,l,description
{org["description"]}
set,c,l,founded
{org["founded"]}
set,c,l,industry
{org["industry"]}
set,c,l,number_of_employees
{org["number_of_employees"]}
link,c,l,/{cabinet_name}
save,c,l
''')

def generate_fake_organizations(cabinet_name, nb_docs = DEFAULT_NB_DOCS):
   from faker import Faker
   import random
   fake = Faker()
   for i in range(nb_docs):
      to_api(cabinet_name,
             entry := {
                         "doc_index": i + 1, # let's make it 1-based;
                         "organization_id": fake.pystr(min_chars = 15, max_chars = 15),
                         "organization_name": fake.company(),
                         "website": fake.url(schemes = ["https"]),
                         "country": fake.country(),
                         "description": fake.text(max_nb_chars = 100).replace('\n', ' '),
                         "founded": fake.date_of_birth(minimum_age=1, maximum_age=50).strftime("%d-%m-%Y"),
                         "industry": random.sample(INDUSTRIES, 1)[0],
                         "number_of_employees": round(random.randint(10, 50000 + 1), -1)
                      }
            )

def generate_organizations_from_csv(cabinet_name, csv_url, nb_docs):
   import requests
   data  = [l.split(",") for l in requests.get(csv_url).text.split("\r\n")[1:]]
   if not data:
      print(f'no data found from {gcsv_url}, exiting ...')
      exit(0)
   doc_index = 0
   from itertools import cycle
   for _, d in zip(range(nb_docs), cycle(data)):
      if len(d) < 9:
         continue
      to_api(cabinet_name,
             entry := {
                         "doc_index": doc_index + 1, # let's make it 1-based;
                         "organization_id": d[1],
                         "organization_name": d[2],
                         "website": d[3],
                         "country": d[4],
                         "description": d[5],
                         "founded": d[6],
                         "industry": d[7],
                         "number_of_employees": d[8]
                      }
            )
      doc_index += 1

#------------------------------
# main;
if __name__ == '__main__':
   gcsv_url, gnb_docs = parse_cmd_line()

   setup_test(WORKING_CABINET)

   if gcsv_url:
      generate_organizations_from_csv(WORKING_CABINET, gcsv_url, gnb_docs)
   else:
      generate_fake_organizations(WORKING_CABINET, gnb_docs)

   # ready to test pagination using the RETURN_RANGE hint;

   # clean up statements;
   print(f'''
#?,c,delete organization objects
#?,c,delete dm_cabinet object where object_name = '{WORKING_CABINET}'
#?,c,drop type organization
''')

The generated api statements can then be submitted to the iapi interpreter for execution, either on the fly:

./populate.py | /home/dmadmin/documentum/product/current/bin/iapi dmtest -Udmadmin -Pxxx

or in 2 steps:

./populate.py > ./populate.api
/home/dmadmin/documentum/product/current/bin/iapi dmtest -Udmadmin -Pxxx -R./populate.api

or anything in between if the interpreter is installed on a remote machine (see the program’s header).
Here is the program’s usage:

 ./paginate.py  --help
usage: paginate.py [-h] [--csv-url CSV_URL] [--nb-docs NB_DOCS]

This script generates Documentum API statements to create content-less documents with fake metadata.
The generated script can later be fed to iapi for execution.

options:
  -h, --help            show this help message and exit
  --csv-url CSV_URL, -c CSV_URL
                        It tells the program to get its csv-formatted data from the given url. By default, the Faker python module is used to internally generate the data.
  --nb-docs NB_DOCS, -n NB_DOCS
                        Number of content-less fake documents to create. It defaults to 100.

So, if Faker is preferred, invoke it with no csv url, like this:

./paginate.py  --nb-docs=200

If the csv data from the web site are preferred, specify the csv url like this:

./paginate.py  --nb-docs=200 --csv-url=https://media.githubusercontent.com/media/datablist/sample-csv-files/main/files/organizations/organizations-100.csv

The available data will be cycled until the specified number of documents is reached.
Here are the first few statements emitted by the program:

set,c,apiconfig,dfc.date_format
d-M-yyyy

create,c,dm_cabinet
set,c,l,object_name
PAGINATE
save,c,l
reset,c,l

?,c,create type organization (doc_index integer, organization_id character(15), organization_name character(255), website character(255), country character(100), description character(100), founded date, industry character(100), number_of_employees integer) with supertype dm_document publish


create,c,organization
set,c,l,subject
Fake document for testing pagination in a result set through the RETURN_RANGE hint;
set,c,l,object_name
Jones and Sons
set,c,l,doc_index
0
set,c,l,organization_id
LdxOVYaDjNMbtGE
set,c,l,organization_name
Jones and Sons
set,c,l,website
https://www.norton.com/
set,c,l,country
Gabon
set,c,l,description
Bit item loss maybe center can. Story expert budget these kitchen along.
set,c,l,founded
18-01-1992
set,c,l,industry
Wine / Spirits
set,c,l,number_of_employees
47550
link,c,l,/PAGINATE
save,c,l


create,c,organization
set,c,l,subject
Fake document for testing pagination in a result set through the RETURN_RANGE hint;
set,c,l,object_name
Bauer Ltd
set,c,l,doc_index
1
set,c,l,organization_id
HJwnJzuXxxmEqDI
set,c,l,organization_name
Bauer Ltd
set,c,l,website
https://kent.net/
set,c,l,country
Heard Island and McDonald Islands
set,c,l,description
Read second six out close shoulder card see. Beyond state back that common fine bank.
set,c,l,founded
18-03-2010
set,c,l,industry
Construction
set,c,l,number_of_employees
43340
link,c,l,/PAGINATE
save,c,l
...
# clean up;
#?,c,delete organization objects
#?,c,delete dm_cabinet object where object_name = 'PAGINATE'
#?,c,drop type organization

After a suitable date format has been set in the session, a cabinet /PAGINATE to contain the fake documents and the doctype organization are created, followed by the creation of the specified number of documents. The attribute doc_index contains the index number of the document in the range 0 .. nb_docs – 1, and subject is set to a constant text explaining the purpose of the document; this makes it easy to identify the test documents (not really necessary here since we only work with one doctype, organization; let’s say it is another faker data, a constant one). The generated API script ends with some clean up statements to rollback the test; they are commented in order to be used later.

Now that we have the documents in the repository, let’s see how they look like as follows:

select count(*) from organization
count(*)              
----------------------
                   200
(1 row affected)

select doc_index, subject, object_name, organization_id, organization_name, website, country, description, founded, industry, number_of_employees from organization order by doc_index
doc_index     subject                                                                                                                                                                                           object_name                                                                                                                                                                                                                                                      organization_id  organization_name                                                                                                                                                                                                                                                website                                                                                                                                                                                                                                                          country                                                                                               description                                                                                           founded                    industry                                                                                              number_of_employees
------------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------  ---------------  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------  ----------------------------------------------------------------------------------------------------  ----------------------------------------------------------------------------------------------------  -------------------------  ----------------------------------------------------------------------------------------------------  -------------------
           1  Fake document for testing pagination in a result set through the RETURN_RANGE hint;                                                                                                               Jones and Sons                                                                                                                                                                                                                                                   LdxOVYaDjNMbtGE  Jones and Sons                                                                                                                                                                                                                                                   https://www.norton.com/                                                                                                                                                                                                                                          Gabon                                                                                                 Bit item loss maybe center can. Story expert budget these kitchen along.                              18-1-1992                  Wine / Spirits                                                                                                      47550
           2  Fake document for testing pagination in a result set through the RETURN_RANGE hint;                                                                                                               Bauer Ltd                                                                                                                                                                                                                                                        HJwnJzuXxxmEqDI  Bauer Ltd                                                                                                                                                                                                                                                        https://kent.net/                                                                                                                                                                                                                                                Heard Island and McDonald Islands                                                                     Read second six out close shoulder card see. Beyond state back that common fine bank.                 18-3-2010                  Construction                                                                                                        43340
           3  Fake document for testing pagination in a result set through the RETURN_RANGE hint;                                                                                                               Erickson, Smith and Jackson                                                                                                                                                                                                                                      xZLxZCzOOLZavxz  Erickson, Smith and Jackson                                                                                                                                                                                                                                      https://www.walls-moore.biz/                                                                                                                                                                                                                                     United States of America                                                                              Create spend summer job age eye able. Learn standard month question full defense.                     12-11-2022                 Research Industry                                                                                                   45570
           4  Fake document for testing pagination in a result set through the RETURN_RANGE hint;                                                                                                               Anthony-Moyer                                                                                                                                                                                                                                                    eiuzuuAhwYFZlrg  Anthony-Moyer                                                                                                                                                                                                                                                    https://www.chambers.com/                                                                                                                                                                                                                                        Vanuatu                                                                                               Meeting pass maintain nothing news. Big simply measure class lead him.                                9-10-2007                  Museums / Institutions                                                                                               3400
           5  Fake document for testing pagination in a result set through the RETURN_RANGE hint;                                                                                                               Richmond-Hudson                                                                                                                                                                                                                                                  hBtsJqBxNyRacQI  Richmond-Hudson                                                                                                                                                                                                                                                  https://www.schneider-green.com/                                                                                                                                                                                                                                 Cote d'Ivoire                                                                                         Strategy team teach building already whose. Letter book both technology base sometimes.               19-5-2002                  Alternative Medicine                                                                                                 1150
           6  Fake document for testing pagination in a result set through the RETURN_RANGE hint;                                                                                                               Robles PLC                                                                                                                                                                                                                                                       budHsKmJHREufEo  Robles PLC                                                                                                                                                                                                                                                       https://maldonado.biz/                                                                                                                                                                                                                                           Austria                                                                                               Face seven important weight will. East claim coach act.                                               6-1-2001                   Government Relations                                                                                                32500
...
(200 rows affected)

Incidentally, we can see in the above output the eternal presentation problem that plagues the iapi/idql command-line tools: the attributes are displayed in fixed-length columns with lots of trailing blanks and the rows are wrapped around if too long to fit on the screen width. This issue has been discussed and solutions have been given several times in this blog. Here is yet another work around (sometimes it is more fun to rewrite it from scratch than to search a script we wrote ourselves !), the script compact.awk:

# cec@dbi-services, January 2024;
# Purpose: compact a tabular DQL SELECT output by stripping the columns'unnecessary trailing blanks;
# Usage:
#    gawk -f compact.awk output.lst
# as usual, input can also be taken from stdin;
# Examples:
#  echo "?,c,select * from organization" | iapi dmtest73 -Udmadmin -Pxxx | gawk -f compact.awk | less
BEGIN {
   while((getline > 0) && !match($0, /^-+/)) header = $0;
   if (!RSTART)
      exit
   sep_line = $0

   nb_cols = NF
   match($0, / +/); sep_length = RLENGTH
   FIELDWIDTHS = ""
   for (i = 1; i <= nb_cols; i++)
      FIELDWIDTHS = sprintf("%s%s%i", FIELDWIDTHS, i > 1 ? " " : "", length($i) + sep_length)
   #attr_re = "^(.+) {" sep_length "}$"
   nb_rows = 0
 }

 {
    if (match($0, /\([0-9]+ row. affected\)/)) {
       footer = $0
       exit
    }
    row[nb_rows++] = $0
    for (i = 1; i <= NF; i++) {
       # reg exp would likely be faster but they don't work here because gawk's regular expressions are greedy and they would consume all the trailing blanks except the ending sep_length ones whereas we want to skip all of them;
       # to work around this, we physically remove them;
       #match($i, attr_re, found)
       #w[i] = max(w[i], found[1])
       gsub(/ +$/, "", $i)
       w[i] = max(w[i], length($i))
    }
}

END {
   for (i = 1; i <= NF; i++)
      pad[i] = repeat(" ", w[i])

   print(compact(header))
   print(compact(sep_line))

   for (i = 0; i < nb_rows; i++)
      print(compact(row[i]))

   print footer
 }

function max(a, b) {
   if (a >= b)
      return a
   else
      return b
}

function compact(line    , s, i) {
   $0 = line
   s = ""
   for (i = 1; i <= NF; i++)
      s = sprintf("%s%s%s", s, i > 1 ? "  " : "", substr($i pad[i], 1, w[i]))
   return s
}

function repeat(ch, nb_times   , s) {
   s = ""
   while (nb_times-- > 0)
      s = s + ch
   return s
}

Usage:

echo "?,c,select doc_index, subject, object_name, organization_id, organization_name, website, country, description, founded, industry, number_of_employees from organization order by doc_index" | iapi dmtest73 -Udmadmin -Pxxx | gawk -f compact.awk 

Output:

doc_ind       subject                                                                              object_name                       organizati  _id  organization_name                 website                              country                                       description                                                                                          founded              industry                                             number_of_empl
------------  -----------------------------------------------------------------------------------  --------------------------------  ---------------  --------------------------------  -----------------------------------  --------------------------------------------  ---------------------------------------------------------------------------------------------------  -------------------  ---------------------------------------------------  -------------------
           1  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Jones and Sons                    LdxOVYaDjNMbtGE  Jones and Sons                    https://www.norton.com/              Gabon                                         Bit item loss maybe center can. Story expert budget these kitchen along.                             1/18/1992 00:00:00   Wine / Spirits                                                     47550
           2  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Bauer Ltd                         HJwnJzuXxxmEqDI  Bauer Ltd                         https://kent.net/                    Heard Island and McDonald Islands             Read second six out close shoulder card see. Beyond state back that common fine bank.                3/18/2010 00:00:00   Construction                                                       43340
           3  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Erickson, Smith and Jackson       xZLxZCzOOLZavxz  Erickson, Smith and Jackson       https://www.walls-moore.biz/         United States of America                      Create spend summer job age eye able. Learn standard month question full defense.                    11/12/2022 00:00:00  Research Industry                                                  45570
           4  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Anthony-Moyer                     eiuzuuAhwYFZlrg  Anthony-Moyer                     https://www.chambers.com/            Vanuatu                                       Meeting pass maintain nothing news. Big simply measure class lead him.                               10/9/2007 00:00:00   Museums / Institutions                                              3400
           5  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Richmond-Hudson                   hBtsJqBxNyRacQI  Richmond-Hudson                   https://www.schneider-green.com/     Cote d'Ivoire                                 Strategy team teach building already whose. Letter book both technology base sometimes.              5/19/2002 00:00:00   Alternative Medicine                                                1150
           6  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Robles PLC                        budHsKmJHREufEo  Robles PLC                        https://maldonado.biz/               Austria                                       Face seven important weight will. East claim coach act.                                              1/6/2001 00:00:00    Government Relations                                               32500
           7  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Nichols Ltd                       hJYonMSFNXaYaQG  Nichols Ltd                       https://www.leon-sims.biz/           Falkland Islands (Malvinas)                   Certain hand impact method somebody thank song. Democrat travel article reach sea article.           5/31/1997 00:00:00   Leisure / Travel                                                   35760
           8  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Elliott-Valdez                    mstxDZfBafifCJn  Elliott-Valdez                    https://www.ward.com/                Hungary                                       Operation item nature stock dog attack into. Vote edge whose keep.                                   12/7/1995 00:00:00   Publishing Industry                                                21080
           9  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Zamora PLC                        TuJPwIFoqEiYFUz  Zamora PLC                        https://www.oliver.com/              Venezuela                                     Chance expert general. Garden this positive. Edge young share beautiful admit thank name.            5/7/1993 00:00:00    Cosmetics                                                          38410
          10  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Hampton Ltd                       jRZDRGrfgyWTpAn  Hampton Ltd                       https://www.gilbert.com/             Estonia                                       Chance discussion call yeah. Measure entire season scene behind radio. Than machine ball material.   4/16/1975 00:00:00   Accounting                                                         36020
          11  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Hernandez, Smith and Mcintosh     QLpUVwnuZVdIoCw  Hernandez, Smith and Mcintosh     https://garcia-peterson.com/         Switzerland                                   Live hundred would let food. Else nice firm door still. Hair technology trial.                       4/28/2007 00:00:00   Supermarkets                                                       11630
          12  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Fernandez LLC                     mCdBZkIqunyExsA  Fernandez LLC                     https://clark-hernandez.com/         Ukraine                                       Health paper child worry thus produce light. Get them hope garden show think.                        5/19/1997 00:00:00   Market Research                                                    46500
          13  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Stevens, Greer and Young          QzSMkppsZEbfJQF  Stevens, Greer and Young          https://www.fuller-moses.com/        Finland                                       Mr discuss pretty after whose actually guy. Music pass movement still.                               12/23/1988 00:00:00  Public Safety                                                       3500
          14  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Zuniga and Sons                   jVMjZmDdeUejohj  Zuniga and Sons                   https://www.porter-richardson.net/   Gibraltar                                     Agent heart who which dog father. Find day eight her loss. Fine agency say player training.          10/11/1994 00:00:00  Information Services                                                 920
          15  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Murphy Group                      zJDwYcXPAnmEvos  Murphy Group                      https://parks.com/                   Guyana                                        Democratic room year nature fact century change. Me great fear able watch father compare.            7/2/1999 00:00:00    Textiles                                                           44560
          16  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Edwards, Howe and Munoz           ChLMutRVzUKnzXN  Edwards, Howe and Munoz           https://nichols-walsh.com/           Vietnam                                       Hotel bring fire laugh really interview remember.                                                    4/21/2006 00:00:00   Aviation / Aerospace                                               30370
          17  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Bowers Inc                        uBmqyxuKEBUcGSg  Bowers Inc                        https://rivera.com/                  Libyan Arab Jamahiriya                        Often than number story land particularly. Term sure surface court drop any crime.                   12/1/1981 00:00:00   Outsourcing / Offshoring                                            7800
          18  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Fisher-Walker                     NyMFDEmxlPertcI  Fisher-Walker                     https://estrada.com/                 Iraq                                          Today feeling amount. Interview take wide report. Fire price foot probably follow sort.              11/29/2016 00:00:00  Legislative Office                                                 28060
          19  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Smith, Hawkins and Baker          GtqNebcnXThOIEz  Smith, Hawkins and Baker          https://wheeler.info/                Senegal                                       Manager eight forward each word. Choice defense responsibility develop half tax prove.               7/6/2017 00:00:00    Recreational Facilities / Services                                  7230
          20  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Lynch PLC                         XBPvgaxcwUplVyX  Lynch PLC                         https://www.ferguson.com/            Mauritius                                     Career region along. School summer stock later item. Deal say seat customer.                         2/21/2013 00:00:00   Political Organization                                             23000
          21  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Harris Inc                        ahvZGlBZLSIxFSU  Harris Inc                        https://www.porter-wells.info/       Sri Lanka                                     Many religious sit enjoy south technology money. Center month stop clearly. Home like carry set.     1/29/1997 00:00:00   Medical Equipment                                                  40240
          22  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Powell Ltd                        UaDDsMjqjxnwdOt  Powell Ltd                        https://alvarado.biz/                United Arab Emirates                          Everything people heavy off. Agree option growth majority economy music TV.                          8/29/2012 00:00:00   Electrical / Electronic Manufacturing                              25180
...
         197  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Hill Ltd                          OgennqjqitKpuYb  Hill Ltd                          https://owen.com/                    New Zealand                                   Trade keep institution source you. Wife quality all risk to.                                         3/26/2003 00:00:00   Telecommunications                                                 45690
         199  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Davila and Sons                   rUdByKYjLVznQSf  Davila and Sons                   https://ho.com/                      Lesotho                                       Themselves none reach realize accept. Style general energy medical address blood scene.              8/16/2004 00:00:00   Market Research                                                     2780
         199  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Hancock, Turner and Robinson      swilppvRqeWoIha  Hancock, Turner and Robinson      https://www.davila-collins.biz/      Sri Lanka                                     Star right effort modern six.                                                                        10/15/1980 00:00:00  Biotechnology / Greentech                                          20300
         200  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Spears Group                      zhLEEoIUUPmHqkF  Spears Group                      https://www.vasquez.info/            Uganda                                        Not staff success possible special. Exactly Congress local billion without fly staff expert.         11/4/2004 00:00:00   Food / Beverages                                                   22240
(200 rows affected)

Much better.

We now have set up the stage for demonstrating the pagination feature of the RETURN_RANGE hint. Let’s show a few examples (for better readability, the output has been edited using the above compact.awk gawk script):

# first page:
select doc_index, subject, object_name, organization_id, organization_name, website, country, description, founded, industry, number_of_employees from organization enable(RETURN_RANGE 1 10 'doc_index ASC')
doc_index     subject                                                                              object_name                  organization_id  organization_name            website                           country                            description                                                                                         founded              industry                number_of_employees  dm_rnum               
------------  -----------------------------------------------------------------------------------  ---------------------------  ---------------  ---------------------------  --------------------------------  ---------------------------------  --------------------------------------------------------------------------------------------------  -------------------  ----------------------  -------------------  ----------------------
           1  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Jones and Sons               LdxOVYaDjNMbtGE  Jones and Sons               https://www.norton.com/           Gabon                              Bit item loss maybe center can. Story expert budget these kitchen along.                            1/18/1992 00:00:00   Wine / Spirits                        47550                       1
           2  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Bauer Ltd                    HJwnJzuXxxmEqDI  Bauer Ltd                    https://kent.net/                 Heard Island and McDonald Islands  Read second six out close shoulder card see. Beyond state back that common fine bank.               3/18/2010 00:00:00   Construction                          43340                       2
           3  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Erickson, Smith and Jackson  xZLxZCzOOLZavxz  Erickson, Smith and Jackson  https://www.walls-moore.biz/      United States of America           Create spend summer job age eye able. Learn standard month question full defense.                   11/12/2022 00:00:00  Research Industry                     45570                       3
           4  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Anthony-Moyer                eiuzuuAhwYFZlrg  Anthony-Moyer                https://www.chambers.com/         Vanuatu                            Meeting pass maintain nothing news. Big simply measure class lead him.                              10/9/2007 00:00:00   Museums / Institutions                 3400                       4
           5  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Richmond-Hudson              hBtsJqBxNyRacQI  Richmond-Hudson              https://www.schneider-green.com/  Cote d'Ivoire                      Strategy team teach building already whose. Letter book both technology base sometimes.             5/19/2002 00:00:00   Alternative Medicine                   1150                       5
           6  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Robles PLC                   budHsKmJHREufEo  Robles PLC                   https://maldonado.biz/            Austria                            Face seven important weight will. East claim coach act.                                             1/6/2001 00:00:00    Government Relations                  32500                       6
           7  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Nichols Ltd                  hJYonMSFNXaYaQG  Nichols Ltd                  https://www.leon-sims.biz/        Falkland Islands (Malvinas)        Certain hand impact method somebody thank song. Democrat travel article reach sea article.          5/31/1997 00:00:00   Leisure / Travel                      35760                       7
           8  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Elliott-Valdez               mstxDZfBafifCJn  Elliott-Valdez               https://www.ward.com/             Hungary                            Operation item nature stock dog attack into. Vote edge whose keep.                                  12/7/1995 00:00:00   Publishing Industry                   21080                       8
           9  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Zamora PLC                   TuJPwIFoqEiYFUz  Zamora PLC                   https://www.oliver.com/           Venezuela                          Chance expert general. Garden this positive. Edge young share beautiful admit thank name.           5/7/1993 00:00:00    Cosmetics                             38410                       9
          10  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Hampton Ltd                  jRZDRGrfgyWTpAn  Hampton Ltd                  https://www.gilbert.com/          Estonia                            Chance discussion call yeah. Measure entire season scene behind radio. Than machine ball material.  4/16/1975 00:00:00   Accounting                            36020                      10
(10 rows affected)

# next page:
select doc_index, subject, object_name, organization_id, organization_name, website, country, description, founded, industry, number_of_employees from organization enable(RETURN_RANGE 11 20 'doc_index ASC')
doc_index     subject                                                                              object_name                    organization_id  organization_name              website                             country                 description                                                                                  founded              industry                            number_of_employees  dm_rnum               
------------  -----------------------------------------------------------------------------------  -----------------------------  ---------------  -----------------------------  ----------------------------------  ----------------------  -------------------------------------------------------------------------------------------  -------------------  ----------------------------------  -------------------  ----------------------
          11  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Hernandez, Smith and Mcintosh  QLpUVwnuZVdIoCw  Hernandez, Smith and Mcintosh  https://garcia-peterson.com/        Switzerland             Live hundred would let food. Else nice firm door still. Hair technology trial.               4/28/2007 00:00:00   Supermarkets                                      11630                      11
          12  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Fernandez LLC                  mCdBZkIqunyExsA  Fernandez LLC                  https://clark-hernandez.com/        Ukraine                 Health paper child worry thus produce light. Get them hope garden show think.                5/19/1997 00:00:00   Market Research                                   46500                      12
          13  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Stevens, Greer and Young       QzSMkppsZEbfJQF  Stevens, Greer and Young       https://www.fuller-moses.com/       Finland                 Mr discuss pretty after whose actually guy. Music pass movement still.                       12/23/1988 00:00:00  Public Safety                                      3500                      13
          14  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Zuniga and Sons                jVMjZmDdeUejohj  Zuniga and Sons                https://www.porter-richardson.net/  Gibraltar               Agent heart who which dog father. Find day eight her loss. Fine agency say player training.  10/11/1994 00:00:00  Information Services                                920                      14
          15  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Murphy Group                   zJDwYcXPAnmEvos  Murphy Group                   https://parks.com/                  Guyana                  Democratic room year nature fact century change. Me great fear able watch father compare.    7/2/1999 00:00:00    Textiles                                          44560                      15
          16  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Edwards, Howe and Munoz        ChLMutRVzUKnzXN  Edwards, Howe and Munoz        https://nichols-walsh.com/          Vietnam                 Hotel bring fire laugh really interview remember.                                            4/21/2006 00:00:00   Aviation / Aerospace                              30370                      16
          17  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Bowers Inc                     uBmqyxuKEBUcGSg  Bowers Inc                     https://rivera.com/                 Libyan Arab Jamahiriya  Often than number story land particularly. Term sure surface court drop any crime.           12/1/1981 00:00:00   Outsourcing / Offshoring                           7800                      17
          18  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Fisher-Walker                  NyMFDEmxlPertcI  Fisher-Walker                  https://estrada.com/                Iraq                    Today feeling amount. Interview take wide report. Fire price foot probably follow sort.      11/29/2016 00:00:00  Legislative Office                                28060                      18
          19  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Smith, Hawkins and Baker       GtqNebcnXThOIEz  Smith, Hawkins and Baker       https://wheeler.info/               Senegal                 Manager eight forward each word. Choice defense responsibility develop half tax prove.       7/6/2017 00:00:00    Recreational Facilities / Services                 7230                      19
          20  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Lynch PLC                      XBPvgaxcwUplVyX  Lynch PLC                      https://www.ferguson.com/           Mauritius               Career region along. School summer stock later item. Deal say seat customer.                 2/21/2013 00:00:00   Political Organization                            23000                      20
(10 rows affected)

# 3rd page, 20 rows instead of 10 this time;
select doc_index, subject, object_name, organization_id, organization_name, website, country, description, founded, industry, number_of_employees from organization enable(RETURN_RANGE 21 40 'doc_index ASC')
doc_index     subject                                                                              object_name                    organization_id  organization_name              website                            country                      description                                                                                         founded              industry                               number_of_employees  dm_rnum               
------------  -----------------------------------------------------------------------------------  -----------------------------  ---------------  -----------------------------  ---------------------------------  ---------------------------  --------------------------------------------------------------------------------------------------  -------------------  -------------------------------------  -------------------  ----------------------
          21  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Harris Inc                     ahvZGlBZLSIxFSU  Harris Inc                     https://www.porter-wells.info/     Sri Lanka                    Many religious sit enjoy south technology money. Center month stop clearly. Home like carry set.    1/29/1997 00:00:00   Medical Equipment                                    40240                      21
          22  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Powell Ltd                     UaDDsMjqjxnwdOt  Powell Ltd                     https://alvarado.biz/              United Arab Emirates         Everything people heavy off. Agree option growth majority economy music TV.                         8/29/2012 00:00:00   Electrical / Electronic Manufacturing                25180                      22
          23  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Orr-Andrews                    mjxWxcxBFUHkdkw  Orr-Andrews                    https://anderson.com/              El Salvador                  Main structure sense black front. Best capital opportunity national past truth kitchen radio.       2/9/1977 00:00:00    Alternative Medicine                                  9510                      23
          24  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Weeks and Sons                 EjzOYRMYOqPGcBh  Weeks and Sons                 https://www.morgan.info/           Bouvet Island (Bouvetoya)    Analysis close live box show even responsibility. Spring worker son common chair traditional.       3/10/2017 00:00:00   Consumer Services                                     1070                      24
          25  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Phillips-Sheppard              vvpgEUvmDTvHVUj  Phillips-Sheppard              https://woods-romero.biz/          Dominican Republic           Same some tough leader drug late. Manage skill media entire rise large two.                         9/20/1993 00:00:00   Aviation / Aerospace                                 18120                      25
          26  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Mcguire, Shepherd and Johnson  suJFaiReXkfmVqm  Mcguire, Shepherd and Johnson  https://www.carr-barker.com/       American Samoa               Citizen approach be offer wear carry tend. Better difference age.                                   12/25/1981 00:00:00  Wireless                                             10380                      26
          27  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Hess LLC                       kMLbTuGukEwDjDB  Hess LLC                       https://carr.com/                  Turks and Caicos Islands     Adult tend night company guess only opportunity. Evening establish contain usually.                 3/6/1996 00:00:00    Religious Institutions                               14130                      27
          28  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Franklin Ltd                   zmYKtQduRPSEfit  Franklin Ltd                   https://www.robinson.biz/          Romania                      First have seem standard protect want service. Occur knowledge loss popular friend number forward.  2/20/1998 00:00:00   Computer Hardware                                    32100                      28
          29  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Leonard Group                  SVkTDkwAHJnFYJl  Leonard Group                  https://www.rojas.com/             Cuba                         Open lose between state capital vote describe. Game sit few nor. One lawyer involve.                7/18/1987 00:00:00   Primary / Secondary Education                        15830                      29
          30  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Frank Ltd                      vAIaBKYKygOJPNV  Frank Ltd                      https://www.mills.com/             Cayman Islands               Institution they seat society. Over instead soon tonight improve collection traditional.            4/29/1977 00:00:00   Individual / Family Services                         45100                      30
          31  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Taylor-Mcclure                 sLSOKbnznjxgnfu  Taylor-Mcclure                 https://www.johnson.com/           Mayotte                      Bar bank sister. Old food threat range none front focus today. Another determine quality each.      8/31/2015 00:00:00   Consumer Goods                                       37740                      31
          32  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Cisneros-Mathews               rlaCfcsmTVhhLHq  Cisneros-Mathews               https://www.rowland.info/          Martinique                   List reflect nice know. Section party interview cut surface evening.                                4/1/2018 00:00:00    Warehousing                                          23880                      32
          33  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Brown, Nunez and Boyd          NXxqXQabfPoLUKB  Brown, Nunez and Boyd          https://www.webb.com/              Denmark                      Score more Mrs stock perhaps. Situation food full raise.                                            9/20/1973 00:00:00   Information Technology / IT                          22270                      33
          34  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Martin-Hodge                   NJDowVoehnKfzUj  Martin-Hodge                   https://young.org/                 Mongolia                     Every contain young tonight. Seat full food ready life. Serve eat yourself recent explain white.    5/4/2013 00:00:00    Aviation / Aerospace                                 49180                      34
          35  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Shannon, Harding and Sanchez   uXxjkktfvjuzCov  Shannon, Harding and Sanchez   https://peterson.net/              French Southern Territories  Business laugh again party. Opportunity arrive choose kitchen toward act under.                     2/19/1979 00:00:00   Religious Institutions                               10370                      35
          36  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Brown, White and Perkins       VoeETffAHGfGXuZ  Brown, White and Perkins       https://www.anderson.com/          Spain                        Who base explain control country drive. Worry picture level forget audience body.                   10/21/2022 00:00:00  Telecommunications                                   21910                      36
          37  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Watson, Murray and Owens       kKmKHiJDZGEQwjn  Watson, Murray and Owens       https://www.baxter-contreras.com/  French Southern Territories  Really drug western law note. Group actually what. Himself full himself into degree feeling talk.   1/2/2001 00:00:00    Wireless                                             45470                      37
          38  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Taylor-Lopez                   fJyeWvzKzIgJdya  Taylor-Lopez                   https://www.schultz.com/           Nigeria                      Never million moment very. Management explain five Mr.                                              4/15/1977 00:00:00   Museums / Institutions                               45630                      38
          39  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Owens PLC                      dysBZBBxVQqPcwZ  Owens PLC                      https://barnes.info/               Jersey                       Her present shake without yard occur conference. Walk arrive learn.                                 6/21/1986 00:00:00   Plastics                                              3060                      39
          40  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Hernandez LLC                  iUAnDDOlNVetiuu  Hernandez LLC                  https://copeland.com/              Norway                       Story boy talk yourself.                                                                            8/13/1986 00:00:00   Other Industry                                       11100                      40
(20 rows affected)

# last 10 rows;
select doc_index, subject, object_name, organization_id, organization_name, website, country, description, founded, industry, number_of_employees from organization enable(RETURN_RANGE 191 200 'doc_index ASC')
oc_index     subject                                                                              object_name                   organization_id  organization_name             website                          country                   description                                                                                   founded              industry                          number_of_employees  dm_rnum               
------------  -----------------------------------------------------------------------------------  ----------------------------  ---------------  ----------------------------  -------------------------------  ------------------------  --------------------------------------------------------------------------------------------  -------------------  --------------------------------  -------------------  ----------------------
         191  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Miller, Watson and Gonzalez   JgwNOKaVUquYcBk  Miller, Watson and Gonzalez   https://tucker-washington.com/   Qatar                     Parent entire religious method eye third. Middle report close get though meet gas.            1/29/2001 00:00:00   Oil / Energy / Solar / Greentech                41490                     191
         192  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Bishop-Perry                  wFTAmAlodFqGBRm  Bishop-Perry                  https://www.davis.com/           Liberia                   Age forget house writer side exactly human. Design memory push else probably design.          11/26/2019 00:00:00  Telecommunications                              14550                     192
         193  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Taylor, Mcintyre and Brady    rLgBOoqjMcAwMzg  Taylor, Mcintyre and Brady    https://gonzalez-collins.org/    Reunion                   Brother just we for book year.                                                                8/12/1992 00:00:00   Law Practice / Law Firms                        19780                     193
         194  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Roth-Wright                   kLqlxKbggAjGGXT  Roth-Wright                   https://ramirez-perkins.org/     Cape Verde                Let believe every environmental. Too matter fly any. Painting exist religious campaign.       4/7/2020 00:00:00    Construction                                    13750                     194
         195  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Melton, Davis and Reed        ePQOLAEfxBUXbDy  Melton, Davis and Reed        https://www.gates.com/           Northern Mariana Islands  Building pass minute strategy program family across. Move save clear moment bring career.     7/21/1989 00:00:00   Automotive                                       5450                     195
         196  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Garcia Inc                    ElmnJXeZtrvjUbf  Garcia Inc                    https://walters.org/             Russian Federation        Your degree billion practice instead measure point much. Area strong notice phone buy.        7/14/1995 00:00:00   Legal Services                                  24030                     196
         197  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Hill Ltd                      OgennqjqitKpuYb  Hill Ltd                      https://owen.com/                New Zealand               Trade keep institution source you. Wife quality all risk to.                                  3/26/2003 00:00:00   Telecommunications                              45690                     197
         198  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Davila and Sons               rUdByKYjLVznQSf  Davila and Sons               https://ho.com/                  Lesotho                   Themselves none reach realize accept. Style general energy medical address blood scene.       8/16/2004 00:00:00   Market Research                                  2780                     198
         199  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Hancock, Turner and Robinson  swilppvRqeWoIha  Hancock, Turner and Robinson  https://www.davila-collins.biz/  Sri Lanka                 Star right effort modern six.                                                                 10/15/1980 00:00:00  Biotechnology / Greentech                       20300                     199
         200  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Spears Group                  zhLEEoIUUPmHqkF  Spears Group                  https://www.vasquez.info/        Uganda                    Not staff success possible special. Exactly Congress local billion without fly staff expert.  11/4/2004 00:00:00   Food / Beverages                                22240                     200
(10 rows affected)

So far, no surprise here, although we note that a new pseudo-column is appended, dm_rnum, for numbering the row from starting_row to ending_row. This column is not accessible directly and trying to do so yields the error [DM_QUERY_E_NOT_ATTRIBUTE]error: "You have specified an invalid attribute name (dm_rnum).".
If we mistakenly include an ORDER BY clause:

select doc_index, subject, object_name, organization_id, organization_name, website, country, description, founded, industry, number_of_employees from organization order by 1 enable(RETURN_RANGE 1 10 'doc_index ASC')
[DM_QUERY_E_CURSOR_ERROR]error:  "A database error has occurred during the creation of a cursor ('ORA-00933: SQL command not properly ended')."

The error message is not very informative but it looks like the generated SQL has a syntax error. Let’s display that SQL statement:

execute get_last_sql
result                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
select * from ( select dm_inline_view_1.*, ROW_NUMBER() OVER(ORDER BY dm_inline_view_1.doc_index ASC ) as dm_rnum from (select distinct organization.doc_index, organization.subject, organization.object_name, organization.organization_id, organization.organization_name, organization.website, organization.country, organization.description, organization.founded, organization.industry, organization.number_of_employees from organization_sp  organization where (organization.i_has_folder = 1 and organization.i_is_deleted = 0) ) dm_inline_view_1) dm_inline_view_2 where dm_rnum >= 1   AND dm_rnum <= 10 order by dm_rnum order by 1

The offending clause is order by dm_rnum order by 1. In effect, our ORDER BY clause was appended to the generated SQL which already contained one such clause, which invalidate the statement’s syntax. Actually, we don’t need to provide such a clause because it can be included in the hint; indeed, it MUST be included since the sorting clause is mandatory.
The above generated SQL is interesting as it shows how the RETURN_RANGE hint is implemented, here in the underlying Oracle RDBMS; the analytical function ROW_NUMBER() is used to number the rows; it determines the facts that rows are 1-based and that sorting_clause is mandatory.

Are they any other quirks to be aware of? Let’s do a few more tests. What about the letter case of identifiers and reserved words?

select organization_id, organization_name, industry, number_of_employees from organization enable(return_range 1 10 'organization_name asc')
...
10 rows affected)

select ORGANIZATION_ID, ORGANIZATION_NAME, INDUSTRY, NUMBER_OF_EMPLOYEES from ORGANIZATION enable(return_range 1 10 'ORGANIZATION_ID asc', GENERATE_SQL_ONLY)
generate_sql                                                                                                                                                                                                                                                                                                                                                                                                                                                                
------------
select * from ( select dm_inline_view_1.*, ROW_NUMBER() OVER(ORDER BY dm_inline_view_1.ORGANIZATION_ID asc ) as dm_rnum from (select distinct organization.ORGANIZATION_ID, organization.ORGANIZATION_NAME, organization.INDUSTRY, organization.NUMBER_OF_EMPLOYEES from organization_sp  organization where (organization.i_has_folder = 1 and organization.i_is_deleted = 0) ) dm_inline_view_1) dm_inline_view_2 where dm_rnum >= 1   AND dm_rnum <= 10 order by dm_rnum 
(1 row affected)

The generated SQL is case-sensitive and handled to the underlying RDBMS, which processes it as expected.

Let’s test the lower range limit:

?,c,select organization_id, organization_name, industry, number_of_employees from organization enable(RETURN_RANGE 0 10 'organization_name ASC')
[DM_QUERY2_E_UNRECOGNIZED_HINT]error:  "RETURN_RANGE is an unknown hint or is being used incorrectly."

The accepted range is 1-based and starting at 0 yields an error. This is a DQL thing because Oracle does not complain here.

What about setting the position of the sliding window outside the available range?

?,c,select organization_id, organization_name, industry, number_of_employees from organization enable(RETURN_RANGE 1000 2000 'organization_name ASC', GENERATE_SQL_ONLY)
...
(0 rows affected)

Specifying a range outside the result set is not a problem and is processed as expected, i.e. nothing gets returned.

Is the hint compatible with other SELECT’s clauses ? Let’s test it with an aggregate function.

select organization_name, count(*) as cc from organization group by organization_name enable(RETURN_RANGE 1 10 'organization_name ASC')
organization_name             cc                      dm_rnum               
----------------------------  ----------------------  ----------------------
Allen, Ortega and Herman                           1                       1
Alvarado and Sons                                  1                       2
Anderson PLC                                       1                       3
Anderson, Watkins and Hansen                       1                       4
Andrews Group                                      1                       5
Anthony-Moyer                                      1                       6
Ball PLC                                           1                       7
Banks PLC                                          1                       8
Banks, Garcia and Wilkins                          1                       9
Barron, Dennis and Hill                            1                      10
(10 rows affected)

It works fine, just make sure that the attribute in the sorting_clause is also referenced in the list of attributes (organization_name here).

Does it also work using in-line views?

select org_name, cc from (select organization_name as org_name, count(*) as cc from organization group by organization_name) enable(RETURN_RANGE 1 10 'org_name ASC')
organization_name             cc                      dm_rnum               
----------------------------  ----------------------  ----------------------
Allen, Ortega and Herman                           1                       1
Alvarado and Sons                                  1                       2
Anderson PLC                                       1                       3
Anderson, Watkins and Hansen                       1                       4
Andrews Group                                      1                       5
Anthony-Moyer                                      1                       6
Ball PLC                                           1                       7
Banks PLC                                          1                       8
Banks, Garcia and Wilkins                          1                       9
Barron, Dennis and Hill                            1                      10
(10 rows affected)

It works fine here too.

Paginating programmatically

Implementing a programmatic way to paginate is is quite easy, although possibly not that efficient as it may put more stress on the system memory and/or no be as fast if a query is to be executed multiple times. Here are a few ways to do it:

Reusing the generated SQL

The most obvious implementation is to directly use the generated SQL query ! Now that we now how the RETURN_RANGE hint was implemented, we can use it directly, example:
Get the SQL implementation:

select doc_index, subject, object_name, organization_id, organization_name, website, country, description, founded, industry, number_of_employees from organization enable(RETURN_RANGE 10 20 'doc_index asc', GENERATE_SQL_ONLY)
generate_sql
----------------------------------
select * from ( select dm_inline_view_1.*, ROW_NUMBER() OVER(ORDER BY dm_inline_view_1.doc_index asc ) as dm_rnum from (select distinct organization.doc_index, organization.subject, organization.object_name, organization.organization_id, organization.organization_name, organization.website, organization.country, organization.description, organization.founded, organization.industry, organization.number_of_employees from organization_sp  organization where (organization.i_has_folder = 1 and organization.i_is_deleted = 0) ) dm_inline_view_1) dm_inline_view_2 where dm_rnum >= 10   AND dm_rnum <= 20 order by dm_rnum 
(1 row affected)

Use it:

execute exec_sql with query = 'select * from ( select dm_inline_view_1.*, ROW_NUMBER() OVER(ORDER BY dm_inline_view_1.doc_index asc ) as dm_rnum from (select distinct organization.doc_index, organization.subject, organization.object_name, organization.organization_id, organization.organization_name, organization.website, organization.country, organization.description, organization.founded, organization.industry, organization.number_of_employees from organization_sp  organization where (organization.i_has_folder = 1 and organization.i_is_deleted = 0) ) dm_inline_view_1) dm_inline_view_2 where dm_rnum >= 10   AND dm_rnum <= 20 order by dm_rnum'
result      
------------
T           
(1 row affected)

What? No output. Actually, exec_sql is not the right call to use here as it is targeted at DDLs and DMLs, not for SELECTs. We must use a DQL SELECT statement but the generated SQL is not DQL, thus it won’t work. Let’s use a temporary SQL table instead:

execute exec_sql with query = 'create table tmp_sql as select * from ( select dm_inline_view_1.*, ROW_NUMBER() OVER(ORDER BY dm_inline_view_1.doc_index asc ) as dm_rnum from (select distinct organization.doc_index, organization.subject, organization.object_name, organization.organization_id, organization.organization_name, organization.website, organization.country, organization.description, organization.founded, organization.industry, organization.number_of_employees from organization_sp  organization where (organization.i_has_folder = 1 and organization.i_is_deleted = 0) ) dm_inline_view_1) dm_inline_view_2 where dm_rnum >= 10   AND dm_rnum <= 20 order by dm_rnum'
result      
------------
T           
(1 row affected)

And now, select from it:

select * from dm_dbo.tmp_sql order by doc_index
doc_index     subject                                                                              object_name                    organization_id  organization_name              website                             country                 description                                                                                         founded     industry                            number_of_employees  dm_rnum               
------------  -----------------------------------------------------------------------------------  -----------------------------  ---------------  -----------------------------  ----------------------------------  ----------------------  --------------------------------------------------------------------------------------------------  ----------  ----------------------------------  -------------------  ----------------------
          10  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Hampton Ltd                    jRZDRGrfgyWTpAn  Hampton Ltd                    https://www.gilbert.com/            Estonia                 Chance discussion call yeah. Measure entire season scene behind radio. Than machine ball material.  16-4-1975   Accounting                                        36020                      10
          11  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Hernandez, Smith and Mcintosh  QLpUVwnuZVdIoCw  Hernandez, Smith and Mcintosh  https://garcia-peterson.com/        Switzerland             Live hundred would let food. Else nice firm door still. Hair technology trial.                      28-4-2007   Supermarkets                                      11630                      11
          12  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Fernandez LLC                  mCdBZkIqunyExsA  Fernandez LLC                  https://clark-hernandez.com/        Ukraine                 Health paper child worry thus produce light. Get them hope garden show think.                       19-5-1997   Market Research                                   46500                      12
          13  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Stevens, Greer and Young       QzSMkppsZEbfJQF  Stevens, Greer and Young       https://www.fuller-moses.com/       Finland                 Mr discuss pretty after whose actually guy. Music pass movement still.                              23-12-1988  Public Safety                                      3500                      13
          14  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Zuniga and Sons                jVMjZmDdeUejohj  Zuniga and Sons                https://www.porter-richardson.net/  Gibraltar               Agent heart who which dog father. Find day eight her loss. Fine agency say player training.         11-10-1994  Information Services                                920                      14
          15  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Murphy Group                   zJDwYcXPAnmEvos  Murphy Group                   https://parks.com/                  Guyana                  Democratic room year nature fact century change. Me great fear able watch father compare.           2-7-1999    Textiles                                          44560                      15
          16  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Edwards, Howe and Munoz        ChLMutRVzUKnzXN  Edwards, Howe and Munoz        https://nichols-walsh.com/          Vietnam                 Hotel bring fire laugh really interview remember.                                                   21-4-2006   Aviation / Aerospace                              30370                      16
          17  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Bowers Inc                     uBmqyxuKEBUcGSg  Bowers Inc                     https://rivera.com/                 Libyan Arab Jamahiriya  Often than number story land particularly. Term sure surface court drop any crime.                  1-12-1981   Outsourcing / Offshoring                           7800                      17
          18  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Fisher-Walker                  NyMFDEmxlPertcI  Fisher-Walker                  https://estrada.com/                Iraq                    Today feeling amount. Interview take wide report. Fire price foot probably follow sort.             29-11-2016  Legislative Office                                28060                      18
          19  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Smith, Hawkins and Baker       GtqNebcnXThOIEz  Smith, Hawkins and Baker       https://wheeler.info/               Senegal                 Manager eight forward each word. Choice defense responsibility develop half tax prove.              6-7-2017    Recreational Facilities / Services                 7230                      19
          20  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Lynch PLC                      XBPvgaxcwUplVyX  Lynch PLC                      https://www.ferguson.com/           Mauritius               Career region along. School summer stock later item. Deal say seat customer.                        21-2-2013   Political Organization                            23000                      20
(11 rows affected)

This time, it is OK.

Set-like operations

Intuitively, the following 11-document select statement:

select doc_index, subject, object_name, organization_id, organization_name, website, country, description, founded, industry, number_of_employees from organization enable(RETURN_RANGE 10 20 'doc_index asc')

is equivalent to:

select doc_index, subject, object_name, organization_id, organization_name, website, country, description, founded, industry, number_of_employees from organization enable(RETURN_TOP 20)
MINUS
select doc_index, subject, object_name, organization_id, organization_name, website, country, description, founded, industry, number_of_employees from organization enable(RETURN_TOP 9)

Unfortunately, there is no diff/MINUS statement in DQL, so this is not possible.
We could try the equivalent not in subquery clause, something like:

select doc_index, subject, object_name, organization_id, organization_name, website, country, description, founded, industry, number_of_employees from organization o where o.doc_index not in (select doc_index from organization enable(RETURN_TOP 10)) order by doc_index enable(RETURN_TOP 10)

but it does not work either; the reason is that although there are 2 RETURN_TOP hints, one in the subquery and one in the enclosing query, only one gets translated into the generated SQL:

select doc_index, subject, object_name, organization_id, organization_name, website, country, description, founded, industry, number_of_employees from organization o where o.doc_index not in (select doc_index from organization enable(RETURN_TOP 10)) order by doc_index enable(RETURN_TOP 10, GENERATE_SQL_ONLY)

select * from ( select distinct o.doc_index, o.subject, o.object_name, o.organization_id, o.organization_name, o.website, o.country, o.description, o.founded, o.industry, o.number_of_employees from organization_sp  o where (o.doc_index not in (select distinct organization.doc_index from organization_sp  organization where (organization.i_has_folder = 1 and organization.i_is_deleted = 0) )) and (o.i_has_folder = 1 and o.i_is_deleted = 0) order by o.doc_index ) where rownum <= 10 
(1 row affected)

and thus the query always returns an empty result. But using a temporary SQL table can help here too:

execute exec_sql with query = 'create table tmp_doc_index as select doc_index from organization_s where rownum < 10'
result      
------------
T           
(1 row affected)

and then select 11 documents:

select doc_index, subject, object_name, organization_id, organization_name, website, country, description, founded, industry, number_of_employees from organization o where doc_index not in (select doc_index from dm_dbo.tmp_doc_index) order by doc_index enable(RETURN_TOP 11)
doc_index     subject                                                                              object_name                    organization_id  organization_name              website                             country                 description                                                                                         founded     industry                            number_of_employees
------------  -----------------------------------------------------------------------------------  -----------------------------  ---------------  -----------------------------  ----------------------------------  ----------------------  --------------------------------------------------------------------------------------------------  ----------  ----------------------------------  -------------------
          10  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Hampton Ltd                    jRZDRGrfgyWTpAn  Hampton Ltd                    https://www.gilbert.com/            Estonia                 Chance discussion call yeah. Measure entire season scene behind radio. Than machine ball material.  16-4-1975   Accounting                                        36020
          11  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Hernandez, Smith and Mcintosh  QLpUVwnuZVdIoCw  Hernandez, Smith and Mcintosh  https://garcia-peterson.com/        Switzerland             Live hundred would let food. Else nice firm door still. Hair technology trial.                      28-4-2007   Supermarkets                                      11630
          12  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Fernandez LLC                  mCdBZkIqunyExsA  Fernandez LLC                  https://clark-hernandez.com/        Ukraine                 Health paper child worry thus produce light. Get them hope garden show think.                       19-5-1997   Market Research                                   46500
          13  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Stevens, Greer and Young       QzSMkppsZEbfJQF  Stevens, Greer and Young       https://www.fuller-moses.com/       Finland                 Mr discuss pretty after whose actually guy. Music pass movement still.                              23-12-1988  Public Safety                                      3500
          14  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Zuniga and Sons                jVMjZmDdeUejohj  Zuniga and Sons                https://www.porter-richardson.net/  Gibraltar               Agent heart who which dog father. Find day eight her loss. Fine agency say player training.         11-10-1994  Information Services                                920
          15  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Murphy Group                   zJDwYcXPAnmEvos  Murphy Group                   https://parks.com/                  Guyana                  Democratic room year nature fact century change. Me great fear able watch father compare.           2-7-1999    Textiles                                          44560
          16  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Edwards, Howe and Munoz        ChLMutRVzUKnzXN  Edwards, Howe and Munoz        https://nichols-walsh.com/          Vietnam                 Hotel bring fire laugh really interview remember.                                                   21-4-2006   Aviation / Aerospace                              30370
          17  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Bowers Inc                     uBmqyxuKEBUcGSg  Bowers Inc                     https://rivera.com/                 Libyan Arab Jamahiriya  Often than number story land particularly. Term sure surface court drop any crime.                  1-12-1981   Outsourcing / Offshoring                           7800
          18  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Fisher-Walker                  NyMFDEmxlPertcI  Fisher-Walker                  https://estrada.com/                Iraq                    Today feeling amount. Interview take wide report. Fire price foot probably follow sort.             29-11-2016  Legislative Office                                28060
          19  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Smith, Hawkins and Baker       GtqNebcnXThOIEz  Smith, Hawkins and Baker       https://wheeler.info/               Senegal                 Manager eight forward each word. Choice defense responsibility develop half tax prove.              6-7-2017    Recreational Facilities / Services                 7230
          20  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Lynch PLC                      XBPvgaxcwUplVyX  Lynch PLC                      https://www.ferguson.com/           Mauritius               Career region along. School summer stock later item. Deal say seat customer.                        21-2-2013   Political Organization                            23000
(11 rows affected)

So, this trick requires a 2-step operation:

1. firstly, select the identifiant of the N rows to skip from the doctype’s SQL table T_s, where N comes from RETURN_RANGE N + 1 M, into a SQL table;
2. secondly, select all the needed attributes from the doctype T and set the RETURN_TOP hint to M - N + 1;

To make it more robust, add an appropriate sort clause in the SQL table creation statement.

Programmatic alternatives

Such alternatives depend on the programming language in use and require that said language be interfaced to Documentum dmapi run-time library or the java Documentum Foundation Classes (DFCs). In several articles in this blog (cf. Adding a Documentum Extension to gawk, Adding a Documentum extension into python, A few scripting languages for Documentum, DctmAPI.awk revisited), we already presented bindings for gawk, perl and python along with JVM-based languages using the DFCs such as java, jython and groovy. Generally speaking, a programmatically implemented pagination can be done several ways, e.g.:

• Read the entire result set at once into memory, e.g. an array or dictionary, and navigate inside that data structure.
• Use the best features provided by the language to minimize memory imprint so the result set gets cached while being loaded piecewise. For example, python’s generators allows just that; they can be used to cache and return one page of rows at a time during forward pagination. While paginating backwards, the rows can be fetched from the cache and returned, cf. DctmAPI.py revisited for an example. Python also has decorators and it is possible to define one that will invisibly and unobtrusively manage the cache while fetching the result set.
• etc…

There are also numerous scripted solutions, i.e. that don’t necessitate Documentum libraries but rely on external utilities, e.g. Enhancing idql/iapi with rlwrap. To conclude this article, let’s show a very simple alternative that uses the less pager command to navigate. The result set come from Documentum’s standard command-line tool, idql. While we are at it, let’s include the preceding compact.awk script for a better display:

echo "select doc_index, subject, object_name, organization_id, organization_name, website, country, description, founded, industry, number_of_employees from organization o where doc_index not in (select doc_index from dm_dbo.tmp_doc_index) order by doc_index" | idql dmtest73 -Udmadmin -Pdmadmin | gawk -f compact.awk | less
API> doc_ind       subject                                                                              object_name                       organizati  _id  organization_name                 website                              country                                       description                                                                                          founded              industry                                             number_of_empl
------------  -----------------------------------------------------------------------------------  --------------------------------  ---------------  --------------------------------  -----------------------------------  --------------------------------------------  ---------------------------------------------------------------------------------------------------  -------------------  ---------------------------------------------------  -------------------
          10  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Hampton Ltd                       jRZDRGrfgyWTpAn  Hampton Ltd                       https://www.gilbert.com/             Estonia                                       Chance discussion call yeah. Measure entire season scene behind radio. Than machine ball material.   4/16/1975 00:00:00   Accounting                                                         36020
          11  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Hernandez, Smith and Mcintosh     QLpUVwnuZVdIoCw  Hernandez, Smith and Mcintosh     https://garcia-peterson.com/         Switzerland                                   Live hundred would let food. Else nice firm door still. Hair technology trial.                       4/28/2007 00:00:00   Supermarkets                                                       11630
          12  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Fernandez LLC                     mCdBZkIqunyExsA  Fernandez LLC                     https://clark-hernandez.com/         Ukraine                                       Health paper child worry thus produce light. Get them hope garden show think.                        5/19/1997 00:00:00   Market Research                                                    46500
          13  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Stevens, Greer and Young          QzSMkppsZEbfJQF  Stevens, Greer and Young          https://www.fuller-moses.com/        Finland                                       Mr discuss pretty after whose actually guy. Music pass movement still.                               12/23/1988 00:00:00  Public Safety                                                       3500
          14  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Zuniga and Sons                   jVMjZmDdeUejohj  Zuniga and Sons                   https://www.porter-richardson.net/   Gibraltar                                     Agent heart who which dog father. Find day eight her loss. Fine agency say player training.          10/11/1994 00:00:00  Information Services                                                 920
          15  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Murphy Group                      zJDwYcXPAnmEvos  Murphy Group                      https://parks.com/                   Guyana                                        Democratic room year nature fact century change. Me great fear able watch father compare.            7/2/1999 00:00:00    Textiles                                                           44560
          16  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Edwards, Howe and Munoz           ChLMutRVzUKnzXN  Edwards, Howe and Munoz           https://nichols-walsh.com/           Vietnam                                       Hotel bring fire laugh really interview remember.                                                    4/21/2006 00:00:00   Aviation / Aerospace                                               30370
          17  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Bowers Inc                        uBmqyxuKEBUcGSg  Bowers Inc                        https://rivera.com/                  Libyan Arab Jamahiriya                        Often than number story land particularly. Term sure surface court drop any crime.                   12/1/1981 00:00:00   Outsourcing / Offshoring                                            7800
          18  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Fisher-Walker                     NyMFDEmxlPertcI  Fisher-Walker                     https://estrada.com/                 Iraq                                          Today feeling amount. Interview take wide report. Fire price foot probably follow sort.              11/29/2016 00:00:00  Legislative Office                                                 28060
          19  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Smith, Hawkins and Baker          GtqNebcnXThOIEz  Smith, Hawkins and Baker          https://wheeler.info/                Senegal                                       Manager eight forward each word. Choice defense responsibility develop half tax prove.               7/6/2017 00:00:00    Recreational Facilities / Services                                  7230
          20  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Lynch PLC                         XBPvgaxcwUplVyX  Lynch PLC                         https://www.ferguson.com/            Mauritius                                     Career region along. School summer stock later item. Deal say seat customer.                         2/21/2013 00:00:00   Political Organization                                             23000
          21  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Harris Inc                        ahvZGlBZLSIxFSU  Harris Inc                        https://www.porter-wells.info/       Sri Lanka                                     Many religious sit enjoy south technology money. Center month stop clearly. Home like carry set.     1/29/1997 00:00:00   Medical Equipment                                                  40240
...
         199  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Hancock, Turner and Robinson      swilppvRqeWoIha  Hancock, Turner and Robinson      https://www.davila-collins.biz/      Sri Lanka                                     Star right effort modern six.                                                                        10/15/1980 00:00:00  Biotechnology / Greentech                                          20300
         200  Fake document for testing pagination in a result set through the RETURN_RANGE hint;  Spears Group                      zhLEEoIUUPmHqkF  Spears Group                      https://www.vasquez.info/            Uganda                                        Not staff success possible special. Exactly Congress local billion without fly staff expert.         11/4/2004 00:00:00   Food / Beverages                                                   22240
(191 rows affected)

Here, the result set is read into memory, page after page, as less requests them, until it gets entirely loaded in memory. Obviously, such an approach is quite awkward if the data are huge but for most interactive work, it is quite acceptable. Actually, the whole concept of paginating through a result set is mainly for interactive work, when data are displayed, paused to be scrutinized, and resumed. Therefore, it may be necessary to scope them as much as possible to reduce their size and also to speed up the query so the process gets swifter, especially if several iterations are necessary. We saw that the RETURN_RANGE hint has the optimize_top_row clause just for that need. In the above one-line, we would include the OPTIMIZE_TOP n to minimize the waiting time.

Conclusion

I’m afraid this is another TL;DR article. Anyway, several interesting takeaways were shown in this article:

1. Using fake data from a website from within a python program;
2. Generating fake data using python’s powerful Faker package;
3. Compacting a iapi/idql SELECT query output by removing each line’s trailing blanks so it reduces the likelihood of line wrapping for a better look on screen;
4. Of course, showing how to paginate through a result set using the RETURN_RANGE hint;
5. Putting together a simple one-liner to paginate a result set potentially fully contained in memory;

I hope that some of them may eventually be useful to the reader.

L’article Paginating Through a Documentum’s Result Set est apparu en premier sur dbi Blog.