SSL Enabled MSAD for EPM Systems

So, after a long hiatus , I am finally ready to update this blog on something that is bound to be catastrophic to your EPM environment. Microsoft is enforcing MSAD connections on SSL and its details can be found here.

My 11.1.2.4 environment MSAD connection was not SSL enabled. Per Oracle, implementing the ssl connectivity is a simple straightforward approach documented in KM Doc ID 1599610.1.

Per the document, You need to import your corporate AD SSL Certs into 3 locations inside the EPM system on all servers.

keytool -import -noprompt -trustcacerts -alias < Alias Name > -file < Certificate for Corporate
Directory > -keystore < Keystore Name and Location > -storepass < Keystore Passphrase >

However, When I did that, I ran into the error below.



SSL Error


Bind Error







































In order to overcome this, after multiple iterations, I have finally come up with the proper steps to get rid of this problem. If you installed the base version of EPM along with the base WebLogic and Java components, Those will need to be updated.




In my case, I had upgraded Weblogic to 10.3.6 per the instructions mentioned in Doc ID 2503568.1





bsu.cmd -install -patch_download_dir=D:\Oracle\Middleware\utils\bsu\cache_dir -patchlist=7HKN - prod_dir=D:\Oracle\Middleware\wlserver_10.3 -log=7HKN.log









The second and by far the most complicated step was to upgrade JDK and JRockIt to Java7. The steps are mentioned in the Doc ID 2351499.1. I chose to move ahead with the less complicated steps in Choice 1. It was not worth going through the struggle from Choice 2. 







Special Note- Don't forget Essbase, EAS and FR Studio Clients. The steps are a little different than that for Web Apps like Foundation, RAF , Web Analysis,Calc Man etc.







Lastly, Don't forget to update the Corporate SSL Certs on the New Java 7 library. Then,proceed to apply the SSL enabled option on your Shared Services console. It should now happily take port 636 and you will keep your security team happy. 




Happy EPMing!

Active-Active Disaster Recovery

In my organization, we are very big on DR site being available . In the event of any outage in the datacenter, the failover and failback processes are required to be nearly seamless. From the end user perspective, It is important for them to launch the same link for Smartview and be able to connect to Essbase, Reports, Planning.

In order to achieve this, we need the following:

Automatic DNS redirection

Any redirection tool of your choice can be implemented in your organization. In my case, both my production and DR VIPs are pointed to a separate dns url which gets the heartbeat information and redirects traffic to the primary or secondary site based on the response it gets from the status file. To make this happen, one can simply rename the file and that will make sure the traffic is redirected to the site that is active.

Oracle Wallet should contain the certificate information

This step is pretty standard and the same approach that needs to be adopted to implement SSL in the first place. In short, copy the cer file over to the destination environment. In Oracle Wallet, Open and import the certificate and save it. Also, if you have a load balanced setup we can simply copy the cwallet.sso and the ewallet.p12 file to Oracle\Middleware\user_projects\epmsystem1\httpConfig\ohs\config\OHS\ohs_component\keystores\default

Update the ssl.conf,mod_wl_ohs.conf and httpd.conf files

In a nutshell, make the entries match the production server setup.

On httpd.conf

include "${ORACLE_INSTANCE}/config/${COMPONENT_TYPE}/${COMPONENT_NAME}/admin.conf"
include "${ORACLE_INSTANCE}/config/${COMPONENT_TYPE}/${COMPONENT_NAME}/mod_wl_ohs.conf"

include "moduleconf/*.conf"
include "conf/ssl.conf"
Include "conf/epm_online_help.conf"
Include "conf/epm_rewrite_rules.conf"
Include "conf/epm.conf"
Include "conf/epm_online_help.conf"
Include "conf/epm_rewrite_rules.conf"
Include "conf/epm.conf"

On ssl.conf


NameVirtualHost *:443
   
        ServerName example.hyperion.com
        RewriteEngine On

Restart OHS with these changes populated. The redirection must happen now.

Calculation Manager Rules that are still on Planning don't show up on Calculation Manager

In my 11.1.2.3.500 setup, I recently faced an issue with Calculation Manager Rules that were present on Planning forms inside the Planning Application, but however, on expanding the Calculation Manager->Planning App->Plan Type->Rules, it did not display the rules inside. These rules were imported manually earlier and suddenly disappeared. I could not really find what caused it but here are the steps to recover the BR's from planning tables to calculation manager:

1. Access the appropriate Planning application repository and its table HSP_CALC_MGR_RULES

2. Create a new text file with following content

3. Copy the contents in the Body column of the HSP_CALC_MGR_RULES table and paste its contents between the tag in the file created in step 2.
4. Save file as XML file
5. Within the Calculation Manager navigate to “File” -> “Import…” to import the Business Rule into Calculation Manager .

Using this I could restore all the rules back.

"Error executing query: -1" When Running Financial Reporting (FR) Reports

When we performed the migration to 11.1.2.3 since it was a new install some of the reports failed with the error "Error executing query:-1" .In addition to this a lot of the missing values were also being generated as a result. To resolve this issue, We had applied a fix in the previous environment as this issue seemed to exist there as well. We found out that this was due to the MDX query setting on FR Server and it was set to the default 'True' value. To carry out the same, follow the steps below.

1. Navigate to C:\Oracle\Middleware\EPMSystem11R1\products\financialreporting\bin and launch FRConfig.cmd

In some instances, this utility doesn't open and throws a java error. To fix that, follow step 2 below.

2. Edit setInstance.cmd and add the

set EPM_ORACLE_INSTANCE=C:\Oracle\Middleware\user_projects\epmsystem1

3. Now, it opens the FRConfig.cmd. In the MBeans tab, expand the Financial Reporting tab and scroll to the EssbaseUseMDX setting.

4. Restart FR services. This should fix the issue. 

I had found that on a parallel environment since it had loopback adapters present for the VIP, It did not work out for me. In that case, launch the enterprise manager module and correct it from there.Click on System MBean Browser.

Navigate to the Application MBeans

Also a side note, PreviewUserPOV can be set to True, so that end users don't have to set their FR Preferences to On. We can make the change on FRStudio.cmd

BI+ distributed setup "startup document does not exist " "GSM unable to acquire LSM"

We have a network deployment for Foundation and BI+ in our environment and the 2nd node when operated on its own stopped being accessible. RAF agent service kept going to "stopping" mode when stopped and not completely stopped. The RAF Agent service and the Foundation Service was set to the installer account.

I thought it will be a good idea to compile a few things I verified in the hopes that it will help anyone with some troubleshooting ideas.

Ensure to set HyS9RaFrameworkAgent_epmsystem1 service to start with Domain account instead of Local System on both RAF servers and also make sure the domain account has local administrator rights .

Before starting services on Server2 make sure server1 is up and running and Workspace is accessible

Verify that HAVA ID in EPM registry matched in css_provisioning_info table

1: Execute the following query against the Shared Services registry database:
2: select application_id from css_provisioning_info where lower(application_id) like 'hava%';
o note this application id, e.g. HAVA:0000012b3072bf63-0000-6717-0acc1bac

3: Run epmsys_registry.bat from \Oracle\Middleware\user_projects\epmsystem1\bin
Open EPM registry.xml and search for HAVA and compare the HAVA ID using the ID above

4: If there is a mismatch, edit the Shared Services Registry:
Go to EPM_ORACLE_INSTANCE/bin and run the following command:
epmsys_registry updateproperty RA_FRAMEWORK/@applicationId

Where applicationId is the id returned from the query above. For example, using the id above:

epmsys_registry updateproperty RA_FRAMEWORK/@applicationId HAVA: 0000012b3072bf63-0000-6717-0acc1bac

5. Restart all services on Server1 and ensure Shared Services and Workspace are accessible

6: Enable the Service Broker service and the GSM on the second server
- Log on to EPM Workspace on Server1
- Select Navigate, then Reporting and Analysis, and then Services.
- Right-click Reporting and Analysis for server2 , and then select Properties.
- On the Services tab, set GSM to Enabled (Yes) and Service Broker to Enable (Yes).

Reproduce the issue and attach the logs from these locations from both servers 1/2
- \Oracle\Middleware\user_projects\epmsystem1\diagnostics\logs\ReportingAnalysis

and run epmsys_registry.bat from \Oracle\Middleware\user_projects\epmsystem1\bin
For my issue however, everything seemed to tie up quite nicely and I was still running into the issue.
This was the log entry from the server 2  raframework_stdout_console.log when started on its own:

Finally, when I started digging a little deeper, I could understand what was causing this to happen.
While Configuring the RAF for the 2nd node, We normally point to the shared server 1 RM1 directory while configuring.

The second Workspace agent was not able to locate the RM1 directory on server 1 .An issue with DNS could not resolve the server 1 hostname correctly.On second Workspace Agent server, add the server 1 hostname/IP address into the HOSTS file

On server 2, update the hosts file (which is under C:/Windows/System32/etc/drivers/hosts folder) with the ipaddress and the hostname of server1 where RM1 directory stored.
xx.xx.xx.xx server1.domain.com

This helped in resolving the issue.

.

Merge Cells Issue on Workspace Report

We have users that run the reporting out of Workspace and then export into Excel. Historically , The reports were formatted the same between Workspace and Smart View so you could easily pull from either and end up with consistent reporting.

Since the upgrade to 11.1.2.3, It appeared the new Workspace reports are no longer formatted the same. They had merged cells.


To fix the issue,

1. On your FR server, navigate to Oracle\Middleware\user_projects\epmsystem1\FinancialReporting\lib\FOProcessor
2. Save a copy of the existing xdo.cfg file.
3. Open xdo.cfg in a text editor.
4. Toward the top of the file, locate this line:

5. Immediately below it (above the
tag), add this line:


6. Save the file.
7. Restart the FR Web service (wait a few minutes to allow the service to fully render).

Calculation Manager Rules Runtime Tuning

We have had a case in 11.1.2.3 Calculation Manager where I have been trying to run the business rule, running them at the Total level in the 11.1.2.3 and I had to kill the calc ran after it ran for 5 hours.

In the process of debugging,I tried at a lower level to see if at least I can get it to complete. I did test the calc at a single intersection and it ran fine in seconds but that really wasn't much good to us
as our users all enter multiple intersections through their ad hoc forms and then need to calc at an aggregate level.

While starting to dig the rootcause , I thought it might be a good idea to share what was done to tune this rule. I started with the Cube cache settings.I have shown the values above.This needs to be followed with a Essbase restart.
Also for the application, I added these two properties . In order to do this, I logged into the planning application, Administration->Application->Properties. There, I added two properties called HBR_MAX_WAIT_FOR_RULE and HBR_MONITOR_DELAY and set them to 480000 which is 8 minutes.

Post this, I carried out a complete system restart and then ran the rule. Now what took a long time was completed in 5 minutes which is a big win. 

Hyperion SmartView Tuning

While driving the 11.1.2.2 to 11.1.2.3 upgrade in my organization, I have run into various problems to which some of my earlier posts are a testament.  Today, I bring to you one from the SmartView trove. We have users who swear by Excel Addin and refuse to let go of it. There have been some instances where users are unable to retrieve large numbers on SmartView.
This was the message that was produced when the retrieve was run on private connection using 19000 url .So, It would be something like, http://ohs.servername.com:19000/aps/SmartView .

 On using the SSL link for the same, This was the error message I encountered,

                                                                                                                                                                  

Listed below are a few ways in which the tuning can help in resolving the issue. First , I would tackle the browser timeout settings by the steps shown below.

On the client machine, update/add the following registry keys:

1. Open the Registry, Start -> Run -> Regedit.

2. Locate the following section(s):

[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings]\

or

[HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Internet Settings]

3.Create the following new DWORD keys with decimal values:

ReceiveTimeout=480000
KeepAliveTimeout=180000
ServerInfoTimeout=180000

In this example, the ReceiveTimeout setting is 8 minutes. The KeepAliveTimeout and ServerInfoTimeout settings are 3 minutes. Set these to greater than the longest running request.

4. Restart the machine for the new settings to take effect.

This in my case did not seem to help. There are some more tuning parameters we can add to aps.properties file located at C:\Oracle\Middleware\EPMSystem11R1\products\Essbase\aps\bin

# Specifies values for Essbase Server network connection
# APS will try netConnectRetry times to connect to an Essbase Server.
olap.server.netRetryCount=6000
olap.server.netConnectRetry=30
olap.server.netDelay=2000
olap.server.netSocketTimeOut=20000
Also, these values need to be added to the C:\Oracle\Middleware\EPMSystem11R1\common\EssbaseJavaAPI\11.1.2.0\bin\essbase.properties  file and try making a private connection to  http://aps.servername.com:13080/aps/SmartView
Adding these values helped me resolve the large retrieve issue. This bodes well with IE9 in the case of 11.1.2.3 .

Another request that is normally made is to increase the max rows to 65000 . In order to make that change, modify the properties shown below.

# Specifies maximum rows/columns for dataquery grid. 0 value indicates no limit.
service.olap.dataQuery.grid.maxRows=65000
service.olap.dataQuery.grid.maxColumns=255
To complete the change, login to EAS and expand the provider server tree. Right click on the provider server and edit the properties .

Restart APS . That Resolves the increase row limit Issue.

An unexpected error has occured while migrating outlines on EAS

I have been trying to migrate Essbase Outlines on 11.1.2.3.503 and have noticed this error some times on EAS. 

On looking on the EASServer.log there is nothing related to these errors and the messages panel below the EAS console doesn't display any error messages either. 

I have noticed that this occurs on two instances:

1. When you are not "connected" to Server1 or Server 2 Explicitly on EAS , i.e. when server 1 or 2 is not expanded .

2. When the target server is lower than 11.1.2.3 and the source server is 11.1.2.3

In my case, I could continually reproduce the error, when I wasn't connected to either of the servers and it went away after expanding server1 and server2.

Loadbalanced EAS 11.1.2.3 throws Login Error

When I first installed 11.1.2.3 on a load balanced instance , I had many challenges. One of the components that has given me problems consistently has been the Essbase Administration Services, EAS . I have covered various posts on 11.1.2.3 EAS  here,here and here . I am adding one more to the collection with this post.

What happened was that each time I would login to EAS after a restart, it would be fine. But the subsequent problems always resulted in a login error.

I found these entries on the easserver.log

[EssbaseAdminServices0] [WARNING] [] [oracle.epm.essbase.eas] [tid: 12] [userId: ] [ecid: 0000KWxWr6H5IfYTTqxkWc1K1ssg000003,0] [APP: EAS#11.1.2.0] Source directory does not exists
[EssbaseAdminServices0] [WARNING] [] [oracle.epm.essbase.eas] [tid: 12] [userId: ] [ecid: 0000KWxWr6H5IfYTTqxkWc1K1ssg000003,0] [APP: EAS#11.1.2.0] Failed to copy storage directory to new location
[EssbaseAdminServices0] [ERROR] [ESSEAS-35505] [oracle.epm.essbase.eas] [tid: 11] [userId: ] [ecid: 0000KWxb0TM5IfYTTqxkWc1K1ssg00001a,0] [APP: EAS#11.1.2.0] Following Exception occurred for user admin. Message exception is EPMCSS-00301: Failed to authenticate user. Invalid credentials. Enter valid credentials.
[EssbaseAdminServices0] [ERROR] [ESSEAS-35505] [oracle.epm.essbase.eas] [tid: 11] [userId: ] [ecid: 0000KWy029F5IfYTTqxkWc1K1ssg000048,0] [APP: EAS#11.1.2.0] Following Exception occurred for user admin. Message exception is EPMCSS-00301: Failed to authenticate user. Invalid credentials. Enter valid credentials.
[ERROR] [ESSEAS-35505] [oracle.epm.essbase.eas] [tid: 11] [userId: ] [ecid: 0000KX1Ie7Y5IfYTTqxkWc1K286Q00000k,0] [APP: EAS#11.1.2.0] Following Exception occurred for user admin. Message exception is java.lang.NullPointerException

In the SharedServices_security_client.log , I saw the following error :

[EssbaseAdminServices0] [ERROR] [EPMCSS-09223] [oracle.EPMCSS.CSS] [tid: 12] [userId: ] [ecid: 0000KX1E_JR5IfYTTqxkWc1K286Q000002,0] [APP: EAS#11.1.2.0] [SRC_CLASS: com.hyperion.css.spi.CSSManager] [SRC_METHOD: updateDcUrl] Failed to retrieve base DNs while updating dcUrl. Error communicating with server. DNS name not found [response code 3]. Invalid Domain, AD Site or DNS Server.

This normally indicates there is an issue with the CSS configuration. It then dawned upon me that I had messed up my search order while configuring the MSAD. I placed my MSAD in the order 1 and the Native Directory in order 2. I reversed the order and followed it up with a complete EPM system restart. The issue was resolved.

Hyperion EAS stops accepting logins after a while

I have a Hyperion 11.1.2.2 instance which has been running for a while now. Since EAS is not used as much , I never knew of this issue until a user reported it. This user kept getting "Login Failed" and it kept some days after restarting the service. On some deeper analysis, I found that the rootcause of this issue was the location tmp folder for EAS .  We can apply the below fix to resolve the issue.

On the server where EAS is installed, navigate to the user_projects level.

Here we can find there is a folder called tmp. Create a new folder by some name, let's say eas_tmp

Go to the registry, and update the setting corresponding to Djava.io.tmpdir as shown below

Restart EAS Server. This error should now stop appearing.

Managed Servers don't talk to Admin Server

I have a network deployment for Hyperion BI+,Calc Man, Planning and EAS where all these components are load balanced with the exception of Essbase and it is a distributed server install, 

I ran into an issue with Weblogic where it displayed the below symptoms:

1. Hyperion Planning, EAS, CalcMan, and other things not running on the Admin Server Machine take FOREVER to startup.

2. Once started, AdminServer still shows all the remote nodes as DOWN

3. Errors in the managed server sysout files show managed servers were trying to talk to the wrong IP address to get to the Admin Server

(it was trying to use the VIP address, and not the address associated with the FQDN for the server.

On digging further, I found that the servers are multihomed, mainly due to an odd load balancer setup where there are local loopback addresses on the

admin server that are really hosted by the VIP on the LB. The managed servers were able to make an initial connection to the

admin server, and the admin server responded back with the WRONG IP (the loopback, not the public IP).

The resolution is to set the LISTEN address on the WL AdminServer "Server" to the public DNS name. example.host.domain.com

For this, Login to the WL Console and click Lock & Edit (see Use the Change Center).

On the Servers page, click the name of the server , Select Configuration > General.On the Severs: Configuration: General page, enter a value in Listen Address.

Restart AdminServer,wait for it to be fully running, Restart all the other services. 

WebAnalysis SSL Link for 11.1.2.2 on a patched platform

When I first installed 11.1.2.2 , there wasn't a need for a WebAnalysis solution. Later, when there was a need , the install wasn't straightforward. Read my previous post on how to fix the installation problem here . I could get through with my installation but however, launching WebAnalysis on SSL became a problem. The 16000/19000 link would launch fine but not 443. I enabled debug trace on and saw these messages on the java console.

For some reason 16 jar files and jre-6u22-windows-i586-s.exe are missing from WebAnalysis.ear file.

These files are present in %HYPERION_HOME%\common\epmstatic\webanalysis folder where as the static folder in WebAnalysis.ear located under ' %HYPERION_HOME%\products\biplus\InstallableApps file doesn't have them.

Copy those files to the tmp directory of Weblogic server as seen below.

C:\Oracle\Middleware\user_projects\domains\EPMSystem\
servers\WebAnalysis0\tmp\servers\WebAnalysis0\tmp\_WL_user\
WEBANALYSIS_11.1.2.0\ts1hx4\war\static

If you are interested in a permenant work around/fix, then you can add the jar files and jre-6u22-windows-i586-s.exe to the WebAnalysis.war file.

Open WebAnalysis.ear file with 7Zip or Winrar
Open WebAnalysis.war file and navigate to static folder
Compare the file present in %HYPERION_HOME%\common\epmstatic\webanalysis against static folder and add missing ones.


BUG INFO:

Web Analysis Studio Unable to Launch When Using Default 16000 Port Error: "class com.hyperion.analyzer.usersession_base.AnalyzerClient not found." [ID 1365054.1]

It says "The ability to login to WA using the listening port:16000 has been removed by development.
This is documented in unpublished (documentation) Bug 13061357"

Assign Access Control View appears in different font on 11.1.2.3

On 11.1.2.3 base version,once you launch Foundation via workspace, expand any Essbase Server and right click to assign access control, the view appears in a completely different font . It reminds me what it used to look like prior to 11.1.1.3.

This is a published Bug 17354487: ASSIGN ACCESS CONTROL BUTTON ICONS MISSING and the solution is to copy “ui_themes” folder from

  C:\Oracle\Middleware\user_projects\domains\EPMSystem\servers\FoundationServices0\tmp\_WL_user\SHAREDSERVICES_11.1.2.0\yidcyl\war

 To:

  C:\Oracle\Middleware\user_projects\domains\EPMSystem\servers\FoundationServices0\tmp\_WL_user\WORKSPACE_11.1.2.0\340u89\war

Stop and start foundation services for the change to be updated.

This problem will then be resolved. 

WebAnalysis on 11.1.2.3 doesn't launch SSL URL

When I installed SSL for all my applications on 11.1.2.3, The link for WebAnalysis Studio did not launch . It started the JRE but failed to launch the logon screen. In an attempt to debug the issue, I opened the java console and enabled logging. I saw the below exception there:

The link worked on 19000 but not on 443. The reason behind this is that rewrite rule was not present in my httpd.conf and adding the entry below in this section helped me resolve the issue.

RewriteRule ^/WebAnalysis/static/(.*) /epmstatic/webanalysis/$1 [PT]

So , the complete section should look like this when done.

RewriteEngine On

RewriteRule ^/interop/bpmstatic/(.*) /epmstatic/bpmui/$1 [PT]

RewriteRule ^/raframework/bpmstatic/(.*) /epmstatic/bpmui/$1 [PT]

RewriteRule ^/workspace/static/(.*) /epmstatic/wspace/$1 [PT]

RewriteRule ^/workspace/bpmstatic/(.*) /epmstatic/bpmui/$1 [PT]

RewriteRule ^/HyperionPlanning/help/(.*) /epmstatic/planning/docs/$1 [PT]

RewriteRule ^/calcmgr/bpmstatic/(.*) /epmstatic/bpmui/$1 [PT]

RewriteRule ^/awb/bpmstatic/(.*) /epmstatic/bpmui/$1 [PT]

RewriteRule ^/WebAnalysis/static/(.*) /epmstatic/webanalysis/$1 [PT]

This fix is explained in more detail in Doc ID 1354981.1

FR Icons missing in BI+ Workspace

When I installed 11.1.2.3, FR was my choice of reporting tool and I noticed something quite odd on the User Interface. The icons for FR were not visible and I would attach a screenshot, But I forgot to take one and I resolved the issue so the issue no longer is there. I have installed quite a few instances and this behavior has not been consistent. In any case, the resolution for it is below. Looks like there is a KM for this for 11.1.2.1 Doc ID 1368253.1 .

• Go to the C:\Oracle\Middleware\EPMSystem11R1\products\biplus\InstallableApps directory 
• Using 7-zip (http://www.7-zip.org), open the raframework.ear file 
• Using 7-zip, open the raframework.war file from within the raframework.ear file 
• Using 7-zip, open the WEB-INF directory from within the raframework.war file 
• Using 7-zip, open the weblogic.xml file from within the WEB-INF directory to modify the file 
• The tag may contain a path that looks like @oracleInstance@/ReportingAnalysis/RAFrameworkWebapp 
• Change so it will look like C:/Oracle/Middleware/user_projects/epmsystem1/ReportingAnalysis/RAFrameworkWebapp 
• Save the file and update the weblogic.xml file in the archive when prompted. 
• Restart the RAF webapp service 

This fix helps resolve the issue. 

Load balanced 2nd node for Hyperion BI+ doesn't work independently

While building 11.1.2.3 from the  ground up, I have run into a string of issues. I will continue to post on all the issues but these tips can help someone who may be in a similar situation and some things to think about while troubleshooting them.

I have every web application load balanced on windows 2008 r2 64 bit . I began testing as soon as I completed configuring everything and started to test indepently to see if node 2 continues to run when node1 is completely down because in an actual DR situation these things matter.Thats when I ran into this message for the first time.

"The Startup Document does not exist in the repository" followed by "GSM_unabletoacquire LSM"

On a network deployment there are several reasons that can cause this:

1. The service account for the HyS9RaFrameworkAgent_epmsystem is not the same as the installer but the "local system" account. This is an absolute must since RM1 is a shared folder across 2 servers.

2. The Service Broker service and the GSM on the second server is not enabled.
To Resolve this,
- Log on to EPM Workspace on Server1
- Select Navigate, then Reporting and Analysis, and then Services.
- Right-click Reporting and Analysis for server2 , and then select Properties.
- On the Services tab, set GSM to Enabled (Yes) and Service Broker to Enable (Yes).

3. Verify that HAVA ID in EPM registry matched in css_provisioning_info table
a: Execute the following query against the Shared Services registry database:
b: select application_id from css_provisioning_info where lower(application_id) like 'hava%';
 note this application id, e.g. HAVA:0000000b0000bf00-0000-0000-0acc1bac
c: Run epmsys_registry.bat from \Oracle\Middleware\user_projects\epmsystem1\bin
Open EPM registry.xml and search for HAVA and compare the HAVA ID using the ID above

d: If there is a mismatch, edit the Shared Services Registry:
Go to EPM_ORACLE_INSTANCE/bin and run the following command:
epmsys_registry updateproperty RA_FRAMEWORK/@applicationId

Where applicationId is the id returned from the query above. For example, using the id above:

epmsys_registry updateproperty RA_FRAMEWORK/@applicationId HAVA: 0000000b0000bf00-0000-0000-0acc1bac

e. Restart all services on Server1 and ensure Shared Services and Workspace are accessible

However, in my case, all of these seemed perfect. I stumbled upon my resolution by chance . The Location for the RM1 shared folder (in the configurator, Eg: \\server1\c$\Oracle\Middleware\user_projects\epmsystem1\ReportingAnalysis\data\RM1 stopped being recognized from server 2 . 

The way in which server 1 is resolved from server 2 turned out to be the root cause of my issue.It used to be accessible by the short name earlier and suddenly started to accept only the FQDN owing to some DNS related changes.  In my opinion, We should go with the same servername that we issue in the host file as part of the preinstall. To override that, I added an entry on the server 2 host file and it started working.

EAS SSL Link offloaded at OHS throws a jnlp file

I am on 11.1.2.3.503 and the SSL link for EAS presents me with a jnlp file and it works fine with the 19000 link .

But when you try to access the SSL link, it throws a " administration server does not exist" error.

While this has been discussed in plenty in various KMs DocID1346806.1 . This fix worked perfectly on 11.1.2.2 but on 11.1.2.3 , There is one additional fix that the KM doesn't cover.


The solution is to modify the link wherever you see a reference to the old link, modify that reference to include the 443 port.

Example of these entries are below:

       
       



Now if you save a local copy of these files, if you double click the jnlp it launches the SSL console and you don't see the error anymore.

SSL offloaded at OHS results in smartview RAF not displaying any reports

On a SSL implemented 11.1.2.3 system, which is offloaded at OHS , there is a problem with displaying the list of reports inside the cname. This is a really weird problem as the link works on 19000 .

The end result might look something like this.

According to the DocID1364770.1 , Applying the fix for the step 1 and 2 should fix the issue. But it doesn't , in my case.

Step 1:

• Configure Weblogic to look for specific HTTP headers, amongst which the "WL-Proxy-SSL" header:

1. Start the Weblogic Administration server
2. Log in to the Weblogic Administration Console, then put the Weblogic console into lock/edit mode
3. On the left-hand side, select the top level branch named "epm_system" (i.e. the domain name)
4. Select "Web Applications" on the right hand side
5. Scroll down and enable the option named "Weblogic Plugin Enabled"
6. Save and activate the changes.

Step 2:

• Configure the OHS component to populate the requests with the "WL-Proxy-SSL" HTTP header:

1. On all servers hosting the OHS component, edit the file called httpd.conf (located in \Middleware\user_projects\epmsystem1\httpConfig\ohs\config\OHS\ohs_component) and append the following line at the bottom:
2. WLProxySSL ON
3. Save and close that file.

To resolve the issue, I applied the below fix:

Backup Shared Services repository database 

1. On the server that hosts the Shared Services Registry, open a command prompt. 

2. Go to C:\Oracle\Middleware\user_projects\epmsystem1\bin and run the following commands: 

epmsys_registry addproperty system9/@enable_ssl_offloading true 

epmsys_registry addproperty system9/@external_url_host  (SSL_off_loader_host_name) 

epmsys_registry addproperty system9/@external_url_port 443 (SSL_off_loader_port) 

3: Restart Foundation, Workspace Agent, and Workspace web applications services 

This fixes the issue and the list can now be seen on Smartview

EAS Timeout Issues

I was trying to see fun ways to start working using a disconnected remote session and one thing that caught me in a web was the endless woes with EAS timeout . Each time I had to open a fresh session and it wasn't until recently I stumbled upon this document from the online help.

http://docs.oracle.com/cd/E40248_01/epm.1112/eas_help/frameset.htm?timeout.html  is the link the document.

To fix this issue, We need to unzip the EPM_ORACLE_HOME\products\Essbase\eas\server\AppServer\InstallableApps\Common\eas.ear file

The document encourages us to edit the zip file seperately. However, we can achieve the same using the 'open archive' option in 7zip.

Once this is open , proceed to open the eas.war file

Navigate to the web.xml file present on WEB-INF tree.

Open the file and scroll to the session-timeout section. The default value is 45 and you can change it to the value appropriate to your setup .  In my case , I set it to 180.

The link does not specify any restart required but it is always better to do it. Post this, you should be able to stay logged in to EAS longer.

PDF preview for Hyperion Financial Reports fails to load

I had an issue once with the PDF preview generation for a FR Report on 11.1.2.2.303. It seemed to affect one out of the two servers which had Financial Reporting deployed on it. The html preview turned out fine but the pdf preview was affected. Also, what was strange was it seemed to affect one server and not the other even though they were on the same patched version.

The resultant report produced a view like this,

The sysout log for FR had references to the errors IOException has occured in FOProcessor.setData(ByteArrayInputStream)

There is a KM document 1564844.1 which details 3 methods to resolve the issue. I did not go with the first option as it involved rolling back a patch and the .303 patch contained some important fixes related to my Financial Reporting setup.  I , however,  followed the third method of manually editing the xdo.cfg file. The steps are mentioned below:


1. Log onto the FR Web App Server
2. Navigate to Oracle\Middleware\EPMSystem11R1\products\financialreporting\lib\FOProcessor
3. Backup the existing xdo.cfg file, then delete or rename the file
4. Open the original xdo.cfg file in a text editor
5. Locate the line containing the text  "<property name="system-temp-dir">c:\temp</property>"
6. Comment the line out using XML comment syntax: <!-- <property name="system-temp-dir">c:temp</property> -->   
7. Repeat the above steps on all FR Servers
8. Restart all affected FR Web App services

After this, I have noticed that the preview is generated without any issues.

Weblogic changes cannot be activated - Timeout Error

For a few months now, I have been dealing with a lot of WebLogic 10.3.6 related frustration because I was running into an issue with updating one setting and each time I tried to activate the changes I used to run into an issue trying to activate the changes .

I have a deployment with 4 Managed Servers each of which host 4 Hyperion services load balanced across 4 VMs. These were the preliminary steps I worked with Oracle on. So, I thought these are some of the steps that we can try on our own and help accelerate the resolution process.

1. Shutdown the weblogic domain admin console then move those files out of the Pending folder and then restart the admin console and try to activate the changes again.

2. Navigate to /pending folder. In this folder, there are configuration files which are temporary config files which are cached with changes made from console. At the time of activating changes, the config.xml from the pending folder replaces the original config.xml and other configuration files inside the /config folder.  Kill the Admin Server, and replace the config.xml from the /config folder with the config.xml from the /pending folder. 

Delete all the .lok files inside like edit.lok and config.lok.  Restart Admin Server. 

3. Flush down the changes to be activated using WLST. 

You can use the below script:

4. And of course the classic, Increase Logging level. This can be again perfomed on the config.xml file.

In my case however, none of these methods helped and each time I kept hitting a roadblock . The solution was to update the Listen-address on the config.xml and to repeat the steps for all the managed servers and the components deployed on the mananged server. It is important also to follow a specific order to carry the steps below ,out. 

a. Stop all the Hyperion Services and the Weblogic service.

b. In your admin server, navigate to where the pending directory lies in this case, X:\Oracle\Middleware\user_projects\domains\EPMSystem and rename pending to pending_bkp.

c. Modify your entry in the config.xml file is in the below syntax.

Change the  to include the IP/FQDN to reflect the new entry. So, Post the change,the entry becomes,

d.After this is done for all the managed server components deployed under Weblogic, Start the Weblogic Admin server and the Hyperion Application Services.

You should be able to activate any pending changes now.

CalcMan 11.1.2.2 does not produce updated view for rules on IE browser

I recently ran into this issue with my clients where on a load balanced setup on Windows 2008R2 Platform , The CalcMan rules that they created were at times visible immediately and at times it doesn't produce the updated results.

Initially , I thought it was a browser related issue so I suggested that they clear their browser cache and retry on some attempts it would work and fail some other times. On digging some more, I could isolate the behaviour to one server.On Stopping and Starting Services , I hit this snag on the diagnostics/services log.

java.rmi.MarshalException: failed to marshal create(Lweblogic.rmi.spi.HostID;ILweblogic.cluster.replication.ROID;Lweblogic.cluster.replication.Replicatable;); nested exception is:

java.io.NotSerializableException: com.hyperion.css.common.CSSProvisionedAppInfo

at weblogic.rjvm.BasicOutboundRequest.marshalArgs(BasicOutboundRequest.java:92)

at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:221)

at weblogic.cluster.replication.ReplicationManager_1036_WLStub.create(Unknown Source)

at sun.reflect.GeneratedMethodAccessor4706.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

Truncated. see log file for complete stacktrace

Caused By: java.io.NotSerializableException: com.hyperion.css.common.CSSProvisionedAppInfo

at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164)

at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)

at java.util.HashMap.writeObject(HashMap.java:1195)

at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

Truncated. see log file for complete stacktrace

>

The solution was  to add this entry to CalcMgrConfig.properties file on both the servers. The normal location for this file is ,

E:\Oracle\Middleware\user_projects\domains\EPMSystem\servers\CalcMgr0\tmp\servers\CalcMgr0\tmp\_WL_user\CALC_11.1.2.0\qxf61x\war\WEB-INF\conf

If the value is already present for DATABASE_CACHE_REFRESHRATE, Change it to 1000.

#Database Caching refresh rate in milliseconds

DATABASE_CACHE_REFRESHRATE=1000

Restart the Calculation Manager Services Post that. It should now produce an updated view on browsers.

Users's last login into the system for Essbase for v11.1.2.2

I am sure time and again various people have posted about how to obtain the user login information from EAS.  Users Last Logon Time post by Celvin provides a good deep dive into how this is achieved across versions.

One method to tune the query Date wise ,published by Celvin is to add ORDER BY "Last Login Date" to the end of the query. So Basically, at the end of the query it should look like,

SELECT username as "User Name", MAX(entrydate) AS "Last Login Date" 

FROM serverlogdetail where username is not null

GROUP BY username ORDER BY "Last Login Date";

However, this query doesn't serve the purpose in the 11.1.2.2 version. 

I researched a little and found the following setting yields the same result as the earlier versions on EAS.
This setting is supposed to populate the Last Login Time" column in the eas console under "Enterprise View > Administration Servers > Essbase Servers > [Hostname] > Security > Users

Add to esbase.cfg the setting: PERSISTUSERATLOGIN TRUE and then restart the Essbase service.

MSAD Settings in Hyperion and their Significance

While configuring Active Directory ( MSAD in this instance) there are quite a few parameters that we overlook and one such time while migrating from 11.1.1.3 to 11.1.2.2,  I used the exact same settings that my prior environment had  and once everything was configured ,while testing , I ran into this issue with users from across the globe got “Invalid login credentials” while logging into Essbase, Workspace or Planning and each time after restarting the entire suite it 

would start working back for a while and would loop back to the same state in some time . 
The logs indicated only numerous references to the Invalid credentials message.
Some good tools that serve as a starting point for investigating the root cause are LDAPBrowser, Wireshark that can help in understanding why this kept happening.

Also, The Oracle Hyperion Shared Services User and Role Security Guide * contains a good write up on the concepts of Active Directory most relevant to Hyperion Foundation. 

Specifically , The section that talks about the DNS lookup.

You can configure Active Directory so Shared Services can perform a static host name lookup or a DNS lookup to identify Active Directory. Static host name lookup does not support Active Directory failover.
Using the DNS lookup ensures high availability of Active Directory in scenarios in which Active Directory is configured on multiple domain controllers to ensure high availability. Whenconfigured to perform a DNS lookup, Shared Services queries the DNS server to identify registered domain controllers and connects to the domain controller with the greatest weight.If the domain controller to which Shared Services is connected fails, Shared Services dynamically switches to the next available domain controller with the greatest weight.

In my case, the Failover Active Directory serving from a different geographic location could not identify the credentials for some reason. So, to restrict the "lookup" to the working Active Directory Site, we had to populate the information for AD Site. The next challenge was to find out the AD Site Information for which running this command is extremely helpful.

nltest [/server:] /dsgetsite 

The result of this command needs to be applied in the AD Site field. Once this setting was applied, the error message stopped appearing.