Monday, January 13, 2020

SSL Enabled MSAD for EPM Systems

So, after a long hiatus , I am finally ready to update this blog on something that is bound to be catastrophic to your EPM environment. Microsoft is enforcing MSAD connections on SSL and its details can be found here.

My 11.1.2.4 environment MSAD connection was not SSL enabled. Per Oracle, implementing the ssl connectivity is a simple straightforward approach documented in KM Doc ID 1599610.1.

Per the document, You need to import your corporate AD SSL Certs into 3 locations inside the EPM system on all servers.




keytool -import -noprompt -trustcacerts -alias < Alias Name > -file < Certificate for Corporate
Directory > -keystore < Keystore Name and Location > -storepass < Keystore Passphrase >


However, When I did that, I ran into the error below.
SSL Error
Bind Error
In order to overcome this, after multiple iterations, I have finally come up with the proper steps to get rid of this problem. If you installed the base version of EPM along with the base WebLogic and Java components, Those will need to be updated.
In my case, I had upgraded Weblogic to 10.3.6 per the instructions mentioned in Doc ID 2503568.1
bsu.cmd -install -patch_download_dir=D:\Oracle\Middleware\utils\bsu\cache_dir -patchlist=7HKN - prod_dir=D:\Oracle\Middleware\wlserver_10.3 -log=7HKN.log
The second and by far the most complicated step was to upgrade JDK and JRockIt to Java7. The steps are mentioned in the Doc ID 2351499.1. I chose to move ahead with the less complicated steps in Choice 1. It was not worth going through the struggle from Choice 2. 
Special Note- Don't forget Essbase, EAS and FR Studio Clients. The steps are a little different than that for Web Apps like Foundation, RAF , Web Analysis,Calc Man etc.
Lastly, Don't forget to update the Corporate SSL Certs on the New Java 7 library. Then,proceed to apply the SSL enabled option on your Shared Services console. It should now happily take port 636 and you will keep your security team happy. 
Happy EPMing!

Thursday, July 16, 2015

Active-Active Disaster Recovery

In my organization, we are very big on DR site being available . In the event of any outage in the datacenter, the failover and failback processes are required to be nearly seamless. From the end user perspective, It is important for them to launch the same link for Smartview and be able to connect to Essbase, Reports, Planning.

In order to achieve this, we need the following:

Automatic DNS redirection

Any redirection tool of your choice can be implemented in your organization. In my case, both my production and DR VIPs are pointed to a separate dns url which gets the heartbeat information and redirects traffic to the primary or secondary site based on the response it gets from the status file. To make this happen, one can simply rename the file and that will make sure the traffic is redirected to the site that is active.

Oracle Wallet should contain the certificate information

This step is pretty standard and the same approach that needs to be adopted to implement SSL in the first place. In short, copy the cer file over to the destination environment. In Oracle Wallet, Open and import the certificate and save it. Also, if you have a load balanced setup we can simply copy the cwallet.sso and the ewallet.p12 file to Oracle\Middleware\user_projects\epmsystem1\httpConfig\ohs\config\OHS\ohs_component\keystores\default

Update the ssl.conf,mod_wl_ohs.conf and httpd.conf files

In a nutshell, make the entries match the production server setup.

On httpd.conf

include "${ORACLE_INSTANCE}/config/${COMPONENT_TYPE}/${COMPONENT_NAME}/admin.conf"
include "${ORACLE_INSTANCE}/config/${COMPONENT_TYPE}/${COMPONENT_NAME}/mod_wl_ohs.conf"

include "moduleconf/*.conf"
include "conf/ssl.conf"
Include "conf/epm_online_help.conf"
Include "conf/epm_rewrite_rules.conf"
Include "conf/epm.conf"
Include "conf/epm_online_help.conf"
Include "conf/epm_rewrite_rules.conf"
Include "conf/epm.conf"

On ssl.conf


NameVirtualHost *:443
   
        ServerName example.hyperion.com
        RewriteEngine On

Restart OHS with these changes populated. The redirection must happen now.


Tuesday, March 24, 2015

Calculation Manager Rules that are still on Planning don't show up on Calculation Manager


In my 11.1.2.3.500 setup, I recently faced an issue with Calculation Manager Rules that were present on Planning forms inside the Planning Application, but however, on expanding the Calculation Manager->Planning App->Plan Type->Rules, it did not display the rules inside. These rules were imported manually earlier and suddenly disappeared. I could not really find what caused it but here are the steps to recover the BR's from planning tables to calculation manager:

1. Access the appropriate Planning application repository and its table HSP_CALC_MGR_RULES

2. Create a new text file with following content


3. Copy the contents in the Body column of the HSP_CALC_MGR_RULES table and paste its contents between the tag in the file created in step 2.
4. Save file as XML file
5. Within the Calculation Manager navigate to “File” -> “Import…” to import the Business Rule into Calculation Manager .

Using this I could restore all the rules back.

Tuesday, March 17, 2015

"Error executing query: -1" When Running Financial Reporting (FR) Reports

When we performed the migration to 11.1.2.3 since it was a new install some of the reports failed with the error "Error executing query:-1" .In addition to this a lot of the missing values were also being generated as a result. To resolve this issue, We had applied a fix in the previous environment as this issue seemed to exist there as well. We found out that this was due to the MDX query setting on FR Server and it was set to the default 'True' value. To carry out the same, follow the steps below.

1. Navigate to C:\Oracle\Middleware\EPMSystem11R1\products\financialreporting\bin and launch FRConfig.cmd


In some instances, this utility doesn't open and throws a java error. To fix that, follow step 2 below.

2. Edit setInstance.cmd and add the
set EPM_ORACLE_INSTANCE=C:\Oracle\Middleware\user_projects\epmsystem1

3. Now, it opens the FRConfig.cmd. In the MBeans tab, expand the Financial Reporting tab and scroll to the EssbaseUseMDX setting.


4. Restart FR services. This should fix the issue. 

I had found that on a parallel environment since it had loopback adapters present for the VIP, It did not work out for me. In that case, launch the enterprise manager module and correct it from there.Click on System MBean Browser.

Navigate to the Application MBeans


Also a side note, PreviewUserPOV can be set to True, so that end users don't have to set their FR Preferences to On. We can make the change on FRStudio.cmd






Thursday, March 12, 2015

BI+ distributed setup "startup document does not exist " "GSM unable to acquire LSM"

We have a network deployment for Foundation and BI+ in our environment and the 2nd node when operated on its own stopped being accessible. RAF agent service kept going to "stopping" mode when stopped and not completely stopped. The RAF Agent service and the Foundation Service was set to the installer account.

I thought it will be a good idea to compile a few things I verified in the hopes that it will help anyone with some troubleshooting ideas.

Ensure to set HyS9RaFrameworkAgent_epmsystem1 service to start with Domain account instead of Local System on both RAF servers and also make sure the domain account has local administrator rights .

Before starting services on Server2 make sure server1 is up and running and Workspace is accessible

Verify that HAVA ID in EPM registry matched in css_provisioning_info table

1: Execute the following query against the Shared Services registry database:
2: select application_id from css_provisioning_info where lower(application_id) like 'hava%';
o note this application id, e.g. HAVA:0000012b3072bf63-0000-6717-0acc1bac

3: Run epmsys_registry.bat from \Oracle\Middleware\user_projects\epmsystem1\bin
Open EPM registry.xml and search for HAVA and compare the HAVA ID using the ID above

4: If there is a mismatch, edit the Shared Services Registry:
Go to EPM_ORACLE_INSTANCE/bin and run the following command:
epmsys_registry updateproperty RA_FRAMEWORK/@applicationId

Where applicationId is the id returned from the query above. For example, using the id above:

epmsys_registry updateproperty RA_FRAMEWORK/@applicationId HAVA: 0000012b3072bf63-0000-6717-0acc1bac

5. Restart all services on Server1 and ensure Shared Services and Workspace are accessible

6: Enable the Service Broker service and the GSM on the second server
- Log on to EPM Workspace on Server1
- Select Navigate, then Reporting and Analysis, and then Services.
- Right-click Reporting and Analysis for server2 , and then select Properties.
- On the Services tab, set GSM to Enabled (Yes) and Service Broker to Enable (Yes).

Reproduce the issue and attach the logs from these locations from both servers 1/2
- \Oracle\Middleware\user_projects\epmsystem1\diagnostics\logs\ReportingAnalysis

and run epmsys_registry.bat from \Oracle\Middleware\user_projects\epmsystem1\bin
For my issue however, everything seemed to tie up quite nicely and I was still running into the issue.
This was the log entry from the server 2  raframework_stdout_console.log when started on its own:

Finally, when I started digging a little deeper, I could understand what was causing this to happen.
While Configuring the RAF for the 2nd node, We normally point to the shared server 1 RM1 directory while configuring.


The second Workspace agent was not able to locate the RM1 directory on server 1 .An issue with DNS could not resolve the server 1 hostname correctly.On second Workspace Agent server, add the server 1 hostname/IP address into the HOSTS file

On server 2, update the hosts file (which is under C:/Windows/System32/etc/drivers/hosts folder) with the ipaddress and the hostname of server1 where RM1 directory stored.
xx.xx.xx.xx server1.domain.com

This helped in resolving the issue.

.

Monday, March 9, 2015

Merge Cells Issue on Workspace Report

We have users that run the reporting out of Workspace and then export into Excel. Historically , The reports were formatted the same between Workspace and Smart View so you could easily pull from either and end up with consistent reporting.

Since the upgrade to 11.1.2.3, It appeared the new Workspace reports are no longer formatted the same. They had merged cells.


To fix the issue,

1. On your FR server, navigate to Oracle\Middleware\user_projects\epmsystem1\FinancialReporting\lib\FOProcessor
2. Save a copy of the existing xdo.cfg file.
3. Open xdo.cfg in a text editor.
4. Toward the top of the file, locate this line:




5. Immediately below it (above the
tag), add this line:


6. Save the file.
7. Restart the FR Web service (wait a few minutes to allow the service to fully render).

Wednesday, February 25, 2015

Calculation Manager Rules Runtime Tuning

We have had a case in 11.1.2.3 Calculation Manager where I have been trying to run the business rule, running them at the Total level in the 11.1.2.3 and I had to kill the calc ran after it ran for 5 hours.

In the process of debugging,I tried at a lower level to see if at least I can get it to complete. I did test the calc at a single intersection and it ran fine in seconds but that really wasn't much good to us
as our users all enter multiple intersections through their ad hoc forms and then need to calc at an aggregate level.

While starting to dig the rootcause , I thought it might be a good idea to share what was done to tune this rule. I started with the Cube cache settings.I have shown the values above.This needs to be followed with a Essbase restart.
Also for the application, I added these two properties . In order to do this, I logged into the planning application, Administration->Application->Properties. There, I added two properties called HBR_MAX_WAIT_FOR_RULE and HBR_MONITOR_DELAY and set them to 480000 which is 8 minutes.

Post this, I carried out a complete system restart and then ran the rule. Now what took a long time was completed in 5 minutes which is a big win. 

Wednesday, February 18, 2015

Hyperion SmartView Tuning

While driving the 11.1.2.2 to 11.1.2.3 upgrade in my organization, I have run into various problems to which some of my earlier posts are a testament.  Today, I bring to you one from the SmartView trove. We have users who swear by Excel Addin and refuse to let go of it. There have been some instances where users are unable to retrieve large numbers on SmartView.
This was the message that was produced when the retrieve was run on private connection using 19000 url .So, It would be something like, http://ohs.servername.com:19000/aps/SmartView .


 On using the SSL link for the same, This was the error message I encountered,

                                                                                                                                                                  
Listed below are a few ways in which the tuning can help in resolving the issue. First , I would tackle the browser timeout settings by the steps shown below.

On the client machine, update/add the following registry keys:

1. Open the Registry, Start -> Run -> Regedit.

2. Locate the following section(s):

[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings]\

or

[HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Internet Settings]

3.Create the following new DWORD keys with decimal values:

ReceiveTimeout=480000
KeepAliveTimeout=180000
ServerInfoTimeout=180000

In this example, the ReceiveTimeout setting is 8 minutes. The KeepAliveTimeout and ServerInfoTimeout settings are 3 minutes. Set these to greater than the longest running request.

4. Restart the machine for the new settings to take effect.

This in my case did not seem to help. There are some more tuning parameters we can add to aps.properties file located at C:\Oracle\Middleware\EPMSystem11R1\products\Essbase\aps\bin

# Specifies values for Essbase Server network connection
# APS will try netConnectRetry times to connect to an Essbase Server.
olap.server.netRetryCount=6000
olap.server.netConnectRetry=30
olap.server.netDelay=2000
olap.server.netSocketTimeOut=20000
Also, these values need to be added to the C:\Oracle\Middleware\EPMSystem11R1\common\EssbaseJavaAPI\11.1.2.0\bin\essbase.properties  file and try making a private connection to  http://aps.servername.com:13080/aps/SmartView
Adding these values helped me resolve the large retrieve issue. This bodes well with IE9 in the case of 11.1.2.3 .

Another request that is normally made is to increase the max rows to 65000 . In order to make that change, modify the properties shown below.

# Specifies maximum rows/columns for dataquery grid. 0 value indicates no limit.
service.olap.dataQuery.grid.maxRows=65000
service.olap.dataQuery.grid.maxColumns=255
To complete the change, login to EAS and expand the provider server tree. Right click on the provider server and edit the properties .


Restart APS . That Resolves the increase row limit Issue.

Tuesday, January 20, 2015

An unexpected error has occured while migrating outlines on EAS

I have been trying to migrate Essbase Outlines on 11.1.2.3.503 and have noticed this error some times on EAS. 

On looking on the EASServer.log there is nothing related to these errors and the messages panel below the EAS console doesn't display any error messages either. 

I have noticed that this occurs on two instances:

1. When you are not "connected" to Server1 or Server 2 Explicitly on EAS , i.e. when server 1 or 2 is not expanded .

2. When the target server is lower than 11.1.2.3 and the source server is 11.1.2.3

In my case, I could continually reproduce the error, when I wasn't connected to either of the servers and it went away after expanding server1 and server2.




Friday, December 19, 2014

Loadbalanced EAS 11.1.2.3 throws Login Error

When I first installed 11.1.2.3 on a load balanced instance , I had many challenges. One of the components that has given me problems consistently has been the Essbase Administration Services, EAS . I have covered various posts on 11.1.2.3 EAS  here,here and here . I am adding one more to the collection with this post.

What happened was that each time I would login to EAS after a restart, it would be fine. But the subsequent problems always resulted in a login error.

I found these entries on the easserver.log

[EssbaseAdminServices0] [WARNING] [] [oracle.epm.essbase.eas] [tid: 12] [userId: ] [ecid: 0000KWxWr6H5IfYTTqxkWc1K1ssg000003,0] [APP: EAS#11.1.2.0] Source directory does not exists
[EssbaseAdminServices0] [WARNING] [] [oracle.epm.essbase.eas] [tid: 12] [userId: ] [ecid: 0000KWxWr6H5IfYTTqxkWc1K1ssg000003,0] [APP: EAS#11.1.2.0] Failed to copy storage directory to new location
[EssbaseAdminServices0] [ERROR] [ESSEAS-35505] [oracle.epm.essbase.eas] [tid: 11] [userId: ] [ecid: 0000KWxb0TM5IfYTTqxkWc1K1ssg00001a,0] [APP: EAS#11.1.2.0] Following Exception occurred for user admin. Message exception is EPMCSS-00301: Failed to authenticate user. Invalid credentials. Enter valid credentials.
[EssbaseAdminServices0] [ERROR] [ESSEAS-35505] [oracle.epm.essbase.eas] [tid: 11] [userId: ] [ecid: 0000KWy029F5IfYTTqxkWc1K1ssg000048,0] [APP: EAS#11.1.2.0] Following Exception occurred for user admin. Message exception is EPMCSS-00301: Failed to authenticate user. Invalid credentials. Enter valid credentials.
[ERROR] [ESSEAS-35505] [oracle.epm.essbase.eas] [tid: 11] [userId: ] [ecid: 0000KX1Ie7Y5IfYTTqxkWc1K286Q00000k,0] [APP: EAS#11.1.2.0] Following Exception occurred for user admin. Message exception is java.lang.NullPointerException


In the SharedServices_security_client.log , I saw the following error :

[EssbaseAdminServices0] [ERROR] [EPMCSS-09223] [oracle.EPMCSS.CSS] [tid: 12] [userId: ] [ecid: 0000KX1E_JR5IfYTTqxkWc1K286Q000002,0] [APP: EAS#11.1.2.0] [SRC_CLASS: com.hyperion.css.spi.CSSManager] [SRC_METHOD: updateDcUrl] Failed to retrieve base DNs while updating dcUrl. Error communicating with server. DNS name not found [response code 3]. Invalid Domain, AD Site or DNS Server.

This normally indicates there is an issue with the CSS configuration. It then dawned upon me that I had messed up my search order while configuring the MSAD. I placed my MSAD in the order 1 and the Native Directory in order 2. I reversed the order and followed it up with a complete EPM system restart. The issue was resolved.


Monday, December 8, 2014

Hyperion EAS stops accepting logins after a while

I have a Hyperion 11.1.2.2 instance which has been running for a while now. Since EAS is not used as much , I never knew of this issue until a user reported it. This user kept getting "Login Failed" and it kept some days after restarting the service. On some deeper analysis, I found that the rootcause of this issue was the location tmp folder for EAS .  We can apply the below fix to resolve the issue.

On the server where EAS is installed, navigate to the user_projects level.






Here we can find there is a folder called tmp. Create a new folder by some name, let's say eas_tmp






















Go to the registry, and update the setting corresponding to Djava.io.tmpdir as shown below













Restart EAS Server. This error should now stop appearing.

Managed Servers don't talk to Admin Server

I have a network deployment for Hyperion BI+,Calc Man, Planning and EAS where all these components are load balanced with the exception of Essbase and it is a distributed server install, 

I ran into an issue with Weblogic where it displayed the below symptoms:

1. Hyperion Planning, EAS, CalcMan, and other things not running on the Admin Server Machine take FOREVER to startup.
2. Once started, AdminServer still shows all the remote nodes as DOWN
3. Errors in the managed server sysout files show managed servers were trying to talk to the wrong IP address to get to the Admin Server
(it was trying to use the VIP address, and not the address associated with the FQDN for the server.









On digging further, I found that the servers are multihomed, mainly due to an odd load balancer setup where there are local loopback addresses on the
admin server that are really hosted by the VIP on the LB. The managed servers were able to make an initial connection to the
admin server, and the admin server responded back with the WRONG IP (the loopback, not the public IP).

The resolution is to set the LISTEN address on the WL AdminServer "Server" to the public DNS name. example.host.domain.com
For this, Login to the WL Console and click Lock & Edit (see Use the Change Center).


On the Servers page, click the name of the server , Select Configuration > General.On the Severs: Configuration: General page, enter a value in Listen Address.



Restart AdminServer,wait for it to be fully running, Restart all the other services. 


WebAnalysis SSL Link for 11.1.2.2 on a patched platform

When I first installed 11.1.2.2 , there wasn't a need for a WebAnalysis solution. Later, when there was a need , the install wasn't straightforward. Read my previous post on how to fix the installation problem here . I could get through with my installation but however, launching WebAnalysis on SSL became a problem. The 16000/19000 link would launch fine but not 443. I enabled debug trace on and saw these messages on the java console.










For some reason 16 jar files and jre-6u22-windows-i586-s.exe are missing from WebAnalysis.ear file.

These files are present in %HYPERION_HOME%\common\epmstatic\webanalysis folder where as the static folder in WebAnalysis.ear located under ' %HYPERION_HOME%\products\biplus\InstallableApps file doesn't have them.

Copy those files to the tmp directory of Weblogic server as seen below.

C:\Oracle\Middleware\user_projects\domains\EPMSystem\
servers\WebAnalysis0\tmp\servers\WebAnalysis0\tmp\_WL_user\
WEBANALYSIS_11.1.2.0\ts1hx4\war\static

If you are interested in a permenant work around/fix, then you can add the jar files and jre-6u22-windows-i586-s.exe to the WebAnalysis.war file.

Open WebAnalysis.ear file with 7Zip or Winrar
Open WebAnalysis.war file and navigate to static folder
Compare the file present in %HYPERION_HOME%\common\epmstatic\webanalysis against static folder and add missing ones.


BUG INFO:

Web Analysis Studio Unable to Launch When Using Default 16000 Port Error: "class com.hyperion.analyzer.usersession_base.AnalyzerClient not found." [ID 1365054.1]

It says "The ability to login to WA using the listening port:16000 has been removed by development.
This is documented in unpublished (documentation) Bug 13061357"

Tuesday, November 18, 2014

Assign Access Control View appears in different font on 11.1.2.3

On 11.1.2.3 base version,once you launch Foundation via workspace, expand any Essbase Server and right click to assign access control, the view appears in a completely different font . It reminds me what it used to look like prior to 11.1.1.3.

This is a published Bug 17354487: ASSIGN ACCESS CONTROL BUTTON ICONS MISSING and the solution is to copy “ui_themes” folder from
  C:\Oracle\Middleware\user_projects\domains\EPMSystem\servers\FoundationServices0\tmp\_WL_user\SHAREDSERVICES_11.1.2.0\yidcyl\war

 To:
  C:\Oracle\Middleware\user_projects\domains\EPMSystem\servers\FoundationServices0\tmp\_WL_user\WORKSPACE_11.1.2.0\340u89\war

Stop and start foundation services for the change to be updated.
This problem will then be resolved. 

Sunday, November 16, 2014

WebAnalysis on 11.1.2.3 doesn't launch SSL URL

When I installed SSL for all my applications on 11.1.2.3, The link for WebAnalysis Studio did not launch . It started the JRE but failed to launch the logon screen. In an attempt to debug the issue, I opened the java console and enabled logging. I saw the below exception there:









The link worked on 19000 but not on 443. The reason behind this is that rewrite rule was not present in my httpd.conf and adding the entry below in this section helped me resolve the issue.

RewriteRule ^/WebAnalysis/static/(.*) /epmstatic/webanalysis/$1 [PT]

So , the complete section should look like this when done.

RewriteEngine On
RewriteRule ^/interop/bpmstatic/(.*) /epmstatic/bpmui/$1 [PT]
RewriteRule ^/raframework/bpmstatic/(.*) /epmstatic/bpmui/$1 [PT]
RewriteRule ^/workspace/static/(.*) /epmstatic/wspace/$1 [PT]
RewriteRule ^/workspace/bpmstatic/(.*) /epmstatic/bpmui/$1 [PT]
RewriteRule ^/HyperionPlanning/help/(.*) /epmstatic/planning/docs/$1 [PT]
RewriteRule ^/calcmgr/bpmstatic/(.*) /epmstatic/bpmui/$1 [PT]
RewriteRule ^/awb/bpmstatic/(.*) /epmstatic/bpmui/$1 [PT]
RewriteRule ^/WebAnalysis/static/(.*) /epmstatic/webanalysis/$1 [PT]

This fix is explained in more detail in Doc ID 1354981.1

FR Icons missing in BI+ Workspace

When I installed 11.1.2.3, FR was my choice of reporting tool and I noticed something quite odd on the User Interface. The icons for FR were not visible and I would attach a screenshot, But I forgot to take one and I resolved the issue so the issue no longer is there. I have installed quite a few instances and this behavior has not been consistent. In any case, the resolution for it is below. Looks like there is a KM for this for 11.1.2.1 Doc ID 1368253.1 .
  • Go to the C:\Oracle\Middleware\EPMSystem11R1\products\biplus\InstallableApps directory 
  • Using 7-zip (http://www.7-zip.org), open the raframework.ear file 
  • Using 7-zip, open the raframework.war file from within the raframework.ear file 
  • Using 7-zip, open the WEB-INF directory from within the raframework.war file 
  • Using 7-zip, open the weblogic.xml file from within the WEB-INF directory to modify the file 
  • The tag may contain a path that looks like @oracleInstance@/ReportingAnalysis/RAFrameworkWebapp 
  • Change so it will look like C:/Oracle/Middleware/user_projects/epmsystem1/ReportingAnalysis/RAFrameworkWebapp 
  • Save the file and update the weblogic.xml file in the archive when prompted. 
  • Restart the RAF webapp service 
This fix helps resolve the issue. 




Thursday, November 13, 2014

Load balanced 2nd node for Hyperion BI+ doesn't work independently

While building 11.1.2.3 from the  ground up, I have run into a string of issues. I will continue to post on all the issues but these tips can help someone who may be in a similar situation and some things to think about while troubleshooting them.

I have every web application load balanced on windows 2008 r2 64 bit . I began testing as soon as I completed configuring everything and started to test indepently to see if node 2 continues to run when node1 is completely down because in an actual DR situation these things matter.Thats when I ran into this message for the first time.

"The Startup Document does not exist in the repository" followed by "GSM_unabletoacquire LSM"

On a network deployment there are several reasons that can cause this:

1. The service account for the HyS9RaFrameworkAgent_epmsystem is not the same as the installer but the "local system" account. This is an absolute must since RM1 is a shared folder across 2 servers.

2. The Service Broker service and the GSM on the second server is not enabled.
To Resolve this,
- Log on to EPM Workspace on Server1
- Select Navigate, then Reporting and Analysis, and then Services.
- Right-click Reporting and Analysis for server2 , and then select Properties.
- On the Services tab, set GSM to Enabled (Yes) and Service Broker to Enable (Yes).

3. Verify that HAVA ID in EPM registry matched in css_provisioning_info table
a: Execute the following query against the Shared Services registry database:
b: select application_id from css_provisioning_info where lower(application_id) like 'hava%';
 note this application id, e.g. HAVA:0000000b0000bf00-0000-0000-0acc1bac
c: Run epmsys_registry.bat from \Oracle\Middleware\user_projects\epmsystem1\bin
Open EPM registry.xml and search for HAVA and compare the HAVA ID using the ID above

d: If there is a mismatch, edit the Shared Services Registry:
Go to EPM_ORACLE_INSTANCE/bin and run the following command:
epmsys_registry updateproperty RA_FRAMEWORK/@applicationId

Where applicationId is the id returned from the query above. For example, using the id above:

epmsys_registry updateproperty RA_FRAMEWORK/@applicationId HAVA: 0000000b0000bf00-0000-0000-0acc1bac

e. Restart all services on Server1 and ensure Shared Services and Workspace are accessible

However, in my case, all of these seemed perfect. I stumbled upon my resolution by chance . The Location for the RM1 shared folder (in the configurator, Eg: \\server1\c$\Oracle\Middleware\user_projects\epmsystem1\ReportingAnalysis\data\RM1 stopped being recognized from server 2 . 
The way in which server 1 is resolved from server 2 turned out to be the root cause of my issue.It used to be accessible by the short name earlier and suddenly started to accept only the FQDN owing to some DNS related changes.  In my opinion, We should go with the same servername that we issue in the host file as part of the preinstall. To override that, I added an entry on the server 2 host file and it started working.




Wednesday, November 12, 2014

EAS SSL Link offloaded at OHS throws a jnlp file



I am on 11.1.2.3.503 and the SSL link for EAS presents me with a jnlp file and it works fine with the 19000 link .

But when you try to access the SSL link, it throws a " administration server does not exist" error.

While this has been discussed in plenty in various KMs DocID1346806.1 . This fix worked perfectly on 11.1.2.2 but on 11.1.2.3 , There is one additional fix that the KM doesn't cover.


The solution is to modify the link wherever you see a reference to the old link, modify that reference to include the 443 port.

Example of these entries are below:


       
       



Now if you save a local copy of these files, if you double click the jnlp it launches the SSL console and you don't see the error anymore.



Monday, November 10, 2014

SSL offloaded at OHS results in smartview RAF not displaying any reports

On a SSL implemented 11.1.2.3 system, which is offloaded at OHS , there is a problem with displaying the list of reports inside the cname. This is a really weird problem as the link works on 19000 .

The end result might look something like this.











According to the DocID1364770.1 , Applying the fix for the step 1 and 2 should fix the issue. But it doesn't , in my case.

Step 1:
  • Configure Weblogic to look for specific HTTP headers, amongst which the "WL-Proxy-SSL" header:
  1. Start the Weblogic Administration server
  2. Log in to the Weblogic Administration Console, then put the Weblogic console into lock/edit mode
  3. On the left-hand side, select the top level branch named "epm_system" (i.e. the domain name)
  4. Select "Web Applications" on the right hand side
  5. Scroll down and enable the option named "Weblogic Plugin Enabled"
  6. Save and activate the changes.

Step 2:

  • Configure the OHS component to populate the requests with the "WL-Proxy-SSL" HTTP header:
  1. On all servers hosting the OHS component, edit the file called httpd.conf (located in \Middleware\user_projects\epmsystem1\httpConfig\ohs\config\OHS\ohs_component) and append the following line at the bottom:
  2. WLProxySSL ON
  3. Save and close that file.
To resolve the issue, I applied the below fix:



Backup Shared Services repository database 

1. On the server that hosts the Shared Services Registry, open a command prompt. 

2. Go to C:\Oracle\Middleware\user_projects\epmsystem1\bin and run the following commands: 

epmsys_registry addproperty system9/@enable_ssl_offloading true 
epmsys_registry addproperty system9/@external_url_host  (SSL_off_loader_host_name) 
epmsys_registry addproperty system9/@external_url_port 443 (SSL_off_loader_port) 

3: Restart Foundation, Workspace Agent, and Workspace web applications services 

This fixes the issue and the list can now be seen on Smartview

Thursday, June 5, 2014

EAS Timeout Issues

I was trying to see fun ways to start working using a disconnected remote session and one thing that caught me in a web was the endless woes with EAS timeout . Each time I had to open a fresh session and it wasn't until recently I stumbled upon this document from the online help.

http://docs.oracle.com/cd/E40248_01/epm.1112/eas_help/frameset.htm?timeout.html  is the link the document.

To fix this issue, We need to unzip the EPM_ORACLE_HOME\products\Essbase\eas\server\AppServer\InstallableApps\Common\eas.ear file














The document encourages us to edit the zip file seperately. However, we can achieve the same using the 'open archive' option in 7zip.

Once this is open , proceed to open the eas.war file














Navigate to the web.xml file present on WEB-INF tree.










Open the file and scroll to the session-timeout section. The default value is 45 and you can change it to the value appropriate to your setup .  In my case , I set it to 180.








The link does not specify any restart required but it is always better to do it. Post this, you should be able to stay logged in to EAS longer.


Wednesday, June 4, 2014

PDF preview for Hyperion Financial Reports fails to load

I had an issue once with the PDF preview generation for a FR Report on 11.1.2.2.303. It seemed to affect one out of the two servers which had Financial Reporting deployed on it. The html preview turned out fine but the pdf preview was affected. Also, what was strange was it seemed to affect one server and not the other even though they were on the same patched version.

The resultant report produced a view like this,



























The sysout log for FR had references to the errors IOException has occured in FOProcessor.setData(ByteArrayInputStream)

There is a KM document 1564844.1 which details 3 methods to resolve the issue. I did not go with the first option as it involved rolling back a patch and the .303 patch contained some important fixes related to my Financial Reporting setup.  I , however,  followed the third method of manually editing the xdo.cfg file. The steps are mentioned below:


1. Log onto the FR Web App Server
2. Navigate to Oracle\Middleware\EPMSystem11R1\products\financialreporting\lib\FOProcessor
3. Backup the existing xdo.cfg file, then delete or rename the file
4. Open the original xdo.cfg file in a text editor
5. Locate the line containing the text  "<property name="system-temp-dir">c:\temp</property>"
6. Comment the line out using XML comment syntax: <!-- <property name="system-temp-dir">c:temp</property> -->   
7. Repeat the above steps on all FR Servers
8. Restart all affected FR Web App services

After this, I have noticed that the preview is generated without any issues.



Thursday, May 29, 2014

Weblogic changes cannot be activated - Timeout Error

For a few months now, I have been dealing with a lot of WebLogic 10.3.6 related frustration because I was running into an issue with updating one setting and each time I tried to activate the changes I used to run into an issue trying to activate the changes .



I have a deployment with 4 Managed Servers each of which host 4 Hyperion services load balanced across 4 VMs. These were the preliminary steps I worked with Oracle on. So, I thought these are some of the steps that we can try on our own and help accelerate the resolution process.

1. Shutdown the weblogic domain admin console then move those files out of the Pending folder and then restart the admin console and try to activate the changes again.

2. Navigate to /pending folder. In this folder, there are configuration files which are temporary config files which are cached with changes made from console. At the time of activating changes, the config.xml from the pending folder replaces the original config.xml and other configuration files inside the /config folder.  Kill the Admin Server, and replace the config.xml from the /config folder with the config.xml from the /pending folder. 
Delete all the .lok files inside like edit.lok and config.lok.  Restart Admin Server. 

3. Flush down the changes to be activated using WLST. 
You can use the below script:



4. And of course the classic, Increase Logging level. This can be again perfomed on the config.xml file.


In my case however, none of these methods helped and each time I kept hitting a roadblock . The solution was to update the Listen-address on the config.xml and to repeat the steps for all the managed servers and the components deployed on the mananged server. It is important also to follow a specific order to carry the steps below ,out. 

a. Stop all the Hyperion Services and the Weblogic service.
b. In your admin server, navigate to where the pending directory lies in this case, X:\Oracle\Middleware\user_projects\domains\EPMSystem and rename pending to pending_bkp.
c. Modify your entry in the config.xml file is in the below syntax.

Change the  to include the IP/FQDN to reflect the new entry. So, Post the change,the entry becomes,






d.After this is done for all the managed server components deployed under Weblogic, Start the Weblogic Admin server and the Hyperion Application Services.

You should be able to activate any pending changes now.


Friday, May 16, 2014

CalcMan 11.1.2.2 does not produce updated view for rules on IE browser

I recently ran into this issue with my clients where on a load balanced setup on Windows 2008R2 Platform , The CalcMan rules that they created were at times visible immediately and at times it doesn't produce the updated results.

Initially , I thought it was a browser related issue so I suggested that they clear their browser cache and retry on some attempts it would work and fail some other times. On digging some more, I could isolate the behaviour to one server.On Stopping and Starting Services , I hit this snag on the diagnostics/services log.

java.rmi.MarshalException: failed to marshal create(Lweblogic.rmi.spi.HostID;ILweblogic.cluster.replication.ROID;Lweblogic.cluster.replication.Replicatable;); nested exception is:
java.io.NotSerializableException: com.hyperion.css.common.CSSProvisionedAppInfo
at weblogic.rjvm.BasicOutboundRequest.marshalArgs(BasicOutboundRequest.java:92)
at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:221)
at weblogic.cluster.replication.ReplicationManager_1036_WLStub.create(Unknown Source)
at sun.reflect.GeneratedMethodAccessor4706.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
Truncated. see log file for complete stacktrace
Caused By: java.io.NotSerializableException: com.hyperion.css.common.CSSProvisionedAppInfo
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
at java.util.HashMap.writeObject(HashMap.java:1195)
at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
Truncated. see log file for complete stacktrace
>

The solution was  to add this entry to CalcMgrConfig.properties file on both the servers. The normal location for this file is ,

E:\Oracle\Middleware\user_projects\domains\EPMSystem\servers\CalcMgr0\tmp\servers\CalcMgr0\tmp\_WL_user\CALC_11.1.2.0\qxf61x\war\WEB-INF\conf
If the value is already present for DATABASE_CACHE_REFRESHRATE, Change it to 1000.

#Database Caching refresh rate in milliseconds
DATABASE_CACHE_REFRESHRATE=1000

Restart the Calculation Manager Services Post that. It should now produce an updated view on browsers.



Thursday, May 15, 2014

Users's last login into the system for Essbase for v11.1.2.2

I am sure time and again various people have posted about how to obtain the user login information from EAS.  Users Last Logon Time post by Celvin provides a good deep dive into how this is achieved across versions.

One method to tune the query Date wise ,published by Celvin is to add ORDER BY "Last Login Date" to the end of the query. So Basically, at the end of the query it should look like,


SELECT username as "User Name", MAX(entrydate) AS "Last Login Date" 
FROM serverlogdetail where username is not null
GROUP BY username ORDER BY "Last Login Date";

However, this query doesn't serve the purpose in the 11.1.2.2 version. 



I researched a little and found the following setting yields the same result as the earlier versions on EAS.
This setting is supposed to populate the Last Login Time" column in the eas console under "Enterprise View > Administration Servers > Essbase Servers > [Hostname] > Security > Users




Add to esbase.cfg the setting: PERSISTUSERATLOGIN TRUE and then restart the Essbase service.






Wednesday, April 9, 2014

MSAD Settings in Hyperion and their Significance

While configuring Active Directory ( MSAD in this instance) there are quite a few parameters that we overlook and one such time while migrating from 11.1.1.3 to 11.1.2.2,  I used the exact same settings that my prior environment had  and once everything was configured ,while testing , I ran into this issue with users from across the globe got “Invalid login credentials” while logging into Essbase, Workspace or Planning and each time after restarting the entire suite it 



would start working back for a while and would loop back to the same state in some time . 
The logs indicated only numerous references to the Invalid credentials message.
Some good tools that serve as a starting point for investigating the root cause are LDAPBrowser, Wireshark that can help in understanding why this kept happening.

Also, The Oracle Hyperion Shared Services User and Role Security Guide * contains a good write up on the concepts of Active Directory most relevant to Hyperion Foundation. 


Specifically , The section that talks about the DNS lookup.



You can configure Active Directory so Shared Services can perform a static host name lookup or a DNS lookup to identify Active Directory. Static host name lookup does not support Active Directory failover.
Using the DNS lookup ensures high availability of Active Directory in scenarios in which Active Directory is configured on multiple domain controllers to ensure high availability. Whenconfigured to perform a DNS lookup, Shared Services queries the DNS server to identify registered domain controllers and connects to the domain controller with the greatest weight.If the domain controller to which Shared Services is connected fails, Shared Services dynamically switches to the next available domain controller with the greatest weight.



In my case, the Failover Active Directory serving from a different geographic location could not identify the credentials for some reason. So, to restrict the "lookup" to the working Active Directory Site, we had to populate the information for AD Site. The next challenge was to find out the AD Site Information for which running this command is extremely helpful.

nltest [/server:] /dsgetsite 

The result of this command needs to be applied in the AD Site field. Once this setting was applied, the error message stopped appearing. 

Wednesday, April 2, 2014

java.net.SocketException: Network is unreachable: connect Error for EPMSYSTEM domain


Recently, I faced an issue on the Epmsystem domain on  Weblogic for Hyperion on one server which had APS,EAS, Planning and Calculation Manager installed on it. All of the sysout logs contained no error messages  but none of the services seemed to be funcitoning. While investigating, The Epmsystem.log was full of error messages of this order, and this seemed to repeat for all projects, like in this instance, i am referring to AnalyticProviderServices1  but the error was the same for EssbaseAdministrationService1, Calculation Manager1, Planning1 etc.

#### <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1393517864032> java.net.SocketException: Network is unreachable: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at weblogic.socket.SocketMuxer.newSocket(SocketMuxer.java:374)
at weblogic.socket.SocketMuxer.newClientSocket(SocketMuxer.java:384)
at weblogic.socket.ChannelSocketFactory.createSocket(ChannelSocketFactory.java:86)
at weblogic.socket.ChannelSocketFactory.createSocket(ChannelSocketFactory.java:56)
at weblogic.net.http.HttpClient.openServer(HttpClient.java:323)
at weblogic.net.http.HttpClient.openServer(HttpClient.java:425)
at weblogic.net.http.HttpClient.New(HttpClient.java:252)
at weblogic.net.http.HttpURLConnection.connect(HttpURLConnection.java:213)
at weblogic.cluster.HTTPExecuteRequest.connect(HTTPExecuteRequest.java:63)
at weblogic.cluster.HTTPExecuteRequest.run(HTTPExecuteRequest.java:82)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
>

The root cause was found to be that the machine though dualstacked and supporting just Ipv4 was somehow binding to the IPv6 Address of the machine.

Resolution - In the SetDomainEnv.cmd file, force the usage of IPv4 on the machine. 

"-Djava.net.preferIPv4Stack=true" is the keyword to be added to the JAVA_OPTIONS Section. An example of the usage will be, 
set JAVA_OPTIONS=%JAVA_OPTIONS% %JAVA_PROPERTIES% -Dwlw.iterativeDev=%iterativeDevFlag% -Dwlw.testConsole=%testConsoleFlag% -Dwlw.logErrorsToConsole=%logErrorsToConsoleFlag% -Djava.net.preferIPv4Stack=true

The Original line will not contain the -Djava.net.preferIPv4Stack=true is not present. After this, a complete restart of services is needed and this will help in resolving the issue.

Wednesday, March 12, 2014

DELETING A CORRUPT PLANNING APPLICATION FROM EPM REGISTRY




A planning application, when it goes corrupt/missing,  can cause a lot of angst. This is one of those occasions where one can truly appreciate the beauty of the epmregistry utility.
We can invoke this utility from this location
MIDDLEWARE_HOME\user_projects\instance_name\bin directory  
In order to first begin , generate a registry report the output can be viewed in the diagnostics/reports directory.
Firstly proceed to delete the instance :
Delete Instance File
epmsys_registry removefile PLANNING_PRODUCT/@APPLICATIONNAME.instance
Now, get to the planning application:
Delete PLANNING  application
epmsys_registry deletecomponent #APPLICATIONNAME.HP

Post this , it should be a relatively straight forward approach of recreating the planning application .The best part about using this method is that you are able to give the application the same name post corruption. Not sure if that would have been possible in earlier releases.

How to deal with cwallet.sso corruption for EAS Service

I recently faced an issue with an EAS service which failed to startup properly. The eas_sysout.log was full of error messages like what is shown below:


Caused By: oracle.security.jps.service.credstore.CredStoreException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IOException: PKI-02002: Unable to open the wallet. Check password. 
at oracle.security.jps.internal.credstore.ssp.CsfWalletManager.openWallet(CsfWalletManager.java:184)
at oracle.security.jps.internal.credstore.ssp.SspCredentialStore.doInit(SspCredentialStore.java:239)
at oracle.security.jps.internal.credstore.ssp.SspCredentialStore.(SspCredentialStore.java:153)
at oracle.security.jps.internal.credstore.ssp.SspCredentialStore.(SspCredentialStore.java:139)
at oracle.security.jps.internal.credstore.ssp.SspCredentialStoreProvider.getInstance(SspCredentialStoreProvider.java:115)
Truncated. see log file for complete stacktrace


This indicates a corruption of the cwallet.sso file . The reasons behind the corruption of the file could be anything.  We should check for permissions and what kind of access is present on the file and the folder containing the file. Typically, Administrators must have ‘full control’ on the file and the folder. If that is set right and the service still doesn’t start up, look up the size difference of the cwallet.sso  in the
MIDDLEWARE_HOME\user_projects\domains\domainname\config\fmwconfig  and the bootstrap folder . If there is a difference, obviously something is wrong in the file present in the fmwconfig. Rename and replace with the file in the bootstrap folder and try and start up the service. If it still errors out, Consider this alternative:

Create a new location for the cwallet.sso that the EAS is going to use: EPM_ORACLE_HOME\products\Essbase\eas\server.

The KM article 1447461.1 , provides a detailed step by step account of how to achieve this.

Follow the steps indicated in that article and grant full permissions on the cwallet.sso regenerated once the service is started up. That should help in resolving the issue. 

Follow by Email