Loadbalanced EAS 11.1.2.3 throws Login Error

When I first installed 11.1.2.3 on a load balanced instance , I had many challenges. One of the components that has given me problems consistently has been the Essbase Administration Services, EAS . I have covered various posts on 11.1.2.3 EAS  here,here and here . I am adding one more to the collection with this post.

What happened was that each time I would login to EAS after a restart, it would be fine. But the subsequent problems always resulted in a login error.

I found these entries on the easserver.log

[EssbaseAdminServices0] [WARNING] [] [oracle.epm.essbase.eas] [tid: 12] [userId: ] [ecid: 0000KWxWr6H5IfYTTqxkWc1K1ssg000003,0] [APP: EAS#11.1.2.0] Source directory does not exists
[EssbaseAdminServices0] [WARNING] [] [oracle.epm.essbase.eas] [tid: 12] [userId: ] [ecid: 0000KWxWr6H5IfYTTqxkWc1K1ssg000003,0] [APP: EAS#11.1.2.0] Failed to copy storage directory to new location
[EssbaseAdminServices0] [ERROR] [ESSEAS-35505] [oracle.epm.essbase.eas] [tid: 11] [userId: ] [ecid: 0000KWxb0TM5IfYTTqxkWc1K1ssg00001a,0] [APP: EAS#11.1.2.0] Following Exception occurred for user admin. Message exception is EPMCSS-00301: Failed to authenticate user. Invalid credentials. Enter valid credentials.
[EssbaseAdminServices0] [ERROR] [ESSEAS-35505] [oracle.epm.essbase.eas] [tid: 11] [userId: ] [ecid: 0000KWy029F5IfYTTqxkWc1K1ssg000048,0] [APP: EAS#11.1.2.0] Following Exception occurred for user admin. Message exception is EPMCSS-00301: Failed to authenticate user. Invalid credentials. Enter valid credentials.
[ERROR] [ESSEAS-35505] [oracle.epm.essbase.eas] [tid: 11] [userId: ] [ecid: 0000KX1Ie7Y5IfYTTqxkWc1K286Q00000k,0] [APP: EAS#11.1.2.0] Following Exception occurred for user admin. Message exception is java.lang.NullPointerException

In the SharedServices_security_client.log , I saw the following error :

[EssbaseAdminServices0] [ERROR] [EPMCSS-09223] [oracle.EPMCSS.CSS] [tid: 12] [userId: ] [ecid: 0000KX1E_JR5IfYTTqxkWc1K286Q000002,0] [APP: EAS#11.1.2.0] [SRC_CLASS: com.hyperion.css.spi.CSSManager] [SRC_METHOD: updateDcUrl] Failed to retrieve base DNs while updating dcUrl. Error communicating with server. DNS name not found [response code 3]. Invalid Domain, AD Site or DNS Server.

This normally indicates there is an issue with the CSS configuration. It then dawned upon me that I had messed up my search order while configuring the MSAD. I placed my MSAD in the order 1 and the Native Directory in order 2. I reversed the order and followed it up with a complete EPM system restart. The issue was resolved.

Hyperion EAS stops accepting logins after a while

I have a Hyperion 11.1.2.2 instance which has been running for a while now. Since EAS is not used as much , I never knew of this issue until a user reported it. This user kept getting "Login Failed" and it kept some days after restarting the service. On some deeper analysis, I found that the rootcause of this issue was the location tmp folder for EAS .  We can apply the below fix to resolve the issue.

On the server where EAS is installed, navigate to the user_projects level.

Here we can find there is a folder called tmp. Create a new folder by some name, let's say eas_tmp

Go to the registry, and update the setting corresponding to Djava.io.tmpdir as shown below

Restart EAS Server. This error should now stop appearing.

Managed Servers don't talk to Admin Server

I have a network deployment for Hyperion BI+,Calc Man, Planning and EAS where all these components are load balanced with the exception of Essbase and it is a distributed server install, 

I ran into an issue with Weblogic where it displayed the below symptoms:

1. Hyperion Planning, EAS, CalcMan, and other things not running on the Admin Server Machine take FOREVER to startup.

2. Once started, AdminServer still shows all the remote nodes as DOWN

3. Errors in the managed server sysout files show managed servers were trying to talk to the wrong IP address to get to the Admin Server

(it was trying to use the VIP address, and not the address associated with the FQDN for the server.

On digging further, I found that the servers are multihomed, mainly due to an odd load balancer setup where there are local loopback addresses on the

admin server that are really hosted by the VIP on the LB. The managed servers were able to make an initial connection to the

admin server, and the admin server responded back with the WRONG IP (the loopback, not the public IP).

The resolution is to set the LISTEN address on the WL AdminServer "Server" to the public DNS name. example.host.domain.com

For this, Login to the WL Console and click Lock & Edit (see Use the Change Center).

On the Servers page, click the name of the server , Select Configuration > General.On the Severs: Configuration: General page, enter a value in Listen Address.

Restart AdminServer,wait for it to be fully running, Restart all the other services. 

WebAnalysis SSL Link for 11.1.2.2 on a patched platform

When I first installed 11.1.2.2 , there wasn't a need for a WebAnalysis solution. Later, when there was a need , the install wasn't straightforward. Read my previous post on how to fix the installation problem here . I could get through with my installation but however, launching WebAnalysis on SSL became a problem. The 16000/19000 link would launch fine but not 443. I enabled debug trace on and saw these messages on the java console.

For some reason 16 jar files and jre-6u22-windows-i586-s.exe are missing from WebAnalysis.ear file.

These files are present in %HYPERION_HOME%\common\epmstatic\webanalysis folder where as the static folder in WebAnalysis.ear located under ' %HYPERION_HOME%\products\biplus\InstallableApps file doesn't have them.

Copy those files to the tmp directory of Weblogic server as seen below.

C:\Oracle\Middleware\user_projects\domains\EPMSystem\
servers\WebAnalysis0\tmp\servers\WebAnalysis0\tmp\_WL_user\
WEBANALYSIS_11.1.2.0\ts1hx4\war\static

If you are interested in a permenant work around/fix, then you can add the jar files and jre-6u22-windows-i586-s.exe to the WebAnalysis.war file.

Open WebAnalysis.ear file with 7Zip or Winrar
Open WebAnalysis.war file and navigate to static folder
Compare the file present in %HYPERION_HOME%\common\epmstatic\webanalysis against static folder and add missing ones.


BUG INFO:

Web Analysis Studio Unable to Launch When Using Default 16000 Port Error: "class com.hyperion.analyzer.usersession_base.AnalyzerClient not found." [ID 1365054.1]

It says "The ability to login to WA using the listening port:16000 has been removed by development.
This is documented in unpublished (documentation) Bug 13061357"

Assign Access Control View appears in different font on 11.1.2.3

On 11.1.2.3 base version,once you launch Foundation via workspace, expand any Essbase Server and right click to assign access control, the view appears in a completely different font . It reminds me what it used to look like prior to 11.1.1.3.

This is a published Bug 17354487: ASSIGN ACCESS CONTROL BUTTON ICONS MISSING and the solution is to copy “ui_themes” folder from

  C:\Oracle\Middleware\user_projects\domains\EPMSystem\servers\FoundationServices0\tmp\_WL_user\SHAREDSERVICES_11.1.2.0\yidcyl\war

 To:

  C:\Oracle\Middleware\user_projects\domains\EPMSystem\servers\FoundationServices0\tmp\_WL_user\WORKSPACE_11.1.2.0\340u89\war

Stop and start foundation services for the change to be updated.

This problem will then be resolved. 

WebAnalysis on 11.1.2.3 doesn't launch SSL URL

When I installed SSL for all my applications on 11.1.2.3, The link for WebAnalysis Studio did not launch . It started the JRE but failed to launch the logon screen. In an attempt to debug the issue, I opened the java console and enabled logging. I saw the below exception there:

The link worked on 19000 but not on 443. The reason behind this is that rewrite rule was not present in my httpd.conf and adding the entry below in this section helped me resolve the issue.

RewriteRule ^/WebAnalysis/static/(.*) /epmstatic/webanalysis/$1 [PT]

So , the complete section should look like this when done.

RewriteEngine On

RewriteRule ^/interop/bpmstatic/(.*) /epmstatic/bpmui/$1 [PT]

RewriteRule ^/raframework/bpmstatic/(.*) /epmstatic/bpmui/$1 [PT]

RewriteRule ^/workspace/static/(.*) /epmstatic/wspace/$1 [PT]

RewriteRule ^/workspace/bpmstatic/(.*) /epmstatic/bpmui/$1 [PT]

RewriteRule ^/HyperionPlanning/help/(.*) /epmstatic/planning/docs/$1 [PT]

RewriteRule ^/calcmgr/bpmstatic/(.*) /epmstatic/bpmui/$1 [PT]

RewriteRule ^/awb/bpmstatic/(.*) /epmstatic/bpmui/$1 [PT]

RewriteRule ^/WebAnalysis/static/(.*) /epmstatic/webanalysis/$1 [PT]

This fix is explained in more detail in Doc ID 1354981.1

FR Icons missing in BI+ Workspace

When I installed 11.1.2.3, FR was my choice of reporting tool and I noticed something quite odd on the User Interface. The icons for FR were not visible and I would attach a screenshot, But I forgot to take one and I resolved the issue so the issue no longer is there. I have installed quite a few instances and this behavior has not been consistent. In any case, the resolution for it is below. Looks like there is a KM for this for 11.1.2.1 Doc ID 1368253.1 .

• Go to the C:\Oracle\Middleware\EPMSystem11R1\products\biplus\InstallableApps directory 
• Using 7-zip (http://www.7-zip.org), open the raframework.ear file 
• Using 7-zip, open the raframework.war file from within the raframework.ear file 
• Using 7-zip, open the WEB-INF directory from within the raframework.war file 
• Using 7-zip, open the weblogic.xml file from within the WEB-INF directory to modify the file 
• The tag may contain a path that looks like @oracleInstance@/ReportingAnalysis/RAFrameworkWebapp 
• Change so it will look like C:/Oracle/Middleware/user_projects/epmsystem1/ReportingAnalysis/RAFrameworkWebapp 
• Save the file and update the weblogic.xml file in the archive when prompted. 
• Restart the RAF webapp service 

This fix helps resolve the issue. 

Load balanced 2nd node for Hyperion BI+ doesn't work independently

While building 11.1.2.3 from the  ground up, I have run into a string of issues. I will continue to post on all the issues but these tips can help someone who may be in a similar situation and some things to think about while troubleshooting them.

I have every web application load balanced on windows 2008 r2 64 bit . I began testing as soon as I completed configuring everything and started to test indepently to see if node 2 continues to run when node1 is completely down because in an actual DR situation these things matter.Thats when I ran into this message for the first time.

"The Startup Document does not exist in the repository" followed by "GSM_unabletoacquire LSM"

On a network deployment there are several reasons that can cause this:

1. The service account for the HyS9RaFrameworkAgent_epmsystem is not the same as the installer but the "local system" account. This is an absolute must since RM1 is a shared folder across 2 servers.

2. The Service Broker service and the GSM on the second server is not enabled.
To Resolve this,
- Log on to EPM Workspace on Server1
- Select Navigate, then Reporting and Analysis, and then Services.
- Right-click Reporting and Analysis for server2 , and then select Properties.
- On the Services tab, set GSM to Enabled (Yes) and Service Broker to Enable (Yes).

3. Verify that HAVA ID in EPM registry matched in css_provisioning_info table
a: Execute the following query against the Shared Services registry database:
b: select application_id from css_provisioning_info where lower(application_id) like 'hava%';
 note this application id, e.g. HAVA:0000000b0000bf00-0000-0000-0acc1bac
c: Run epmsys_registry.bat from \Oracle\Middleware\user_projects\epmsystem1\bin
Open EPM registry.xml and search for HAVA and compare the HAVA ID using the ID above

d: If there is a mismatch, edit the Shared Services Registry:
Go to EPM_ORACLE_INSTANCE/bin and run the following command:
epmsys_registry updateproperty RA_FRAMEWORK/@applicationId

Where applicationId is the id returned from the query above. For example, using the id above:

epmsys_registry updateproperty RA_FRAMEWORK/@applicationId HAVA: 0000000b0000bf00-0000-0000-0acc1bac

e. Restart all services on Server1 and ensure Shared Services and Workspace are accessible

However, in my case, all of these seemed perfect. I stumbled upon my resolution by chance . The Location for the RM1 shared folder (in the configurator, Eg: \\server1\c$\Oracle\Middleware\user_projects\epmsystem1\ReportingAnalysis\data\RM1 stopped being recognized from server 2 . 

The way in which server 1 is resolved from server 2 turned out to be the root cause of my issue.It used to be accessible by the short name earlier and suddenly started to accept only the FQDN owing to some DNS related changes.  In my opinion, We should go with the same servername that we issue in the host file as part of the preinstall. To override that, I added an entry on the server 2 host file and it started working.

EAS SSL Link offloaded at OHS throws a jnlp file

I am on 11.1.2.3.503 and the SSL link for EAS presents me with a jnlp file and it works fine with the 19000 link .

But when you try to access the SSL link, it throws a " administration server does not exist" error.

While this has been discussed in plenty in various KMs DocID1346806.1 . This fix worked perfectly on 11.1.2.2 but on 11.1.2.3 , There is one additional fix that the KM doesn't cover.


The solution is to modify the link wherever you see a reference to the old link, modify that reference to include the 443 port.

Example of these entries are below:

       
       



Now if you save a local copy of these files, if you double click the jnlp it launches the SSL console and you don't see the error anymore.

SSL offloaded at OHS results in smartview RAF not displaying any reports

On a SSL implemented 11.1.2.3 system, which is offloaded at OHS , there is a problem with displaying the list of reports inside the cname. This is a really weird problem as the link works on 19000 .

The end result might look something like this.

According to the DocID1364770.1 , Applying the fix for the step 1 and 2 should fix the issue. But it doesn't , in my case.

Step 1:

• Configure Weblogic to look for specific HTTP headers, amongst which the "WL-Proxy-SSL" header:

1. Start the Weblogic Administration server
2. Log in to the Weblogic Administration Console, then put the Weblogic console into lock/edit mode
3. On the left-hand side, select the top level branch named "epm_system" (i.e. the domain name)
4. Select "Web Applications" on the right hand side
5. Scroll down and enable the option named "Weblogic Plugin Enabled"
6. Save and activate the changes.

Step 2:

• Configure the OHS component to populate the requests with the "WL-Proxy-SSL" HTTP header:

1. On all servers hosting the OHS component, edit the file called httpd.conf (located in \Middleware\user_projects\epmsystem1\httpConfig\ohs\config\OHS\ohs_component) and append the following line at the bottom:
2. WLProxySSL ON
3. Save and close that file.

To resolve the issue, I applied the below fix:

Backup Shared Services repository database 

1. On the server that hosts the Shared Services Registry, open a command prompt. 

2. Go to C:\Oracle\Middleware\user_projects\epmsystem1\bin and run the following commands: 

epmsys_registry addproperty system9/@enable_ssl_offloading true 

epmsys_registry addproperty system9/@external_url_host  (SSL_off_loader_host_name) 

epmsys_registry addproperty system9/@external_url_port 443 (SSL_off_loader_port) 

3: Restart Foundation, Workspace Agent, and Workspace web applications services 

This fixes the issue and the list can now be seen on Smartview

EAS Timeout Issues

I was trying to see fun ways to start working using a disconnected remote session and one thing that caught me in a web was the endless woes with EAS timeout . Each time I had to open a fresh session and it wasn't until recently I stumbled upon this document from the online help.

http://docs.oracle.com/cd/E40248_01/epm.1112/eas_help/frameset.htm?timeout.html  is the link the document.

To fix this issue, We need to unzip the EPM_ORACLE_HOME\products\Essbase\eas\server\AppServer\InstallableApps\Common\eas.ear file

The document encourages us to edit the zip file seperately. However, we can achieve the same using the 'open archive' option in 7zip.

Once this is open , proceed to open the eas.war file

Navigate to the web.xml file present on WEB-INF tree.

Open the file and scroll to the session-timeout section. The default value is 45 and you can change it to the value appropriate to your setup .  In my case , I set it to 180.

The link does not specify any restart required but it is always better to do it. Post this, you should be able to stay logged in to EAS longer.

PDF preview for Hyperion Financial Reports fails to load

I had an issue once with the PDF preview generation for a FR Report on 11.1.2.2.303. It seemed to affect one out of the two servers which had Financial Reporting deployed on it. The html preview turned out fine but the pdf preview was affected. Also, what was strange was it seemed to affect one server and not the other even though they were on the same patched version.

The resultant report produced a view like this,

The sysout log for FR had references to the errors IOException has occured in FOProcessor.setData(ByteArrayInputStream)

There is a KM document 1564844.1 which details 3 methods to resolve the issue. I did not go with the first option as it involved rolling back a patch and the .303 patch contained some important fixes related to my Financial Reporting setup.  I , however,  followed the third method of manually editing the xdo.cfg file. The steps are mentioned below:


1. Log onto the FR Web App Server
2. Navigate to Oracle\Middleware\EPMSystem11R1\products\financialreporting\lib\FOProcessor
3. Backup the existing xdo.cfg file, then delete or rename the file
4. Open the original xdo.cfg file in a text editor
5. Locate the line containing the text  "<property name="system-temp-dir">c:\temp</property>"
6. Comment the line out using XML comment syntax: <!-- <property name="system-temp-dir">c:temp</property> -->   
7. Repeat the above steps on all FR Servers
8. Restart all affected FR Web App services

After this, I have noticed that the preview is generated without any issues.

Weblogic changes cannot be activated - Timeout Error

For a few months now, I have been dealing with a lot of WebLogic 10.3.6 related frustration because I was running into an issue with updating one setting and each time I tried to activate the changes I used to run into an issue trying to activate the changes .

I have a deployment with 4 Managed Servers each of which host 4 Hyperion services load balanced across 4 VMs. These were the preliminary steps I worked with Oracle on. So, I thought these are some of the steps that we can try on our own and help accelerate the resolution process.

1. Shutdown the weblogic domain admin console then move those files out of the Pending folder and then restart the admin console and try to activate the changes again.

2. Navigate to /pending folder. In this folder, there are configuration files which are temporary config files which are cached with changes made from console. At the time of activating changes, the config.xml from the pending folder replaces the original config.xml and other configuration files inside the /config folder.  Kill the Admin Server, and replace the config.xml from the /config folder with the config.xml from the /pending folder. 

Delete all the .lok files inside like edit.lok and config.lok.  Restart Admin Server. 

3. Flush down the changes to be activated using WLST. 

You can use the below script:

4. And of course the classic, Increase Logging level. This can be again perfomed on the config.xml file.

In my case however, none of these methods helped and each time I kept hitting a roadblock . The solution was to update the Listen-address on the config.xml and to repeat the steps for all the managed servers and the components deployed on the mananged server. It is important also to follow a specific order to carry the steps below ,out. 

a. Stop all the Hyperion Services and the Weblogic service.

b. In your admin server, navigate to where the pending directory lies in this case, X:\Oracle\Middleware\user_projects\domains\EPMSystem and rename pending to pending_bkp.

c. Modify your entry in the config.xml file is in the below syntax.

Change the  to include the IP/FQDN to reflect the new entry. So, Post the change,the entry becomes,

d.After this is done for all the managed server components deployed under Weblogic, Start the Weblogic Admin server and the Hyperion Application Services.

You should be able to activate any pending changes now.

CalcMan 11.1.2.2 does not produce updated view for rules on IE browser

I recently ran into this issue with my clients where on a load balanced setup on Windows 2008R2 Platform , The CalcMan rules that they created were at times visible immediately and at times it doesn't produce the updated results.

Initially , I thought it was a browser related issue so I suggested that they clear their browser cache and retry on some attempts it would work and fail some other times. On digging some more, I could isolate the behaviour to one server.On Stopping and Starting Services , I hit this snag on the diagnostics/services log.

java.rmi.MarshalException: failed to marshal create(Lweblogic.rmi.spi.HostID;ILweblogic.cluster.replication.ROID;Lweblogic.cluster.replication.Replicatable;); nested exception is:

java.io.NotSerializableException: com.hyperion.css.common.CSSProvisionedAppInfo

at weblogic.rjvm.BasicOutboundRequest.marshalArgs(BasicOutboundRequest.java:92)

at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:221)

at weblogic.cluster.replication.ReplicationManager_1036_WLStub.create(Unknown Source)

at sun.reflect.GeneratedMethodAccessor4706.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

Truncated. see log file for complete stacktrace

Caused By: java.io.NotSerializableException: com.hyperion.css.common.CSSProvisionedAppInfo

at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1164)

at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)

at java.util.HashMap.writeObject(HashMap.java:1195)

at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

Truncated. see log file for complete stacktrace

>

The solution was  to add this entry to CalcMgrConfig.properties file on both the servers. The normal location for this file is ,

E:\Oracle\Middleware\user_projects\domains\EPMSystem\servers\CalcMgr0\tmp\servers\CalcMgr0\tmp\_WL_user\CALC_11.1.2.0\qxf61x\war\WEB-INF\conf

If the value is already present for DATABASE_CACHE_REFRESHRATE, Change it to 1000.

#Database Caching refresh rate in milliseconds

DATABASE_CACHE_REFRESHRATE=1000

Restart the Calculation Manager Services Post that. It should now produce an updated view on browsers.

Users's last login into the system for Essbase for v11.1.2.2

I am sure time and again various people have posted about how to obtain the user login information from EAS.  Users Last Logon Time post by Celvin provides a good deep dive into how this is achieved across versions.

One method to tune the query Date wise ,published by Celvin is to add ORDER BY "Last Login Date" to the end of the query. So Basically, at the end of the query it should look like,

SELECT username as "User Name", MAX(entrydate) AS "Last Login Date" 

FROM serverlogdetail where username is not null

GROUP BY username ORDER BY "Last Login Date";

However, this query doesn't serve the purpose in the 11.1.2.2 version. 

I researched a little and found the following setting yields the same result as the earlier versions on EAS.
This setting is supposed to populate the Last Login Time" column in the eas console under "Enterprise View > Administration Servers > Essbase Servers > [Hostname] > Security > Users

Add to esbase.cfg the setting: PERSISTUSERATLOGIN TRUE and then restart the Essbase service.

MSAD Settings in Hyperion and their Significance

While configuring Active Directory ( MSAD in this instance) there are quite a few parameters that we overlook and one such time while migrating from 11.1.1.3 to 11.1.2.2,  I used the exact same settings that my prior environment had  and once everything was configured ,while testing , I ran into this issue with users from across the globe got “Invalid login credentials” while logging into Essbase, Workspace or Planning and each time after restarting the entire suite it 

would start working back for a while and would loop back to the same state in some time . 
The logs indicated only numerous references to the Invalid credentials message.
Some good tools that serve as a starting point for investigating the root cause are LDAPBrowser, Wireshark that can help in understanding why this kept happening.

Also, The Oracle Hyperion Shared Services User and Role Security Guide * contains a good write up on the concepts of Active Directory most relevant to Hyperion Foundation. 

Specifically , The section that talks about the DNS lookup.

You can configure Active Directory so Shared Services can perform a static host name lookup or a DNS lookup to identify Active Directory. Static host name lookup does not support Active Directory failover.
Using the DNS lookup ensures high availability of Active Directory in scenarios in which Active Directory is configured on multiple domain controllers to ensure high availability. Whenconfigured to perform a DNS lookup, Shared Services queries the DNS server to identify registered domain controllers and connects to the domain controller with the greatest weight.If the domain controller to which Shared Services is connected fails, Shared Services dynamically switches to the next available domain controller with the greatest weight.

In my case, the Failover Active Directory serving from a different geographic location could not identify the credentials for some reason. So, to restrict the "lookup" to the working Active Directory Site, we had to populate the information for AD Site. The next challenge was to find out the AD Site Information for which running this command is extremely helpful.

nltest [/server:] /dsgetsite 

The result of this command needs to be applied in the AD Site field. Once this setting was applied, the error message stopped appearing. 

java.net.SocketException: Network is unreachable: connect Error for EPMSYSTEM domain

Recently, I faced an issue on the Epmsystem domain on  Weblogic for Hyperion on one server which had APS,EAS, Planning and Calculation Manager installed on it. All of the sysout logs contained no error messages  but none of the services seemed to be funcitoning. While investigating, The Epmsystem.log was full of error messages of this order, and this seemed to repeat for all projects, like in this instance, i am referring to AnalyticProviderServices1  but the error was the same for EssbaseAdministrationService1, Calculation Manager1, Planning1 etc.

#### <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1393517864032> java.net.SocketException: Network is unreachable: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at weblogic.socket.SocketMuxer.newSocket(SocketMuxer.java:374)
at weblogic.socket.SocketMuxer.newClientSocket(SocketMuxer.java:384)
at weblogic.socket.ChannelSocketFactory.createSocket(ChannelSocketFactory.java:86)
at weblogic.socket.ChannelSocketFactory.createSocket(ChannelSocketFactory.java:56)
at weblogic.net.http.HttpClient.openServer(HttpClient.java:323)
at weblogic.net.http.HttpClient.openServer(HttpClient.java:425)
at weblogic.net.http.HttpClient.New(HttpClient.java:252)
at weblogic.net.http.HttpURLConnection.connect(HttpURLConnection.java:213)
at weblogic.cluster.HTTPExecuteRequest.connect(HTTPExecuteRequest.java:63)
at weblogic.cluster.HTTPExecuteRequest.run(HTTPExecuteRequest.java:82)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
>

The root cause was found to be that the machine though dualstacked and supporting just Ipv4 was somehow binding to the IPv6 Address of the machine.

Resolution - In the SetDomainEnv.cmd file, force the usage of IPv4 on the machine. 

"-Djava.net.preferIPv4Stack=true" is the keyword to be added to the JAVA_OPTIONS Section. An example of the usage will be, 

set JAVA_OPTIONS=%JAVA_OPTIONS% %JAVA_PROPERTIES% -Dwlw.iterativeDev=%iterativeDevFlag% -Dwlw.testConsole=%testConsoleFlag% -Dwlw.logErrorsToConsole=%logErrorsToConsoleFlag% -Djava.net.preferIPv4Stack=true

The Original line will not contain the -Djava.net.preferIPv4Stack=true is not present. After this, a complete restart of services is needed and this will help in resolving the issue.

DELETING A CORRUPT PLANNING APPLICATION FROM EPM REGISTRY

A planning application, when it goes corrupt/missing,  can cause a lot of angst. This is one of those occasions where one can truly appreciate the beauty of the epmregistry utility.

We can invoke this utility from this location
MIDDLEWARE_HOME\user_projects\instance_name\bin directory  

In order to first begin , generate a registry report the output can be viewed in the diagnostics/reports directory.

Firstly proceed to delete the instance :

Delete Instance File
epmsys_registry removefile PLANNING_PRODUCT/@APPLICATIONNAME.instance

Now, get to the planning application:

Delete PLANNING  application
epmsys_registry deletecomponent #APPLICATIONNAME.HP

Post this , it should be a relatively straight forward approach of recreating the planning application .The best part about using this method is that you are able to give the application the same name post corruption. Not sure if that would have been possible in earlier releases.

How to deal with cwallet.sso corruption for EAS Service

I recently faced an issue with an EAS service which failed to startup properly. The eas_sysout.log was full of error messages like what is shown below:

Caused By: oracle.security.jps.service.credstore.CredStoreException: JPS-01050: Opening of wallet based credential store failed. Reason java.io.IOException: PKI-02002: Unable to open the wallet. Check password. 
at oracle.security.jps.internal.credstore.ssp.CsfWalletManager.openWallet(CsfWalletManager.java:184)
at oracle.security.jps.internal.credstore.ssp.SspCredentialStore.doInit(SspCredentialStore.java:239)
at oracle.security.jps.internal.credstore.ssp.SspCredentialStore.(SspCredentialStore.java:153)
at oracle.security.jps.internal.credstore.ssp.SspCredentialStore.(SspCredentialStore.java:139)
at oracle.security.jps.internal.credstore.ssp.SspCredentialStoreProvider.getInstance(SspCredentialStoreProvider.java:115)
Truncated. see log file for complete stacktrace

This indicates a corruption of the cwallet.sso file . The reasons behind the corruption of the file could be anything.  We should check for permissions and what kind of access is present on the file and the folder containing the file. Typically, Administrators must have ‘full control’ on the file and the folder. If that is set right and the service still doesn’t start up, look up the size difference of the cwallet.sso  in the

MIDDLEWARE_HOME\user_projects\domains\domainname\config\fmwconfig  and the bootstrap folder . If there is a difference, obviously something is wrong in the file present in the fmwconfig. Rename and replace with the file in the bootstrap folder and try and start up the service. If it still errors out, Consider this alternative:

Create a new location for the cwallet.sso that the EAS is going to use: EPM_ORACLE_HOME\products\Essbase\eas\server.

The KM article 1447461.1 , provides a detailed step by step account of how to achieve this.

Follow the steps indicated in that article and grant full permissions on the cwallet.sso regenerated once the service is started up. That should help in resolving the issue. 

How to sync time across servers

I cannot stress enough on the importance of the need for all the servers in the Hyperion stack to be in sync.

In any traditional setup one will mostly see , a database server , all hyperion application servers, a server for programs to pull/push data from/to hyperion systems.

If any of these servers are out of sync in time due to ACL issues or networks issues , the impact is felt widely on the Hyperion side of things.  Just to sight a few examples, Users could face issues logging into the system or programs that run on hyperion might not run at the intended time or even produce erroneous results depending on the nature of the program. 

I am about to cover how to achieve a sync of time data with the NTP Server. 

On Windows operating system , the w32time utility proves very effective in achieving the same. 

net stop w32time

w32tm /config /syncfromflags:manual /manualpeerlist:"NTP.SERVER.COM"

w32tm /config /reliable:yes

net start w32time

w32tm /query /configuration

How to Renew SSL Certificate using Oracle HTTP Server Wallet Manager

In my current setup I have an SSL Certificate valid for a year and every year it requires renewal of the certificate.There are a couple of good KMs on metalink that can be followed to carry this out. Some of them that serve as a good starting point are:

Doc ID 1530169.1
Doc ID 317405.1
Doc ID 1391487.1 

I am going to mention briefly what I had to do to achieve this in my environment:

1. Login to the Wallet Manager and Wallet-> Open . You will be prompted with the 'default directory does not exist . Would you like to continue?' . Select 'Yes' and navigate to the location of the wallet and type in the password.

2. Inside the wallet tree, Right click and "Add Certificate Request" . Enter the details for the certificate for which extension is required and Save it. Export the CSR using "Export User Certificate" option.

3. Send the CSR to your security admin and once it is signed and sent back, "Import User Certificate" and this should take care of the part.
Note- In some cases, you might encounter an error because the intermdiate and the root certificate are not present.  These will need to be imported prior to importing the user certificate. 
Save your wallet at this stage.

4. Move on to the certificate that is about to expire and  Remove that user Certificate and Save your wallet. Restart OHS at the end of everything. 

These should help you in effectively renewing the certificate.

Upgrade from MDM to DRM in 10 steps

1. Download Software from Oracle for DRM 11.1.2.2.302  . It can be installed on Windows 2003 and 2008R2.  I recommend 2008 R2 for best results.

2. Login as admin user/creator

3. Create a Schema with SYSDBA rights

4. Run the software and install DRM product. This step is quite straightforward.

5. Run the Repository Wizard.
 a. Provide Source Connection Information
 b. Provide Target  Connection Information
 c. Provide UNC path information
 d. Wait for Repository creation information

6. Click on Upgrade.

7. Create an Application and confirm the configuration with the drm_admin user and Save it.

8. Start the service.

9. Place Web client on Desktop and allow access for essbase users.

10. Login to Web Client to verify the hierarchy . Default login id is admin/Welcome!

Essbase Agent Issue on 11.1.2.2.102

I have Essbase running on 11.1.2.2 upgraded from 11.1.1.3 on Windows 2008 R2 64 bit platform.
On 11.1.2.2 , I would often run into an issue with the Essbase Agent suddenly starting to freeze and would not allow any new requests.
On the MaxL Client, Once I logged in , There would be no response . It would just stay right there and nothing would happen.

The Essbase.log file would not populate any new information at that moment the freeze occurs and previous messages would contain numerous references to the following 2 messages:


Local/ESSBASE0///14460/Warning(1051003)
Error -1 processing request [Multiple Async Logout] - disconnecting

Local/ESSBASE0///14460/Error(1042013)
Network error [10053]: Failed to receive data

NETDELAY AND NETRETRYCOUNT tweaking did not help much.


Root Cause:

Issue found Closing and re-initialize the mutex while it is in use during certain race conditions.T

Solution:

PSU 11.1.2.2.104

WebAnalysis Installation on a Patched EPM Platform

I had a unique situation present itself in front of me when business wanted me to install WebAnalysis on top of a complete with patches, pretty looking platform.  I got to it with much gusto and was able to get through with the installation smoothly.

The Configuration went fine as well except for the "Reregistration of Scripts",which failed. This is essential for creation of the WebAnalysis Tables in the BI+ Schema.

That didn't preven't me from logging into WebAnalysis Studio however and when I tried to create a connection and save it, it would never happen and the java console gave messages like

Java Plug-in 1.6.0_29

Using JRE version 1.6.0_29-b11 Java HotSpot(TM) Client VM

User home directory = C:\Users\XXXXXX

----------------------------------------------------

c:   clear console window

f:   finalize objects on finalization queue

g:   garbage collect

h:   display this help message

l:   dump classloader list

m:   print memory usage

o:   trigger logging

q:   hide console

r:   reload policy configuration

s:   dump system and deployment properties

t:   dump thread list

v:   dump thread stack

x:   clear classloader cache

0-5: set trace level to

---------------------------------------------------

load: class com.hyperion.analyzer.usersession_base.AnalyzerClient not found.

java.lang.ClassNotFoundException: com.hyperion.analyzer.usersession_base.AnalyzerClient

at sun.plugin2.applet.Applet2ClassLoader.findClass(Unknown Source)

at sun.plugin2.applet.Plugin2ClassLoader.loadClass0(Unknown Source)

at sun.plugin2.applet.Plugin2ClassLoader.loadClass(Unknown Source)

at sun.plugin2.applet.Plugin2ClassLoader.loadClass(Unknown Source)

at java.lang.ClassLoader.loadClass(Unknown Source)

at sun.plugin2.applet.Plugin2ClassLoader.loadCode(Unknown Source)

at sun.plugin2.applet.Plugin2Manager.createApplet(Unknown Source)

at sun.plugin2.applet.Plugin2Manager$AppletExecutionRunnable.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

The following was the solution:

1. Update 'EPM_ORACLE_HOME\upgrades\webanalysis\update_registry.bat' script:
"%JAVA_HOME%\bin\java" -DEPM_ORACLE_HOME=%EPM_ORACLE_HOME%
"%JAVA_HOME%\bin\java" -DEPM_ORACLE_HOME=%EPM_ORACLE_HOME%

Replace:

-DEPM_ORACLE_INSTANCE=%EPM_ORACLE_INSTANCE% %ODL_PROP% %LOCALEPROP%

com.hyperion.analyzer.upgrade.HYARegistryMigrator %SOURCE_SYSTEM% "  with

-DEPM_ORACLE_INSTANCE=%EPM_ORACLE_INSTANCE% %ODL_PROP% %LOCALEPROP%

com.hyperion.analyzer.upgrade.HYARegistryMigrator  "

and rerun the EPM System Configurator. The "Process Reregistration Script" portion should go through perfectly after this.

2. Run analyzer_upgrades.sql line by line on your BI plus schema.

3. Run your Web Server Config as the last step .

How to set DB Schema password to 'never expire'

As a Hyperion admin, I have time and again run into this issue with my hyperion schemas where the password sometimes expires and wreaks havoc on the application.

The key indicator for these error messages are the log files. They are quite descriptive .

Basic test to make sure that the schema is working correctly will be to login to the DB hosts and issue this command:

conn username/password;

To resolve this issue ,

connect to your DB as sysdba and run

ALTER PROFILE DEFAULT LIMIT PASSWORD_LIFE_TIME UNLIMITED;


Restart all Services and ensure all the log files under diagnostics\logs\services directory startup properly.

This should resolve the issue.

Update/Replace Hyperion MSAD with new MSAD

 I had an organizational need for updating my current active directory and replace it with a new active directory.
We have a rather simple structure in our provisioning data, mostly controlled by native groups and AD users being part of those groups.

These are the steps I followed to make that happen:

1. Export a good working copy of current Foundation project from LCM. To do this, navigate to Application groups->Foundation->Shared Services->Native Directory. Highlight everything and export it to the server.
2. Open Groups.csv and scroll to #groups_children tree section.
A typical section will look something like this.

#group_children
id,group_id,group_provider,user_id,user_provider
Sample Group,,,Username,DomainNameX

Do a find and replace for DomainNameX with DomainNameY
and save the file. Repeat for all the files inside the Export.

3.  Login to Shared Services and navigate to Administration->Configure User Directories and add the information for the New MSAD

4. Verify that the old MSAD is moved to search order 3 and is "Excluded" . Save the Settings.

5. Perform a full restart of all the products.

6. Login as admin to Shared Services and Import the file updated in Step 2.

7. You should now find updated provisioning information on Shared Services.

Spl Note for Planning Apps:

You will need to "Refresh Database->Security" for each planning application. Otherwise you are likely to encounter 'unable to sync provisioning error' when you login to planning.

Hope this helps.