Loadbalanced EAS 11.1.2.3 throws Login Error

When I first installed 11.1.2.3 on a load balanced instance , I had many challenges. One of the components that has given me problems consistently has been the Essbase Administration Services, EAS . I have covered various posts on 11.1.2.3 EAS  here,here and here . I am adding one more to the collection with this post.

What happened was that each time I would login to EAS after a restart, it would be fine. But the subsequent problems always resulted in a login error.

I found these entries on the easserver.log

[EssbaseAdminServices0] [WARNING] [] [oracle.epm.essbase.eas] [tid: 12] [userId: ] [ecid: 0000KWxWr6H5IfYTTqxkWc1K1ssg000003,0] [APP: EAS#11.1.2.0] Source directory does not exists
[EssbaseAdminServices0] [WARNING] [] [oracle.epm.essbase.eas] [tid: 12] [userId: ] [ecid: 0000KWxWr6H5IfYTTqxkWc1K1ssg000003,0] [APP: EAS#11.1.2.0] Failed to copy storage directory to new location
[EssbaseAdminServices0] [ERROR] [ESSEAS-35505] [oracle.epm.essbase.eas] [tid: 11] [userId: ] [ecid: 0000KWxb0TM5IfYTTqxkWc1K1ssg00001a,0] [APP: EAS#11.1.2.0] Following Exception occurred for user admin. Message exception is EPMCSS-00301: Failed to authenticate user. Invalid credentials. Enter valid credentials.
[EssbaseAdminServices0] [ERROR] [ESSEAS-35505] [oracle.epm.essbase.eas] [tid: 11] [userId: ] [ecid: 0000KWy029F5IfYTTqxkWc1K1ssg000048,0] [APP: EAS#11.1.2.0] Following Exception occurred for user admin. Message exception is EPMCSS-00301: Failed to authenticate user. Invalid credentials. Enter valid credentials.
[ERROR] [ESSEAS-35505] [oracle.epm.essbase.eas] [tid: 11] [userId: ] [ecid: 0000KX1Ie7Y5IfYTTqxkWc1K286Q00000k,0] [APP: EAS#11.1.2.0] Following Exception occurred for user admin. Message exception is java.lang.NullPointerException

In the SharedServices_security_client.log , I saw the following error :

[EssbaseAdminServices0] [ERROR] [EPMCSS-09223] [oracle.EPMCSS.CSS] [tid: 12] [userId: ] [ecid: 0000KX1E_JR5IfYTTqxkWc1K286Q000002,0] [APP: EAS#11.1.2.0] [SRC_CLASS: com.hyperion.css.spi.CSSManager] [SRC_METHOD: updateDcUrl] Failed to retrieve base DNs while updating dcUrl. Error communicating with server. DNS name not found [response code 3]. Invalid Domain, AD Site or DNS Server.

This normally indicates there is an issue with the CSS configuration. It then dawned upon me that I had messed up my search order while configuring the MSAD. I placed my MSAD in the order 1 and the Native Directory in order 2. I reversed the order and followed it up with a complete EPM system restart. The issue was resolved.

Hyperion EAS stops accepting logins after a while

I have a Hyperion 11.1.2.2 instance which has been running for a while now. Since EAS is not used as much , I never knew of this issue until a user reported it. This user kept getting "Login Failed" and it kept some days after restarting the service. On some deeper analysis, I found that the rootcause of this issue was the location tmp folder for EAS .  We can apply the below fix to resolve the issue.

On the server where EAS is installed, navigate to the user_projects level.

Here we can find there is a folder called tmp. Create a new folder by some name, let's say eas_tmp

Go to the registry, and update the setting corresponding to Djava.io.tmpdir as shown below

Restart EAS Server. This error should now stop appearing.

Managed Servers don't talk to Admin Server

I have a network deployment for Hyperion BI+,Calc Man, Planning and EAS where all these components are load balanced with the exception of Essbase and it is a distributed server install, 

I ran into an issue with Weblogic where it displayed the below symptoms:

1. Hyperion Planning, EAS, CalcMan, and other things not running on the Admin Server Machine take FOREVER to startup.

2. Once started, AdminServer still shows all the remote nodes as DOWN

3. Errors in the managed server sysout files show managed servers were trying to talk to the wrong IP address to get to the Admin Server

(it was trying to use the VIP address, and not the address associated with the FQDN for the server.

On digging further, I found that the servers are multihomed, mainly due to an odd load balancer setup where there are local loopback addresses on the

admin server that are really hosted by the VIP on the LB. The managed servers were able to make an initial connection to the

admin server, and the admin server responded back with the WRONG IP (the loopback, not the public IP).

The resolution is to set the LISTEN address on the WL AdminServer "Server" to the public DNS name. example.host.domain.com

For this, Login to the WL Console and click Lock & Edit (see Use the Change Center).

On the Servers page, click the name of the server , Select Configuration > General.On the Severs: Configuration: General page, enter a value in Listen Address.

Restart AdminServer,wait for it to be fully running, Restart all the other services. 

WebAnalysis SSL Link for 11.1.2.2 on a patched platform

When I first installed 11.1.2.2 , there wasn't a need for a WebAnalysis solution. Later, when there was a need , the install wasn't straightforward. Read my previous post on how to fix the installation problem here . I could get through with my installation but however, launching WebAnalysis on SSL became a problem. The 16000/19000 link would launch fine but not 443. I enabled debug trace on and saw these messages on the java console.

For some reason 16 jar files and jre-6u22-windows-i586-s.exe are missing from WebAnalysis.ear file.

These files are present in %HYPERION_HOME%\common\epmstatic\webanalysis folder where as the static folder in WebAnalysis.ear located under ' %HYPERION_HOME%\products\biplus\InstallableApps file doesn't have them.

Copy those files to the tmp directory of Weblogic server as seen below.

C:\Oracle\Middleware\user_projects\domains\EPMSystem\
servers\WebAnalysis0\tmp\servers\WebAnalysis0\tmp\_WL_user\
WEBANALYSIS_11.1.2.0\ts1hx4\war\static

If you are interested in a permenant work around/fix, then you can add the jar files and jre-6u22-windows-i586-s.exe to the WebAnalysis.war file.

Open WebAnalysis.ear file with 7Zip or Winrar
Open WebAnalysis.war file and navigate to static folder
Compare the file present in %HYPERION_HOME%\common\epmstatic\webanalysis against static folder and add missing ones.


BUG INFO:

Web Analysis Studio Unable to Launch When Using Default 16000 Port Error: "class com.hyperion.analyzer.usersession_base.AnalyzerClient not found." [ID 1365054.1]

It says "The ability to login to WA using the listening port:16000 has been removed by development.
This is documented in unpublished (documentation) Bug 13061357"