MSAD Settings in Hyperion and their Significance

While configuring Active Directory ( MSAD in this instance) there are quite a few parameters that we overlook and one such time while migrating from 11.1.1.3 to 11.1.2.2,  I used the exact same settings that my prior environment had  and once everything was configured ,while testing , I ran into this issue with users from across the globe got “Invalid login credentials” while logging into Essbase, Workspace or Planning and each time after restarting the entire suite it 

would start working back for a while and would loop back to the same state in some time . 
The logs indicated only numerous references to the Invalid credentials message.
Some good tools that serve as a starting point for investigating the root cause are LDAPBrowser, Wireshark that can help in understanding why this kept happening.

Also, The Oracle Hyperion Shared Services User and Role Security Guide * contains a good write up on the concepts of Active Directory most relevant to Hyperion Foundation. 

Specifically , The section that talks about the DNS lookup.

You can configure Active Directory so Shared Services can perform a static host name lookup or a DNS lookup to identify Active Directory. Static host name lookup does not support Active Directory failover.
Using the DNS lookup ensures high availability of Active Directory in scenarios in which Active Directory is configured on multiple domain controllers to ensure high availability. Whenconfigured to perform a DNS lookup, Shared Services queries the DNS server to identify registered domain controllers and connects to the domain controller with the greatest weight.If the domain controller to which Shared Services is connected fails, Shared Services dynamically switches to the next available domain controller with the greatest weight.

In my case, the Failover Active Directory serving from a different geographic location could not identify the credentials for some reason. So, to restrict the "lookup" to the working Active Directory Site, we had to populate the information for AD Site. The next challenge was to find out the AD Site Information for which running this command is extremely helpful.

nltest [/server:] /dsgetsite 

The result of this command needs to be applied in the AD Site field. Once this setting was applied, the error message stopped appearing. 

java.net.SocketException: Network is unreachable: connect Error for EPMSYSTEM domain

Recently, I faced an issue on the Epmsystem domain on  Weblogic for Hyperion on one server which had APS,EAS, Planning and Calculation Manager installed on it. All of the sysout logs contained no error messages  but none of the services seemed to be funcitoning. While investigating, The Epmsystem.log was full of error messages of this order, and this seemed to repeat for all projects, like in this instance, i am referring to AnalyticProviderServices1  but the error was the same for EssbaseAdministrationService1, Calculation Manager1, Planning1 etc.

#### <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1393517864032> java.net.SocketException: Network is unreachable: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at weblogic.socket.SocketMuxer.newSocket(SocketMuxer.java:374)
at weblogic.socket.SocketMuxer.newClientSocket(SocketMuxer.java:384)
at weblogic.socket.ChannelSocketFactory.createSocket(ChannelSocketFactory.java:86)
at weblogic.socket.ChannelSocketFactory.createSocket(ChannelSocketFactory.java:56)
at weblogic.net.http.HttpClient.openServer(HttpClient.java:323)
at weblogic.net.http.HttpClient.openServer(HttpClient.java:425)
at weblogic.net.http.HttpClient.New(HttpClient.java:252)
at weblogic.net.http.HttpURLConnection.connect(HttpURLConnection.java:213)
at weblogic.cluster.HTTPExecuteRequest.connect(HTTPExecuteRequest.java:63)
at weblogic.cluster.HTTPExecuteRequest.run(HTTPExecuteRequest.java:82)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
>

The root cause was found to be that the machine though dualstacked and supporting just Ipv4 was somehow binding to the IPv6 Address of the machine.

Resolution - In the SetDomainEnv.cmd file, force the usage of IPv4 on the machine. 

"-Djava.net.preferIPv4Stack=true" is the keyword to be added to the JAVA_OPTIONS Section. An example of the usage will be, 

set JAVA_OPTIONS=%JAVA_OPTIONS% %JAVA_PROPERTIES% -Dwlw.iterativeDev=%iterativeDevFlag% -Dwlw.testConsole=%testConsoleFlag% -Dwlw.logErrorsToConsole=%logErrorsToConsoleFlag% -Djava.net.preferIPv4Stack=true

The Original line will not contain the -Djava.net.preferIPv4Stack=true is not present. After this, a complete restart of services is needed and this will help in resolving the issue.