1 of 1 people found this helpful
ok...I'm not 100% sure where to look, but I can tell you what's 'not' happening....I'm sure Doug Mueller will have a finite answer for you though.
So, based on the error message, I don't think that your LB is bouncing you between MT's. That would give you a 'session is invalid' error, which you aren't getting...so that indicates to me that your sticky bit on the MT LB seems configured properly from that perspective.
The App server doesn't have 'sessions' per se...so I don't expect the other LB to be an issue either.....
So...to prevent you from accessing the Remedy from multiple systems at the same time, BMC generates a unique ID, that's not based on IP...but in some fashion follows the machine uniquely, to prevent you from using another machine. So, somehow this uniqueID is getting changed between the client and the MT server.
I'm assuming that when you hit the MT directly, you don't have any problems?
2 of 2 people found this helpful
The unique ID that LJ refers to is owned by your client machine. When a session starts up, we create and store an identifier on the client. This allows for other processes running from that same client to all come in as being on the same system. If you start two different browsers and connect (IE and Firefox for example), that would be two connections, but they should be sharing the same identifier so that although there are two programs accessing the environment, they are from the same system, so there is no issue.
If you have connections coming in with different identifiers, then the system believes you have access coming from different machines -- and you get the error noted above.
Now, the basic configuration you have described is fine. We have seen some issues with F5 load balancers where after periods of inactivity closes out the connection so that a new connection comes in next time you access and with that, you get "rebalanced" so you may be directed to a new system. But, as LJ indicated, you should get an invalid session issue there as the new midtier doesn't know about you (in the future we will migrate sessions so this issue goes away and you will not need the sticky bit above midtiers either). Now, is this an issue for now or later, I don't know. There is something about a "keep alive" setting that prevents the F5 load balancer from tearing down the connection. But, this is an aside and does not seem involved in the current situation.
Is it the same users all the time? Do they get the error every time? Soon after starting? Random?
If the same users from the same system and all the time, then we start looking for something environmental that is preventing them from getting/using the ID generated. I think it is shared using a cookie so if the user was configured to not allow cookies to be stored, we have a problem (not only with this but some other featues as well). Unlikely this is the issue, just an example of an environmental thing that can cause problems.
This is not a problem that customer have in general. A bit more information is needed about whether it is the same users, all the time or sometimes, on the same systems. Things like if the same users, have they tried logging into a system that is one that other users are are logging into without issue and how do things go there? What if someone who has no problem elsewhere, logs out and then tries to log into the system the user with problems is using (make sure they log out or they will definitely get the error of course). If no consistency to who gets the
errors, then no need to try any combinations like above of course.
We are seeing the error with random users and has been difficult to reproduce. That makes it harder to get logs for support.
I just got an update from support that the Midtier Load balancer needs to be cookie based. I checked with our network engineers and they have indicated its IP based. They are willing to change it, but would like to know if its:
Is there a specific cookie name? What else needs to be provided to set this up
Also could this be the issue?
Appreciate your time
So we made the change for session affinity to be cookie based and were also advised to add an iRule to the Midtier load balancer. After making those changes none of our users could login. It would keep them on the login page, almost like it immediately disconnected them after they logged in.
What we also noticed is if a user is logged in from one desktop and he logs out. Goes to another desktop and logs in he gets the "Connected from another machine" error. It looks like in some cases the old session is not released even when the user has logged out.
First, I am going to suggest working with the support team on this issue as things get rather involved in interactions and it is important to be able to go through specific use cases and patterns.
Now, on to your comments...
I am not an expert on load balancers so I can only offer what seems from the outside to be true and it is not with expertise on load balancers themselves.
With that said, I don't believe the issue being related to the F5 configuration to IP vs. Cookie. In fact, I would expect things to work with IP and not with Cookie... Why? Because there is no cookie involved in the interaction between the browser and the mid-tier. There is just traffic. How could it key off a cookie? Where would it get that cookie? The only thing the load balancer has is the IP address and the packet that is being sent.
So, I don't understand the cookie method vs. the IP method, but someone would have to tell me where the cookie comes from and how that works and I just don't see how there is data to work with.
Again, an expert with load balancers may have information that clarifies things and makes it make sense. But, as you tried and it didn't work....
Second, the comment that is standing out is your comment that you had a user LOG OUT and then go to another machine and log in and they got the message about being logged in elsewhere.
So, let's explore this. Did the user login, open one window, hit a logout button to do an explicit logout, get the logout screen, then go to another machine in try to login and go the error? Definitely not expected and not the behavior we see when trying this. I am going to assume that there is something more involved than the simple test that I noted above. Did they run multiple tools? Did they just close the browser and immediately tried to login to the other machine -- there is a delay from closing the window to a logout occurring that may be a minute or two. Were there multiple browser windows open -- a logout should logout the session, but just checking.
Whatever triggered the issue in the logout, login to another system scenario seems like the thing to be exploring here. If that is reproducible, working with support on that situation is your best bet.
I opened a ticket with support and they could reproduce the issue with the floating license (app and AR) not being released and have provided a hotfix. I am monitoring the user logs to check all users who are getting this if they have a floating token. That should hopefully identify the pattern. We have already implemented the hotfix on our dev server and are testing now. I will post more details shortly.
Thanks for your inputs they have been very helpful.
Can you please give me the BMC issue ID or Defect ID under which hot fix is delivered?
I do not have a defect id. I would advise you to log a ticket with BMC support if you are noticing that a user who logged out is still visible in the license review table in the AR System Administration Console. This only happens if the user is assigned any type of floating license tokens and they use the SHR:LandingConsole as the home page. The fix is to replace 2 jar files on the AR System application server.
Hi Saby Fernandes..
We are tying to configure F5 load balancer in our environment only in the midtier server end, not on the AR Server.
Is there any specific configuration that need to be performed in our midtier servers in order to configure the load balancer? I am new to load balancer configuration and any inputs will be of much help.
We have 2 mid-tier servers and the load balancer ip is configured in host file of the midtier servers for now.
The iRules are set at F5 load balancer, however, the connection is not established between the load balancer and Midtier. We do not use IIS or Apache. It is just Tomcat. From load balancer we are able to ping the midtier servers and there are no firewalls between loadbalancer and midtier.
Please advice, if we are missing anything here.
Thanks in advance.
Are you seeing any error messages?
The usual setup would involve a Load Balancer before the mid-tier servers and a load balancer between the mid-tier servers and the ar servers.
Optimum setup would be
Client/browser -> LB(1) -> Mid-Tiers -> LB(2) -> AR App Servers -> LB(3) -> DB Servers
You did not indicate what version of Remedy AR and mid-tiers you are using. For 8 onwards you need to setup as follows:
LB(1) before the mid-tiers enable sticky bit
LB(2) between mid-tiers and AR App disable sticky bit.
Also in the mid-tier configuration when giving the name of the AR Server, use the name of the LB between mid-tier and app servers. In the depiction above it would be the name of LB(2).
Thanks for the prompt response. We have only one app server, so no loadbalancer in between the app and mid-tier servers. We are just trying to configure the load balancer front of our 2 mid-tier servers. We are using 7.6.04 ARS and Midtier.
We could not find any errors in the mid-tier logs as well in the load balancer side. Is there any specific configuration required to connect the LB with the mid-tier servers from Remedy end? Please advice.
Do you see the query in the Tomcat Access logs, does he even try to "get" a page?
Btw, if you directly access the Tomcat, without the LB, does it work?
No. We could not find any request comming to access the page in any of the midtier and tomcat logs.
Yes, when we try to login directly to our midtiers, we are able to access remedy without any issues.
Check again with your LB team that queries are indeed forwarded to the right server and port then...