Remember when Mid-Tier was on version 7.5? This was one of those errors that kept coming up, no clear reason or root cause, it just happened. The error was difficult to reproduce, it wouldn’t show up in any logs and for some reason it only happened in your production environment.
I wasn’t sure if I should write about this, it’s an error which rarely comes up anymore due to a different approach in more recent Mid-Tier versions, but then again that’s of course assuming you did upgrade. That’s not always the case for everyone. Furthermore, if you understand the error message you also understand how the system deals with something that can be quite complicated: HTTP cookies.
Mid-Tier uses HTTP cookies in a similar fashion. We store session data, timeout information, but also data from global fields. Those are the fields that can be used between various forms. We just store the data in a cookie and it keeps referring to it. In the same session it will always be valid.
Now here’s where that error comes in: the available space to store HTTP cookies is limited. There’s only so much space available and if you store too much it’s not going to fit. That’s when we throw that error. What we’re saying is: stop, you’ve reached the maximum available space, it’s not going to fit anymore. Agreed, that’s not clear from the error description. What it says is ‘The current global field values are too large to be stored’ which tends to steer people in the wrong direction. Yes, global fields are stored as HTTP cookies and could cause this problem, but that’s not necessarily the root cause. Before you start reviewing your use of global fields, let’s have a look at the browser’s capabilities of storing cookie data.
Every browser has a maximum amount of data that can be stored in cookies per domain. With domain I mean the context, so it’s the amount of data per /arsys or /midtier. In 7.5 our code assumes that max is 4096 bytes. If you go beyond this you get the error. What you need to do is first understand what’s stored in that space. Even including global field data, the amount of cookie data usually doesn’t take up half of that space, so there’s something else stored there.
My tool-of-choice is Fiddler, the web debugger I’ve mentioned before. Run it alongside your session and wait until the error happens. This is the log of a test system where everything is okay. What I’m doing here is counting the size of the HTTP cookies in bytes.
Notice the size of the cookies which I highlighted. I’m checking every 5 to 10 sessions and see if it increases and if so, what rate. In this case it’s well below 4096 bytes, so the error won’t happen.
Here's another screenshot, this time a server which uses SSO to authenticate. Notice it's the cookie value is a lot higher.
Keeping this in mind, let’s look at a different log where the error did happen. I’m doing the same thing, I go through it every 5 to 10 sessions and check the HTTP cookie size.
Again it’s very high and getting even higher. In the last screenshot it’s 4181 bytes. That’s more than the max of 4096 bytes. The error is therefore expected. Notice you won’t actually see the error in the logs (at least not directly), you need to check the cookie data. This happens because it’s a client error, not server. It won’t appear in any of the conventional log files as such.
Let’s check what’s in there. Why it is so much bigger than my first test? Let’s copy the cookie data and have a quick look at them:
I know it's a bit fuzzy (I had to hide a lot of the details), but notice there are lot of extra HTTP cookies. If I compare this to my first test I immediately notice of lot of the data is not Mid-Tier related. In this case the Mid-Tier related cookies take up less than 33%. Even the screenshot with SSO shows a significant increase. Why is this? This is happening because there are other components like web optimisers or SSO plugins which also use the same space to store their cookies. They can’t avoid this, just like Mid-Tier they need to have cookies stored under the same domain. But that’s a problem here, because the combined size of Mid-Tier and the other components exceeds the max.
Can you reduce these? Probably not. You can’t really reduce the amount of cookies Mid-Tier uses. The same goes for the web optimiser (or your SSO plugin or proxy, or whatever component stores the data).
We recognised this problem and in 7.6.04 we’ve redesigned this. Rather than relying on HTTP cookies to store everything we started using Flash cookies. If it’s small enough store it in HTTP cookies, if it’s too big store it as Flash. The biggest problem here was that we couldn’t control Flash cookies. The management is up to Flash and this has some draw backs. Data didn’t always get deleted, there were performance problems, etc. It wasn’t great.
So in 8.1 we came up with a better solution. Rather than using Flash we decided to use DOM storage. The principle is the same, small stuff is stored as HTTP cookie, bigger pieces of data are pushed to DOM cookies. The cookie management is well within our control.
So that’s probably why we haven’t seen the error being mentioned in recent Mid-Tier versions. But what to do if you’re still on 7.x and can’t upgrade? Remember I mentioned that the max we check for is 4096 bytes? That’s actually only the maximum size for IE6 and IE7. The real maximum cookie size is different depending on the browser:
- IE6: 4096
- IE7: 4096
- IE8+: 10234
- Firefox: no max
- Chrome no max
The 7.5 code doesn’t take this into account so if you can’t upgrade Mid-Tier, consider adding some customisation. I wrote some code to check what browser you’re on and apply a different max. So as long as you’re on IE8 or higher (or a different browser) this will get you around the problem. See KA361590 for more details.
Right, that's all from me again. Until next time,