Recently we managed to get a customer running WebSphere Commerce V6 on WebSphere 6.1 (had to upgrade from V6), in conjunction with WebSphere Extreme Scale 7.1 It had been a long path and far from as simple as things perhaps made out, just plug it in and away you go not quite like that. I had written an initial post on setting up WebSphere Extreme Scale and the environment we used, not perhaps realising how far away we were at that point from running this live.
There was still a lot of learning and some quite major changes to make, that you don’t really don’t come across when you use Dynacache within the WebSphere Application Server server for a Commerce environment.
If you are wondering what is Dynacache and you use WebSphere Commerce then it is important that you understand, not running Cache in a Commerce environment is a little like having a fast car and only using your first three gears maybe. What is possible will always be limited and will not do the environment any good at all, affecting the end users of the site. You will never really maximise the performance that is possible from your Commerce environment, have a look at this Redbook to get an understanding ‘Mastering Dynacache in WebSphere Commerce‘. I have also split this blog post into two parts, just so we do not end up with a huge page to read and get bored in, perhaps it could even be three.
The aim of WebSphere Commerce and WebSphere Extreme Scale
This article is not about the details of WXS there are some excellent pieces available such as ‘why WXS‘ for that, especially from Billy Newport. There is also a Elastic Cache redbook which gives some excellent background and also some WC integration options plus part of its team was Jonathan Marshall, who helped us along the WXS path.
For the environment we decided to go for WXS 7.1 and have kept that up to date as well with the latest fixpacks so 7.1.0.2 is the current version. We then went for the standalone process model rather than using WebSphere to provide the WXS grid, that model is discussed in the redbook. There are advantages to running on WAS, but it seemed to make sense plus save on licences just using the standalone process mode for the WXS grid on Ubuntu (see previous blog for setup).
The aim of integrating WXS with Commerce was to improve several areas.
- Capability to hold more cache items as the products available was going to grow substantially on the stores.
- Reduce the overhead on the application servers when dealing with the cache, such as disk offload and the space the cache takes up within the JVM.
- Improve startup times for servers when coming on line as the cache would already exist rather than having to build.
For the grid setup we initially have three machines each running a WXS catalogue service process and then having three containers processes on each machines. This is a starting point and will be further tuned, but it provides the ability to balance in terms of resources and failure.
WebSphere Application Server Configuration
On the application server we had installed WXS onto each machine, and created a catalogue service domain with the three catalogue service processes defined. It’s worth noting here that the name you give the grid means nothing, you can call it anything you would like.
Also we deployed the enhanced cache monitor rather than the standard version, as this provides a better view of the cache environment when using WXS.
Underneath each of the servers in the environment the following properties need to be added to the Java Virtual Machine custom settings. They are defining the change in cache environment that will be used and its setup, also the cache is remote. The first part in the following is the name of the property and after the / is the value. There may be other properties in there already, depending on your setup. One setting that was interesting is that you need to tell WXS to turn compression on.
com.ibm.websphere.xs.dynacache.disable_recursive_invalidate / true
com.ibm.websphere.xs.dynacache.enable_compression / true
com.ibm.websphere.xs.dynacache.ignore_value_in_change_event / true
com.ibm.websphere.xs.dynacache.num_initial_containers / 1
com.ibm.websphere.xs.dynacache.topology / remote
com.ibm.ws.cache.CacheConfig.cacheProviderName / com.ibm.ws.objectgrid.dynacache.CacheProviderImpl
com.ibm.ws.cache.CacheConfig.enableCacheReplication / true
One last thing is that when you go into the console even with the fixes applied it will still show the WXS level shown as 7.1.0.0. So you need to look at the systemout.log on startup to really see what fix level has been applied.
Dynacache Configuration
The dynacache configuration was the point at which, we found that the most serious changes were going to have to be made yet we had assumed it would be minimal. Problems you will have are not apparent when you run cache within the JVM of the application server, compared to how it will look with a remote grid. When you use WXS it is the case lots of bad requests or constant chatter for cache content is going to add overhead when the grid is remote. Even if the grid sits on Gigabit ethernet then ‘remote’ is remote and there will be network overhead. What we saw initially is that with local dynacache pages would take a second or so to build transfer that to a remote cache page and suddenly it was twelve seconds or more. Yet the only change was the introduction of WXS, so what could be going on? What then followed was a lot of time with the support team and here is what we found out.
WebSphere Commerce Data Caches
First we had to implement WebSphere Commerce Data Caches, it is a good article but to me not the most obvious to follow on how you do this. What it is simply saying is that there are certain types of data you can now hold in cache, that will be generated by commands. So the cache now goes beyond page fragments and pages, which is what you would typically have setip. In the case of WXS this is data that could change frequently and have a lot of invalidations, and as we will see lots of invalidation requests are not great.
So rather than holding this data in the grid in this case we want some things to still be held at the application server level, but to do this we have to define additional cache containers. It is worth noting that making this change will also help your standard Commerce environment with no WXS, especially if you are on V6, where this functionality was added later in the product. It will be a reason why a lot of sites under V6 with cache will not have this enabled.
In order to do this the first thing that needs creating is a cacheinstance.properties file, you need to put that into the /stores.war/WEB-INF/classes directory in your Commerce deployment. The file we have have created it looks something like this although it does an area to be tuned in terms of the number of cache entries. We are creating two containers one general for the DistributedMapCache and the other for ListMemberGroupForUsers, but it can be split how you see fit. When you look in the Cache Monitor you will see these containers.
cache.instance.0=/services/cache/WCDistributedMapCache
cache.instance.0.cacheSize=6000
cache.instance.0.enableDiskOffload=false
cache.instance.0.flushToDiskOnStop=false
cache.instance.0.useListenerContext=false
cache.instance.0.replicationType=1
cache.instance.0.disableDependencyId=false
cache.instance.0.filterTimeOutInvalidation=true
cache.instance.0.filterLRUInvalidation=true
cache.instance.0.ignoreValueInInvalidationEvent=true
cache.instance.0.disableTemplatesSupport=true
cache.instance.0.useServerClassLoader=true
cache.instance.1=/services/cache/com.ibm.commerce.membergroup.commands.ListMemberGroupsForUserCmdImpl
cache.instance.1.cacheSize=4000
cache.instance.1.enableDiskOffload=false
cache.instance.1.flushToDiskOnStop=false
cache.instance.1.useListenerContext=false
cache.instance.1.replicationType=1
cache.instance.1.disableDependencyId=false
cache.instance.1.filterTimeOutInvalidation=true
cache.instance.1.filterLRUInvalidation=true
cache.instance.1.ignoreValueInInvalidationEvent=true
cache.instance.1.disableTemplatesSupport=true
cache.instance.1.useServerClassLoader=true
With that file deployed then you need to add the content to define the data items into the cachespec.xml. IBM have provided an example of what data to cache in samples/dynacache/cacheinstances.properties, so take the content defining the commands and copy it into your cachespec.xml, then deploy that update.
Changes to Custom Dynacache Entries
One of the reasons that things had appeared so slow when we enabled WXS was the amount of invalidation requests that were issues, had and the way they were grouped. This is an important change you would not see locally because there is no overhead. In the cachespec we had several different stores defined and they were all doing invalidations, on content within them. It was just something which had built up over time as people had added content into the cachespec.xml file.
Because we listed all the commands that could cause an invalidation to run, what would happen is say we had a basket update the invalidations would run and eighteen different requests would be fired at the WXS environment. Each one takes time and added together that was taking quite a significant amount of overhead on the request. So we went back and updated the cachespec so that all the components we were invalidating had the same dependancy-id, and then cut down the invalidations to a single request.
On the dependancy ID that was defined on fragments such as the mini basket, previously each one had its own identifier. So the dependancy ID was Minicart:storeId:DC_userid which is how the IBM sample file would define user based fragments. Now all user based fragments in the stores say exactly the same for the ID we will use for invalidation and by doing that they are all removed in a single request on the store rather than multiple requests.
<dependency-id>StoreIDUserIDForInvalidation
<component id=”storeId” type=”parameter”>
<required>true</required>
</component>
<component id=”DC_userId” type=”attribute”>
<required>true</required>
</component>
</dependency-id>
Then after the commands that we would run invalidation such as add to basket, we define a single invalidation.
<invalidation>StoreIDUserIDForInvalidation
<component type=”method” id=”getCommandContext”>
<method>getStoreId</method>
<required>true</required>
</component>
<component type=”method” id=”getCommandContext”>
<method>getUserId</method>
<required>true</required>
</component>
</invalidation>
Once we did this we saw a major change in how the environment performed, again it would even help locally but the measurements would be much smaller periods of time than the remote requests.
Server Restart
Although changes to the cachespec.xml file will be picked up dynamically the other changes will require a server restart. If your catalogue processes are running in WXS you should see it showing that the grid is active from inside of WAS also in the systemout log file for WAS will show the containers and connections being made. If the WXS environment is not up and running I have noticed that your WAS server will just stop on startup if the grid is not available. It does not throw and error it will instead appear to do nothing.
Part II
In the next part I will look at the changes we had to make in the WXS environment from the standard setup plus any other general tips we found useful.